27 Commits

Include renames

Author	SHA1	Message	Date
timothyandClaude Opus 5	528383cf3a	fix(880): an absent recurrence array means unrestricted, an explicit [] is rejected (#892) ... The three recurrence arrays are read CONJUNCTIVELY by AlternateScheduleSelector.GetScheduleForDate, so an empty set matches no date. `?? []` on an omitted array therefore returned HTTP 200 while storing an alternate-schedule or template item that could never apply, silently -- while the read side (#823) already read a NULL column as the All*() sets. Absent and explicitly-empty are two different requests and get two answers: ABSENT (missing, or explicit null) normalizes to AlternateScheduleSelector.All*(), the same symbols the read side substitutes; EXPLICIT [] is rejected with a 422 naming the consequence, via RecurrenceSetBounds called from both replace handlers. The rejection lives in the handlers, not the controller, because api.ffmpeg-profile-numeric-bounds' "accept an UNCHANGED bad value" rule binds hardest here: both PUT paths are whole-list replaces, so rejecting a pre-existing empty set would make every OTHER item in the list uneditable. That comparison needs the stored row. The validated set is derived from `incoming`, so the highest-Index catch-all -- whose recurrence the handler discards -- is excluded by construction. Verified: full ErsatzTV.Tests suite green; three mutation proofs with disjoint reddened sets; live-E2E against a real instance confirmed an OMITTED property round-trips as unrestricted (the Newtonsoft missing-property chain unit tests cannot reach), an explicit [] returns the 422, and [] on the catch-all is accepted. Cross-family cold review BLOCKED the first implementation with 3 findings, all real and all fixed; re-review returned MERGEABLE. Follow-up #894 filed: the SPA can still build the empty state the server rejects. fixes #880 Decisions-Edit: yes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-30 09:29:02 +00:00
timothyandtimothy	ed8b602445	feat(735): bound the numeric FFmpeg profile fields with a 422, and expose readrate pacing (#847) ... Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>	2026-08-26 22:05:38 +00:00
timothyandtimothy	08cd3a002d	fix(690,758): count the same query a paged handler pages (#833) ... Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>	2026-08-26 07:39:23 +00:00
timothyandClaude Opus 5	2249a806c9	fix(671): review round 4 -- fix the chapter-title entity interpolation ... Final cold review of 34eee753b: the commit's own changes were confirmed correct, but it flagged a real pre-existing bug in the exact block I had just edited, and I was adding the first-ever tests for that method without covering it. `Playouts/Mapper.GetDisplayTitle`'s Song arm interpolated `{s}` -- the `case Song s` ENTITY -- into its chapter-title branch instead of `{t}`, the composed title. Song has no ToString() override, so a chaptered song rendered as the literal "ErsatzTV.Core.Domain.Song (Chapter 3)" in the playout guide, troubleshooting, media-item info and channel states. The sibling MusicVideo and OtherVideo arms are correct only because they happen to name their lambda `s`. Pre-existing on main; fixed here because it is one token inside the block this branch already touches. Two tests pin it; reverting renders the type name. Also: completed the guard on that arm (`Optional(s.SongMetadata).Flatten()`, the other half of the sibling pattern I claimed to have copied), added the new mechanism to the record's `mechanics:`, added the symptom tokens a future session would actually search for (ArgumentNullException, Artists, primitive collection, chaptered song) to `signals:`, restored the remedy sentence an earlier trim dropped, and trimmed to 59 prose lines for margin under the 60-line ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 20:27:08 +02:00
timothyandClaude Opus 5	34eee753b2	fix(671): review round 3 -- sweep Artists by FIELD, correct the archaeology ... Cold review of 572737a29. Findings taken; two are corrections to my own claims. CORRECTION: "a regression this branch INTRODUCED" was wrong, and I verified the reviewer's counter-claim against origin/main before accepting it. That handler already included SongMetadata AND already routed Song there, so GET /api/v1/playlists/{id}/items was ALREADY a live 500 for a null-Artists song. This branch only made the same throw reachable on a second path. The record said so twice; both are fixed, because a wrong explanation outlives a wrong line. SWEEP: fixing one site left the mirror standing -- Playouts/Mapper.GetDisplayTitle had the identical unguarded join on a path that also eager-loads SongMetadata, so it too was live, feeding the playout guide, troubleshooting, media-item info and channel states. Guarded, with a unit test; reverting it reproduces ArgumentNullException. LibraryBrowseItemMapper already wrote `Artists ?? []`, so the nullability was known in-tree and these sites were simply unswept. Filed #691 for the remaining SongVideoGenerator dereferences on the playback path. Also: documented that the shared matrix is the RERUN predicate used as a superset for playlists (the playlist write path rejects RemoteStream today); noted the third, inert consumer ReplacePlaylistItemsHandler; added the new mechanisms to the record's `mechanics:` field; and trimmed the record under the 60-line prose ceiling -- it was tipping the corpus p90 above the ceiling and reddening the calibration test in scripts/tests, which passes again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 20:09:07 +02:00
timothyandClaude Opus 5	572737a29e	fix(671): review round 2 -- guard Song.Artists, cover the second consumer ... Cold independent review of 017ef988d. Three findings taken, one filed out. The important one is a regression this branch INTRODUCED. `SongMetadata.Artists` is a nullable EF primitive collection (JSON in one column, not a navigation) that FallbackMetadataProvider leaves unassigned when a song's tags fail to read, and `string.Join` throws ArgumentNullException on a null sequence. The rerun list previously did not load SongMetadata at all, so the throw was unreachable there; adding the include promoted it to a live 500 that would have failed the whole page. Confirmed by reverting the guard: ArgumentNullException, parameter 'values'. The file header claiming every member was guarded was false. The empty case is filtered too, so an artist-less song loses its bare " - ". Second: `GetPlaylistItemsHandler` had no handler-level test at all (its controller tests stub the mediator), so the RemoteStream include added last round was discharged by inspection -- the same method that produced #671. It now runs the same 13-type matrix via a shared SelectionSeedData; removing the include fails that matrix. Third: dropped the dead `(i as Season).SeasonMetadata` include leg -- the Season projection reads Show.ShowMetadata and the scalar SeasonNumber, never SeasonMetadata. Filed #690 for the pre-existing, out-of-scope finding: the paged TotalCount ignores the search query, so the SPA renders empty pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 19:33:52 +02:00
timothyandClaude Opus 5	017ef988d0	fix(671): review round 1 -- pin exact names, complete the playlist include ... Cross-family (Codex) adversarial review of 8523088ce. All three findings taken: - The name assertion only checked "not a placeholder", so it could not see a missing NESTED include leg: dropping Episode -> Season -> Show still renders "s00e04 - Selected episode", which contains no placeholder marker and passed. Now every type pins its whole expected string; re-removing that leg fails, as verified before restoring it. - Widening the shared switch with a RemoteStream arm put `GetPlaylistItemsHandler` one include short -- it loaded metadata for the other nine types, so playlist RemoteStream names alone would have degraded to "???". - `?? 0` rendered an unloaded Season as "s00", which conventionally means Specials and so fabricated plausible-looking real data; it now renders "s??". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 19:00:08 +02:00
timothyandClaude Opus 5	8523088ceb	fix(671): resolve rerun-collection selections through one shared include chain ... The paged list handler eager-loaded nothing, so `ProjectToViewModel` read four unloaded navigations and every row of every collection type projected a null selection. Because the selected id and the display name are read off the SAME navigation, this dropped the id too -- the harm is not an unlabelled badge but an editor that round-trips a null and clears the user's stored selection. The by-id handler loaded metadata for only four of the ten selectable media types: Song/OtherVideo/Image/RemoteStream returned a null-ish selection and Episode/MusicVideo threw an NRE that surfaced as a 500. Fixed at the boundary rather than per call site: - `RerunCollectionQueryExtensions.IncludeSelectionDetails()` is now the single include chain, called by both handlers, joining the existing `ProgramScheduleItemQueryExtensions.IncludeScheduleItemDetails()` precedent (#229). Artwork legs are deliberately omitted -- this projection reads only ids and titles. - The media-item switch was duplicated verbatim for RerunCollection and PlaylistItem; both now call one `ProjectMediaItemToViewModel`, which handles `RemoteStream` (via a new `ProjectToNamedViewModel`, since the existing `ProjectToViewModel(RemoteStream)` returns an unrelated type) and never falls through to null -- an unknown subtype keeps its id and takes a conspicuous name, because throwing would fail a whole paged GET over one bad row. - Every metadata navigation in `MediaItems.Mapper` is now read through `Optional(...).Flatten()`, so an un-included nav degrades to "???" instead of being a latent 500 for whichever caller loads least. Tests enumerate all 13 supported CollectionTypes for both handlers, with the matrix derived from `IsSupportedSelectionType` so a newly-supported type joins it automatically, plus a completeness guard on the set. Each mechanism was removed in turn and confirmed red first. fixes #671 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 18:48:24 +02:00
timothy	f2d9c0dc8e	fix(668): review round 5 -- three prose nits, including an off-by-one I filed ... Final sweep confirmed the retracted MySQL over-match claim survives in no file on the branch (only in two immutable commit messages, which stay -- rewriting history would invalidate every sha-bound review verdict). Three nits remained. - The fixture's class docstring said the on-MySQL claim "rests on the server's collation", which is the one thing the decision record says it does NOT rest on. It rests on Unicode-aware LOWER(); the executed comparison bypasses the collation entirely. Reworded. - The record's `rule:` enumerated the covered fields but omitted show_genre, which GetSource and the fold both handle ("genre" or "show_genre"). Added. - My own #688 write-up was wrong twice: the 60-line ceiling warning is NON-blocking by design, and the calibration pytest reds at >=61, not >=60 -- main's p90 is 59, so a 60-line record makes p90 == ceiling and PASSES. The bullet even contradicted itself, since the next sentence relies on 60 passing. Corrected in the PR body and in the issue. Decisions-Edit: yes	2026-07-27 22:21:55 +02:00
timothy	07723e418b	fix(668): review round 4 -- sweep the retracted claim by SUBJECT, not by memory ... Round 3 said "correct the claim everywhere" and missed two places, which is the same mistake in a new coat: I fixed the spots I remembered instead of grepping for the subject. - The class-level summary of the very file round 3 edited still said MySQL's "ci collation OVER-matches instead", contradicting the method docstring forty lines below it. Reworded. - The PR body still carried the retracted over-match story -- and round 3's commit message claimed it had been corrected. It had not. Now corrected, with the measurement table and the retraction stated openly. This time the sweep was `grep -i over-match` across every file the branch touches; the remaining hits are the SQLite-fold invariant and #578 history, which are correct and stay. Also softened two overclaims the reviewer flagged. Round 3 deleted the predecessor's "configuration-incidental, not designed" hedge and replaced it with a firmer statement than the evidence supports: that MySQL cannot over-match is contingent on MySqlConnector fixing the connection collation to utf8mb4_bin, not a property of MySQL. A driver, protocol or prepared-statement change could restore it. The record and the fixture docstring now say "driver-contingent, not a law"; the code is safe either way because the ordinal filter stays regardless. Filed #689 for the source of the copied falsehood -- LibraryFolderDedupeMigrationTests on main carries the same "CI sets ETV_REQUIRE_MYSQL_TESTS=1" sentence. Decisions-Edit: yes	2026-07-27 22:09:26 +02:00
timothy	dda98efcc4	fix(668): review round 3 -- MySQL does NOT over-match; correct the claim everywhere ... Review BLOCKED on a false CI claim I copied from the sibling fixture ("CI sets ETV_REQUIRE_MYSQL_TESTS=1"). Nothing sets it; the doc now says plainly that CI does not arm this lane and points at ersatztv#627. That was the blocker. Chasing the reviewer's second finding then overturned something bigger. It predicted that seeding an unaccented "Edith" would make the in-memory ordinal filter load-bearing on MySQL, since utf8mb4_0900_ai_ci treats é as e. Mutation test says otherwise: with the filter deleted the MySQL test stays GREEN. Measured against a live 8.4 to find out why: LOWER(Name) LIKE 'é%' (literal) -> Édith AND Edith LOWER(Name) LIKE @v (ai_ci variable) -> Édith AND Edith LOWER(Name) LIKE @v COLLATE _bin -> Édith only the EF query, executed -> Édith only The driver binds the pattern with a BINARY collation, so the executed comparison is accent-SENSITIVE and MySQL does not over-match at all. MySQL's correctness rests on its Unicode-aware LOWER(), not on the collation. My earlier probe used a LITERAL pattern -- a different query from the one the code runs -- and I wrote its result into the handler comment, the decision record and the PR body. All three now say what actually happens, and the record carries the lesson: measure the query the CODE runs, not one you type. The "Edith" row stays as a near-miss control, with a docstring that says what it does and does not prove rather than the over-match story it was added for. Also moved EnsureCreatedAsync out of [SetUp]: NUnit skips [TearDown] when [SetUp] throws, so a mid-create failure would strand the database. Decisions-Edit: yes	2026-07-27 21:52:16 +02:00
timothy	ac67c9ee74	fix(668): review round 1 -- make two guards actually guard ... Independent review found both new guard tests weaker than they read. 1. A false docstring. It claimed the SQL stage "genuinely returns 'ſweet' for q='S'". It does not: 'S' is ASCII, so ContainsNonAscii is false and the fold branch is SKIPPED. Those three negative cases exercise the ASCII fast path, which is worth pinning but is not what the comment said -- and the consequence was that NO test drove a row through the fold for the ordinal filter to discard, i.e. the harmless over-match direction the whole design rests on was untested. Comment corrected and Unicode_Fold_Over_Match_Is_Discarded_By_The_Ordinal_Filter added (stored "Sword", q="ſ" -> fold runs, SQL pattern S%, SQLite returns the row, filter drops it, response empty). 2. Unicode_Fold_Escapes_Like_Wildcards could not fail if the %/_ escaping it names were deleted -- the in-memory filter masks the over-match, so the counts stay right. The escaping's real role is preventing LIMIT crowding, so Unicode_Fold_Escaping_Prevents_Limit_Crowding pins that instead. Verified by mutation: with the %/_ replaces removed the new test fails while the original two still pass. Also: the crowding residual in the decision record was attributed to MySQL alone; the SQLite fold shares it in principle, so "no accepted loss" is narrowed to mean no unreachable VALUE rather than a guaranteed count. And a comment says why the provider check is derived per-context instead of reading TvContext.IsSqlite (that static is scoped host-only by ProviderStaticsWiringTests, and reading it here would falsify the exemption). Refs #668 Decisions-Edit: yes	2026-07-27 20:50:48 +02:00
timothy	05542946ad	fix(668): reach accented facet values via a registered Unicode fold on SQLite ... SQLite's LOWER() folds ASCII only -- lower('Édith') is 'Édith' unchanged -- so the EF-sourced facet fields UNDER-matched any stored value whose prefix carries an uppercase non-ASCII character. An under-match is unrecoverable: no later stage can reintroduce a row SQL never returned. Adds a SECOND, ADDITIVE query taken only when the provider is SQLite and q contains a non-ASCII character: raw Dapper SQL folding through etv_upper(), a SqliteConnection.CreateFunction scalar implementing ToUpperInvariant. Every other case -- all-ASCII q, and MySQL for all q -- runs the existing EF query byte-identically. MySQL needed no change and gets none: verified on MySQL 8.4 that its LOWER() is Unicode-aware and its ci collation makes the predicate OVER-match, which the existing ordinal filter already discards. The fold is ToUpperInvariant because OrdinalIgnoreCase equality is a strict SUBSET of invariant-uppercase equality, so the SQL stage yields a superset of the final filter's matches and can never under-match. Note OrdinalIgnoreCase is NOT "invariant-upper then ordinal": ToUpperInvariant('ſ') is 'S', yet "ſweet".StartsWith("S", OrdinalIgnoreCase) is false. Tests pin that. No migration, no model change; both provider snapshots are untouched. Refs #668 Decisions-Edit: yes	2026-07-27 20:36:28 +02:00
timothy	fc3ede09bc	docs(578): the retracted claim survived in 9 places, including the record title and rule ... Comment- and docs-only; verified no non-comment line changed in any .cs. I reported last round that I had "classified every surviving hit". That was false, and the false confidence is the expensive part: a confidently-stated "I checked everything" stops anyone else checking. The retracted wording survived in nine places, two of them the record's title and rule: — and the catalog copies rule: verbatim, so the generated entry point and the record disagreed semantically while docs/decisions.md said the correct thing. Root cause of the miss, because it will recur otherwise: I built the sweep term list from the DELETED MECHANISM's vocabulary (LIKE, superset, keyspace, anchor, over-match) and never added the RETRACTED CLAIM's own words. "no predicate", "bound on work", "index entries", "no gap" and "holds in memory" were never grepped. After a retraction the subject list has to include the words of the thing being retracted, not just the thing already deleted. Second, worse: my first attempt at this round's sweep printed nothing for every term and I nearly read that as "all clear". zsh does not word-split an unquoted $FILES, so grep received one giant non-existent path — and the `|| echo "(none)"` never fired because the pipeline's exit status was sed's. Same failure shape as the bug arc itself: a check reporting success while examining nothing. Re-run with a proper array plus a control term ("SongMetadata" -> 42 hits) so an empty result is distinguishable from a broken grep. Fixed all nine, replacing "no predicate" with the seekable-cursor-vs-residual distinction already written correctly elsewhere: - handler: the "real bound on work" claim, the short-page rationale - SearchFieldValuesQueryShapeTests: "ANY predicate" + "reads exactly n index entries", and added what the test can and cannot pin (a SQL string, not a plan / visibility work / payload I/O) - GetSearchFieldValuesHandlerTests: "no gap between what the engine looks at and what it hands back", and the current-behaviour comment - record title, rule:, attempt-5 table row; api-conventions - regenerated docs/decisions/README.md so catalog and record agree again Tenth item, the same overclaim one level down and it survived the first retraction: the row bound was said to cap what the process holds in memory. It does not — payload width is unrestricted and one JSON array can contain arbitrarily many strings, each of which may enter the in-memory distinct set. It caps logical rows returned/materialized and round-trip count, nothing about bytes. Added as a third struck-through bullet next to the other two retractions.	2026-07-27 03:10:36 +02:00
timothy	5f73cd4482	docs(578): scope the bound to LOGICAL ROWS — physical work is not bounded, and I claimed it was ... Comment- and docs-only. Verified: the diff for both .cs files contains no non-comment line. HIGH (claim). "LIMIT n reads exactly n index entries and n rows" is false, and the bounded-physical- work reading it implied is false with it. Two mechanisms, both retracted explicitly in the record rather than quietly reworded: - MySQL purge lag. Deleted clustered-index records survive until purge runs and a range scan still traverses them, so returning 2,000 VISIBLE rows can touch far more index records. Deletion history therefore STILL affects physical work — precisely what attempt 4's failure was supposed to have made irrelevant. Attempt 5 removes the LOGICAL dependence on Id distribution; it does not make physical work independent of deletion history. - Unbounded row width. Artists/AlbumArtists are unrestricted TEXT/longtext and both SQLite and InnoDB spill to overflow pages, so a row count implies neither a byte count nor a page-read count. The guarantee is now stated as exactly what it is: at most 20,000 LOGICAL rows returned/materialized, and at most 10 round trips (11 for artist). The 392 KiB measurement is labelled one data point on one library whose credits average ~20 B/row, with an instruction to re-measure rather than extrapolate for long credit lists or MySQL over a network. Also noted: the query-shape test pins the SQL STRING — it cannot pin a plan, MVCC visibility work or payload I/O, and on MySQL using the index to satisfy ORDER BY is an optimizer choice, not a semantic guarantee. Precision fix, and the reviewer is right that the sloppy version misleads: "any predicate defeats LIMIT" is wrong, since the query's own cursor is a predicate. The distinction is a SEEKABLE PREDICATE ON THE ORDERING KEY (positions the scan, never discards a row) versus a RESIDUAL predicate (discards rows the engine already produced, so LIMIT bounds survivors and says nothing about how many were produced). Restated in the handler, the record and api-conventions. MEDIUM. docs/decisions.md still advertised "a superset LIKE pre-filter that may over-match but never under-match" for list-valued columns. That is the documented entry point for convention lookups, so a maintainer starting there was told to preserve exactly what round 4 deleted, in direct contradiction of the linked active record. Fixed. LOW. Two test comments read as current: "however many non-matching rows" (false past the ceiling — now bounded and cross-referenced to the test that pins that boundary) and the ampersand case "widens to the bare anchor" (no prefix predicate exists; kept because it is the input shape that broke the old scheme, now labelled as such). Fifth consecutive round of stale text, so this sweep was done by grepping the subject across handler, tests, record, decisions.md, api-conventions.md and the endpoint description: LIKE, ESCAPE, pre-filter, superset, over-match, under-match, anchor, keyspace, window, candidate, row cap, index entries, 392, and every deleted constant name. Every surviving hit is now either current-and-correct or explicitly framed as history.	2026-07-27 03:10:36 +02:00
timothy	373956fcee	fix(578): delete the SQL predicate — LIMIT only bounds work when there is nothing to discard ... BLOCKER. Attempt 4 bounded the Id KEYSPACE, and keyspace is not rows. Delete 20,000 historical rows, put one song at Id 20001, query artist?q=que: the walk burned all ten windows on empty ranges and returned [] for a table containing exactly one row. Capacity fell linearly with deletion ratio and no ratio was safe — one placed gap hides the next match. My record called that "heavily fragmented" and the endpoint description said loss happens "on a very large library"; the one-row example disproves both. Option A. The query now carries NO predicate at all — no LIKE, no LOWER, not even IS NOT NULL: SELECT Id, Artists AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch That is the whole fix, and it is the point. LIMIT truncates what survives a predicate, so with any predicate present it bounds the OUTPUT and says nothing about the WORK; the engine may evaluate and discard arbitrarily many rows first. Stripped to a bare primary-key range, LIMIT n reads exactly n index entries and n rows — independent of sparsity, deletion history or where the gaps fall. All selectivity moves into memory. A short page can now only mean exhaustion, which is precisely what it could not mean while a predicate was present. Four attempts, four wrong quantities: the result (a fixed budget the over-matching pre-filter starved), candidates returned (a no-match query must evaluate every eligible row before returning an empty page), keyspace width (above), and finally actual rows. The record carries the table; it is worth more than the code. Deleting the predicate deletes a whole bug family with it: the JSON-escape reasoning, the narrow-only-on-verbatim-ASCII rule, the exhaustive Unicode sweep that proved it sound, the ESCAPE '/' portability workaround, and the may-over-match-never-under-match invariant that turned out to be conditional on something untrue. SearchFieldValuesPrefilterSupersetTests is deleted entirely; the one assertion worth keeping — that the SQL has no predicate — moved to the query-shape suite, which pins the SQL string exactly so "just a cheap filter" fails a test instead of silently unbounding the walk. Measured cost of no server-side narrowing, on a seeded 20,000-song library (in-memory SQLite): worst case (no match, full walk) 20,000 rows / 10 round trips / 391.9 KiB / 119ms SQL, ~40ms warm end-to-end. Empty q, dense and non-ASCII prefixes all stop on page 1 at ~39 KiB and ~40ms. Judged acceptable for a debounced typeahead against a local file. If it ever is not, the answer is #669, not reintroducing selectivity — the record says so explicitly. Also fixed: - Round-trip count was advertised as 10; it is at most 10 for album_artist and 11 for artist, which also runs its EF query. The MAX(Id) probe is gone with the keyspace scheme, so there is no extra scalar call. - The duplicated-formula ceiling test is deleted rather than rewritten. It re-implemented the loop's arithmetic and would have passed through an off-by-one or a stall in the real loop; the dense integration tests carry that coverage. Its MaxVisited >= Window assertion was a style constraint in correctness clothing. - Stale text swept by grepping the mechanism nouns rather than re-reading: candidate/keyspace/ pre-filter/superset/row cap/LIKE/ESCAPE and the removed constant names, across handler, tests, record, api-conventions and the endpoint description. The two surviving "pre-filter" mentions are deliberate history. Test comments that rendered escaped non-ASCII as literal characters (which contradicted the raw-storage assertion in the same file) now show the escape text. New test List_Valued_Walk_Reads_Live_Rows_Regardless_Of_Id_Density reproduces the one-row killer and fails against attempt 4.	2026-07-27 03:10:36 +02:00
timothy	fbc7b2a1dd	fix(578): bound the Id KEYSPACE — LIMIT cannot bound a query that matches nothing ... BLOCKER. Round 3's ceiling counted LIKE-positive candidates, which is not the quantity that needed bounding. To return an empty page the engine must first evaluate every eligible row, so a no-match query came back with rows.Count == 0, ended the walk having counted zero against the ceiling, and had already inspected the entire table. Round trips and materialized rows were bounded; database work was not. Worse, a dense widened prefix could materialize 20x the candidates and make 10x the round trips of round 1 — a regression dressed as a bound. Third time bounding the wrong quantity: revisions 1-2 bounded the RESULT (a fixed LIMIT budget the widened pattern starved), revision 3 bounded the CANDIDATES, and neither bounds what the database LOOKS AT. Now the Id range is closed on both sides — `Id > @AfterId AND Id <= @AfterId + @Window` — so each round trip is a primary-key range scan of known width. The LIKE still decides what comes back; it no longer decides how much gets looked at. The walk advances by the WINDOW, never by what returned (an empty page means "nothing matched in this stretch", not "exhausted"), and there is deliberately no LIMIT in the SQL — the window caps the row count, and a LIMIT would only restore the illusion that it is doing the bounding. One indexed SELECT MAX(Id) up front stops the walk burning windows on empty keyspace. Result: at most 10 round trips and 20,000 rows inspected for any q, matching or not. New test `List_Valued_Walk_Cannot_Inspect_Past_The_Bound_When_Nothing_Matches` covers the case every earlier bound test missed — they all gave the pre-filter something to return. With a narrow pattern and a needle at row 30,000, revision 3 returns the needle (right answer, unbounded work) and this revision returns []. Paired with a positive control at row 20,000 inside the bound. The ceiling's remaining false negative (20,000 "zzz" rows then "éclair" at 20,001 -> []) is kept and still pinned: it is the documented bounded-best-effort contract at a rarer threshold, not a defect. Stale/false comments, several of them repeat offenders: - The handler still used "Zulu"/"Éclair" as the truncation example, where both orderings pick "Zulu". I had corrected the test and left the comment. Now "Zulu"/"apple" in both. - "leans entirely on the row cap" described a mechanism that no longer exists. - The escape example rendered as literal Édith/É/é rather than the escape TEXT (Édith, É, é) in both the handler and the record, contradicting the explanation it was giving. - "every accented artist was unsuggestable" overstated it: exactly three of the nine pinned cases fail 1b78dc9e (those where query and stored casing differ, so the escape texts diverge); the other six pass. The record now says what the test comment already said. - The record claimed the ordinal switch left response SETS unchanged. False — ordering runs before Take(limit), so with "Zulu"/"apple", empty q and limit=1 the comparer changes which value survives. - The_Scan_Ceiling_Is_A_Whole_Number_Of_Batches was a style preference wearing a correctness costume (a 20,500 ceiling is perfectly safe — the final window clamps). Replaced with the invariant that actually matters: the walk's own arithmetic terminates and lands exactly on the ceiling.	2026-07-27 03:10:35 +02:00
timothy	a37847e509	fix(578): a fixed row budget turns an over-matching prefilter into a false negative ... BLOCKER 1. The invariant a883e5f0 established — "the pre-filter may over-match, it must never under-match" — is only sound while the candidate set is not truncated. It was truncated. A non-ASCII or JSON-escaped prefix collapses the pattern to the bare `%"%` anchor, so every row becomes a candidate, and `ORDER BY Id LIMIT 1000` then spent the whole budget on rows that could not match. Seed 1000 songs by "zzz", put the only "éclair" in row 1001, ask for album_artist?q=é: a883e5f0 returns [], while 1b78dc9e returned "éclair" because its (separately broken) tighter pattern kept the candidate set small. Neither revision was correct — the old one under-matched at the pattern, the new one under-matched at the cap. Widening a predicate under a fixed budget starves it. So the budget is gone. Candidate rows are now walked keyset-paged on Id (`Id > @AfterId … ORDER BY Id LIMIT @Batch`), continuing past non-matching candidates and stopping on the first of: enough distinct exact matches for `limit`, a short page (source exhausted), or a 20,000-candidate-row ceiling in 2,000-row batches. The bound is on effort; it no longer silently decides the result, and the lossy case needs 20,000 rows that already passed the pre-filter before it bites. BLOCKER 2. The endpoint description and the record's rule claimed ordinal matching/dedup/ordering endpoint-wide. False for EF-backed fields: the database runs LOWER/DISTINCT/ORDER BY/LIMIT before any ordinal code, so `genre?q=é` still misses a stored "Éclair" on SQLite. Both are now scoped to the final in-memory stages, and the underlying gap is referenced as #668 rather than described as fixed. #669 (normalized SongArtist table) is referenced as the follow-up for the scan cost. Accuracy corrections to my own claims, all verified by re-running the mutations: - Only THREE of the nine Unicode cases fail 1b78dc9e (é/édith/BJÖRK — where query and stored casing differ, so the escape texts diverge); the other six pass it. The comment said all nine. They stay as continuity coverage, now labelled as such rather than as regression guards. - Ordering_Is_Best_Effort used "Zulu"/"Éclair", where DB and ordinal orderings BOTH pick "Zulu" — it could not demonstrate the divergence it claimed. Now "Zulu"/"apple", which actually diverges: ordinal ranks "Zulu" first, the DB ranks "apple" first, and limit=1 returns ["apple"]. The record sentence was false and is corrected. - The record printed literal "Édith"/"é" where it needed to show the escape TEXT (Édith, é), contradicting the very explanation it was giving. - Corrected the cost claim: the leading wildcard forces scan ACCESS, but each page stops once it has filled @Batch, so a dense query finishes early — it is not necessarily a full table scan. - The Unicode sweep is labelled a PROOF OBLIGATION: it is revision-independent and passes every revision, which is correct for what it is but must not read as regression coverage. One process note: the new record's frontmatter had a lone apostrophe inside a single-quoted YAML scalar ("SQLite's"). decisions_validate.py's hand parser accepted it; scripts/tests caught it.	2026-07-27 03:10:35 +02:00
timothy	1641ca8305	fix(578): the LIKE prefilter under-matched every accented artist; make the superset provable ... Review of 1b78dc9e found the pre-filter's correctness claim was false, and the claim was in the decision record as well as the code. F1 (high). The pattern JSON-encoded the whole query prefix on the reasoning that the stored text escapes non-ASCII, so encoding the prefix the same way would line up. It does not: SQL LOWER() lowercases the *escape text* (`É` -> `é`); it cannot case-fold the codepoint that escape denotes. So `q=é` built `%"é%`, the stored `Édith Piaf` never matched, and the row was discarded before the in-memory filter could accept it. Every accented artist — Beyoncé, Björk, Sigur Rós, Édith Piaf — was silently unsuggestable, which in a music library is the common case. The invariant that was missing, now stated in the code: the SQL pre-filter is an OPTIMIZATION. It may over-match; it must never under-match. Correctness lives in the in-memory filter. So the pattern now narrows only on the leading run of characters the JSON writer stores verbatim and stops at the first character it cannot prove — `q=Beyoncé` still narrows on `beyonc`, `q=é` narrows on nothing and leans on the row cap. Soundness rests on two facts now asserted by exhaustive computation rather than argued: no non-ASCII codepoint in U+0080..U+10FFFF OrdinalIgnoreCase-equals a printable ASCII character (false for InvariantCultureIgnoreCase, which folds ~190 — the choice of Ordinal is load-bearing), and the exact set of ASCII the encoder escapes. F1b. `UseRequestLocalization` honours Accept-Language, so the culture was caller-controlled and `ToLower()` plus the default linguistic `StartsWith(string)` let a header change the answer. Comparison is now OrdinalIgnoreCase and ordering StringComparer.Ordinal throughout — including the shared FilterSortTake that state/video_dynamic_range/content_rating also use. Sets unchanged, order now ordinal rather than culture-dependent. F2. The merge comment asserted an exactness the code does not have: sources truncate by their own ordering (DB collation / primary key), not the merge's, so a dropped value can outrank a survivor. Comment and record now say best-effort, exact only below the truncation points. F3/F4. The cap now rides `ORDER BY Id` rather than the JSON column: MySQL sorts TEXT by only max_sort_length bytes, so the old ordering was not deterministic there, and sorting the whole matching set was avoidable work. What the cap still does NOT bound is the scan — a leading-wildcard LIKE cannot seek an index — so that cost is now documented as accepted, with a normalized `SongArtist` table named as the follow-up candidate rather than left implicit. Every clause above is covered by a test verified to FAIL when that clause is mutated (old pattern builder: 5 red; culture chain: 3 red; cap=3 / cap=limit / ORDER BY json / no cap: red each). F5. Converted to a proper supersession. The old record did not merely hold a stale fact — it recorded song/music-video credits as an "intentionally-uncovered gap" and album_artist as unsupported, and this reverses that call, which `docs.decision-lifecycle` says is never a line-edit. `api.search-field-values` is archived with its original prose restored, and `api.search-field-values-sources` replaces it carrying the whole endpoint contract.	2026-07-27 03:10:35 +02:00
timothy	cd6f36185c	feat(578): artist typeahead covers music-video and song credits; album_artist stops 404ing ... `GET /api/v1/search/fields/{name}/values` sourced `artist` from `ArtistMetadata.Title` only — entity artists — so the free-text credits that `LuceneSearchIndex` also writes to the `artist` field (`MusicVideoArtist.Name`, `SongMetadata.Artists`) produced no suggestions, and `album_artist` 404'd outright. `MusicVideoArtist` turned out to be a real entity table, so it just joins the existing server-side pipeline as a `Concat` — one bounded `UNION ALL` + `LOWER(...) LIKE ... LIMIT` on both providers. `SongMetadata.Artists`/`AlbumArtists` are the hard case: EF 9 maps them as primitive collections (one JSON array per row in a single column), and neither provider can project the elements server-side — SQLite needs the SQL APPLY operator it lacks, Pomelo MySQL 9.0.0 has no primitive-collection support at all. Both failures are now pinned by a test, so a provider upgrade that fixes them shows up as a red rather than as a stale workaround. For those columns the handler pre-filters on the raw JSON (`LOWER(col) LIKE '%"<encoded-prefix>%' ESCAPE '/'` — a deliberate superset, since it matches a row and not an element), caps the rows at 1000 with an `ORDER BY` that makes the truncation deterministic, then splits and exact-filters in memory. Provider portability is by construction rather than by trusting `LIKE`: the prefix is JSON-encoded before matching (which is how it is stored, and which makes the pattern pure ASCII, so SQLite's ASCII-only `lower()` and MySQL's Unicode-aware `LOWER()` agree) and lowercased in C#, so the match is correct under a case-sensitive MySQL collation as well as a case-insensitive one. The escape character is `/`, never `\`: `ESCAPE '\'` is not a portable SQL literal. An empty `q` stays supported for these fields — the row cap already bounds it, and a non-empty-`q` carve-out would make one group of fields behave differently for the same client code. Docs: new `api.search-field-values-list-columns` record (additive sibling — the base record's rule still reads true; only its body's "intentionally-uncovered gap" claim was stale), plus `api-conventions.md`, `spa-conventions.md` §12 and the regenerated `v1.json`. Decisions-Edit: yes	2026-07-27 03:10:35 +02:00
timothyandClaude Opus 5	37fd30dce7	fix(631): quote a record title whose unquoted # was a YAML comment ... Third latent defect surfaced purely by running scripts/tests in CI — and this one was already red on origin/main, where test_frontmatter_reader_matches_pyyaml_on_every_real_record fails. `api/paging-zero-based.md` carried an UNQUOTED plain scalar title ending "... (#616, #633)". In YAML a space followed by `#` starts a COMMENT, so PyYAML reads the title as truncated at "(#616," while the hand-written dependency-free reader keeps the whole line. PyYAML is correct per spec; the record's title was genuinely losing "#633)" for any spec-compliant consumer, which includes MemPalace's ingest. Fixed by quoting the scalar; both parsers now agree byte-for-byte. Asked "instance or class?" before fixing: swept every frontmatter field of all 179 records under both parsers and compared. This is the ONLY divergence in the corpus, so a one-line fix is the right scope — no lint rule needed, and #621's structural guard already makes a parse FAILURE loud (this was a parse-to-different, which no structural check can see). 216 passed under jq 1.8.2 and jq 1.6. Refs #631 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-26 13:27:05 +02:00
timothy	214fad2dcd	fix(633): document the 0-based paging contract on the OpenAPI parameters ... `api.paging-zero-based` says `pageNum` is 0-based across `/api/v1` and every wrapper of it. That was true of the MCP tool catalog and the docs, and not true of the generated OpenAPI document: all 24 paging parameters across the 12 paged operations were emitted with no `description` at all, so a consumer reading only `v1.json` — the intended contract, and what generated clients surface to their users — had to infer the base from `default: 0`. That is the same inference that cost #487 a verification pass on the MCP side, where the description was present but wrong. Annotates each `[FromQuery]` paging parameter with `[Description]` (`System.ComponentModel`), the mechanism `parentId` already used in ImagesController, and regenerates `v1.json`. `pageSize` states the endpoint's OWN cap, because the caps genuinely differ — 100 typical, 200 auto-tune members, 1000 search/all-items — and the record forbids documenting one global number; it also states that the offset derives from the effective (capped) size, so an over-large `pageSize` narrows the page instead of widening the offset. The generated TypeScript client covers DTOs only, not query parameters, so it is unchanged; `endpoint-index.md` carries summaries, not parameter descriptions, so it is unchanged too. Pinned by OpenApiPagingContractTests against the in-process generated document. The test NAMES the expected set of 12 paged operations rather than only filtering for parameters called `pageNum`: a filter cannot see an endpoint that should page and doesn't, which is exactly how two MCP tools escaped the equivalent check in #616. Set equality is asserted in both directions, and the caps are pinned per endpoint so a description naming the wrong cap fails — a wrong justification outlives a wrong line. Mutation-verified both ways: dropping one `[Description]` reddens the description test, and making one endpoint stop exposing `pageNum`/`pageSize` under those names reddens the set-equality test. Refs #633 Decisions-Edit: yes	2026-07-26 11:10:22 +02:00
timothyandClaude Opus 5	78ec997eae	docs(616): stop the record title and mcp.md from contradicting their own bodies ... Two LOWs from the re-review of af39123e. Both are the same defect I have already corrected twice on this branch: a universal claim that the corrected body right below it disproves. These read as normative, so a reader who stops at the title gets the false version. - The record was titled "...and every wrapper says so" while its own body admits the 12 OpenAPI pageNum parameters carry no description at all (tracked as #633). Retitled to state the target and name the exception. - docs/mcp.md led with "Rows that reference another entity carry that entity's id" — but PlayoutListItemResponseModel.ScheduleName ships with no schedule id. Restated as a rule about ACTIONABLE references, matching the wording the record already uses, with the known exception named. The two remaining LOWs are accepted deferrals, not fixed here: the >100 rerun collection truncation needs paging-to-completeness plus a multi-page fixture (#634), and the catalog test's forward-looking gap (it compares tool names, not PathTemplate against the paged-endpoint set) would need the MCP test project to reference the controllers to close properly. Docs only; catalog regenerated. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-25 23:51:14 +02:00
timothyandClaude Opus 5	af39123e8e	fix(616): fix a live 1-based SPA caller and un-cap two MCP playout tools ... Findings from the cold cross-family review of 8d35a279/5648f8e9. The review confirmed the three conclusions in 8d35a279 (the offset math is sound on all 12 paged endpoints; the list DTO really did already carry ChannelId; the new tests are non-vacuous) but found the previous commits had documented a convention without checking who actually violates it. HIGH — a live instance of this issue's own bug class, in the SPA. web/src/screens/SchedulesScreen.tsx asked for the rerun-collection picker with `pageNum: 1, pageSize: 1000`. pageNum is 0-based and pageSize clamps to 100, so the request skipped the first 100 rows: with <=100 rerun collections (the normal case) the picker was served an EMPTY page and silently offered no rerun collections at all; above 100 it dropped rows 1-100. Exactly the silent-short-set failure #616 is about, shipped in the UI. Now `pageNum: 0`, with a vitest that asserts the offset and is mutation-verified (restoring `pageNum: 1` fails it). Checked the rest of the SPA rather than assuming: every other pageNum caller is 0-based. CollectionsScreen's paging loop starts at 1 but only after fetching page 0 explicitly, so it is correct — verified before touching it. MEDIUM — two MCP tools wrapped paged endpoints while declaring no paging args. ersatztv_list_playouts and ersatztv_get_playout_items had no pageNum/pageSize, and ToolArgumentValidator rejects undeclared arguments, so an agent was hard capped at the first 100 rows with no way to ask for more and no error saying so. Both now take Page(). A catalog-wide sweep confirmed these were the only two: the other paged endpoints are not exposed as MCP tools at all. That same gap made the new ToolCatalog test vacuous in the direction that mattered — it filtered on tools that ALREADY declare pageNum, so a tool missing paging entirely escaped it. It now pins the expected set by name, so a new tool over a paged endpoint has to be added deliberately. Record corrections (these are read as normative, so over-broad claims are defects): "every wrapper says 0-based" was false for the OpenAPI surface, whose 12 pageNum parameters carry no description — named as a remaining gap instead of claimed as done. "Every controller floors with Math.Max" ignored SearchController's Math.Clamp. The ids-in-rows corollary was stated as an audit result when PlayoutListItemResponseModel.ScheduleName has no schedule id; restated as a rule about actionable ids. Verification: .NET 1900 + MCP 59 pass; web 996 pass across 105 files; tsc clean; eslint clean; format gate exit 0; no BOMs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-25 23:41:54 +02:00
timothyandClaude Opus 5	5648f8e92e	docs(616): scope the pageSize cap per endpoint — it is not one number ... Self-review of the previous commit caught an overclaim I introduced. I wrote the paging docs as if the pageSize cap were uniformly 100. It is not: most reads (collections, library browse, logs, playouts) Math.Clamp(.., 1, 100) GetAutoTuneChannelMembers pageSize <= 0 ? 100 : Min(.., 200) GET /api/v1/search/all-items default 500, cap 1000 That made a concrete example in docs/mcp.md simply false. It claimed `pageSize=500&pageNum=1` returns items 101-200 "not 501-1000" — but ersatztv_search_all_items also takes Page(), and 500 is UNDER its cap, so there page 1 really is items 501-1000. A caller following that example on the one tool most likely to be paged hard would have mis-derived its offsets, which is the same class of silent-short-set error this issue is about. The invariant that actually holds everywhere is the derivation, not any single cap: the offset comes from the EFFECTIVE (bounded) page size, never the requested one. Reworded to say that, in docs/mcp.md, the api.paging-zero-based record (rule + mechanics + body), and the ToolCatalog Page() comment. Catalog regenerated. The record's mechanics line no longer claims every controller uses Math.Clamp — ChannelController does not. No behaviour change; MCP suite still 59/59. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-25 23:24:40 +02:00
timothyandClaude Opus 5	8d35a2792f	fix(616): document paging as 0-based, expose channelId on playout detail ... #616 filed three MCP/API paging traps. Two were real; one was not, and one was already half-fixed on main. Verified each against the code before changing it. REAL — pageNum documented as 1-based. `ToolCatalog.Page()` described pageNum as "1-based page number" while every paged controller defaults it to 0, floors it with `Math.Max(0, pageNum)`, and skips `PageNum * PageSize`. A caller that trusted the description started at page 1 and silently lost the first page: no error, just a short set that reads as data loss rather than an off-by-one (it cost #487 a verification pass). Fixed in the description rather than by making the MCP layer 1-based: /api/v1 is additive-only post-freeze, 0-based is load-bearing in a dozen controllers and the SPA, and a 1-based wrapper over a 0-based API would make the same parameter name mean two different things on two surfaces a reader reads together. NOT REAL — "pageSize caps the page but the offset honors the requested value". Not reproducible on any endpoint. Every controller clamps before passing, every handler skips by the clamped size, and GetCollectionItemsHandler re-clamps defensively. The reported observation (pageSize=500&pageNum=2 on a 204-item collection returning 4 items) is exactly correct 0-based behaviour at the clamped width of 100 — page 2 is items 201-204. The issue's own trap-1 table states this. Pinned by test rather than "fixed". ALREADY FIXED — playout LIST rows gained channelId in #297 (2026-07-22), three days before #616 was filed; the report was measured against prod, which runs an older :prod image. The DETAIL response (PlayoutResponseModel) genuinely still lacked it, so channelId is added there (additive) and the reset_channel_playout argument now names the trap: the id spaces overlap numerically, so passing a playout id silently resets a different channel and returns a plausible 202. Tests, both mutation-verified (each fails when its fix is reverted): - ToolCatalogTests pins "0-based" on EVERY paged tool's pageNum description, with a non-empty guard so it can't pass vacuously over an empty tool set. - GetCollectionItemsHandlerTests pins 0-based page boundaries and proves the offset derives from the clamped pageSize (page 1 at pageSize=500 returns items 101-150; the mutation that honors 500 returns an empty page). Docs: new decision record api.paging-zero-based (catalog regenerated), the api-conventions paging bullet, and a Paging section in docs/mcp.md. OpenAPI v1.json + web/src/api/generated/v1.d.ts regenerated for the added field. fixes #616 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-25 23:16:00 +02:00
timothy	fba5233caf	feat(610): split the decision corpus into one YAML-frontmatter file per record ... 168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key, so one-active-record-per-key becomes a filesystem property rather than a validator check, and supersession becomes a `git mv`. WHY: the monolith was a concurrency problem before an aesthetic one. A 3,900-line append target made parallel sessions collide -- PR #605 and PR #614 both hit append-vs-append conflicts during routine rebases, and hand-resolving those inside the corpus is exactly the operation the rationale-rewrite guard exists to police. HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness does not rest on reading it. The parser was taught BOTH formats first, so the body-diff guard parses the old form at the merge-base and the new form at head -- the migration validates itself, no bypass. The proof is a field-level equivalence harness: 168 records before and after, zero lost, zero gained, zero field mismatches, zero rationale bodies differing. Reviewers should scrutinise the harness; it is the actual evidence. What measuring caught that reading would not have: - ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each topic file's preamble, mostly the only copy. Source files are kept and stripped, never deleted. They also cannot be filed per-area: topic files hold several areas and 4 of 23 areas span several files. - Archive discovery was a non-recursive glob; after the split it found ZERO archived records, surfacing as four bogus "supersedes points to unknown key" errors rather than an obvious failure. - ~32 live docs point into the corpus BY DATE, which the split dangles. Each stripped file now ends with a generated "Records formerly in this file" index, which also rescues the identical breadcrumbs in old issue comments. - decisions.md's "In this file:" list was 97 same-file anchor bullets that the split makes WRONG, not merely stale. Dropped; the generated index replaces them with links that resolve. The equivalence harness now runs against a checked-in FIXTURE, not the live corpus. The earlier version migrated the real tree, which made it a one-shot: the moment the migration landed there was nothing left to move and the tests failed for reasons unrelated to the code. A fixture keeps them testing the SCRIPT rather than the repo's current state. Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain two directories for one concept. Renaming a key is not a move -- it changes identity, breaks the equivalence proof, and invalidates MemPalace's per-key drawers. Taxonomy normalisation is separate work. refs #610	2026-07-25 19:45:09 +02:00