16 Commits

Include renames

Author	SHA1	Message	Date
timothy	373956fcee	fix(578): delete the SQL predicate — LIMIT only bounds work when there is nothing to discard ... BLOCKER. Attempt 4 bounded the Id KEYSPACE, and keyspace is not rows. Delete 20,000 historical rows, put one song at Id 20001, query artist?q=que: the walk burned all ten windows on empty ranges and returned [] for a table containing exactly one row. Capacity fell linearly with deletion ratio and no ratio was safe — one placed gap hides the next match. My record called that "heavily fragmented" and the endpoint description said loss happens "on a very large library"; the one-row example disproves both. Option A. The query now carries NO predicate at all — no LIKE, no LOWER, not even IS NOT NULL: SELECT Id, Artists AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch That is the whole fix, and it is the point. LIMIT truncates what survives a predicate, so with any predicate present it bounds the OUTPUT and says nothing about the WORK; the engine may evaluate and discard arbitrarily many rows first. Stripped to a bare primary-key range, LIMIT n reads exactly n index entries and n rows — independent of sparsity, deletion history or where the gaps fall. All selectivity moves into memory. A short page can now only mean exhaustion, which is precisely what it could not mean while a predicate was present. Four attempts, four wrong quantities: the result (a fixed budget the over-matching pre-filter starved), candidates returned (a no-match query must evaluate every eligible row before returning an empty page), keyspace width (above), and finally actual rows. The record carries the table; it is worth more than the code. Deleting the predicate deletes a whole bug family with it: the JSON-escape reasoning, the narrow-only-on-verbatim-ASCII rule, the exhaustive Unicode sweep that proved it sound, the ESCAPE '/' portability workaround, and the may-over-match-never-under-match invariant that turned out to be conditional on something untrue. SearchFieldValuesPrefilterSupersetTests is deleted entirely; the one assertion worth keeping — that the SQL has no predicate — moved to the query-shape suite, which pins the SQL string exactly so "just a cheap filter" fails a test instead of silently unbounding the walk. Measured cost of no server-side narrowing, on a seeded 20,000-song library (in-memory SQLite): worst case (no match, full walk) 20,000 rows / 10 round trips / 391.9 KiB / 119ms SQL, ~40ms warm end-to-end. Empty q, dense and non-ASCII prefixes all stop on page 1 at ~39 KiB and ~40ms. Judged acceptable for a debounced typeahead against a local file. If it ever is not, the answer is #669, not reintroducing selectivity — the record says so explicitly. Also fixed: - Round-trip count was advertised as 10; it is at most 10 for album_artist and 11 for artist, which also runs its EF query. The MAX(Id) probe is gone with the keyspace scheme, so there is no extra scalar call. - The duplicated-formula ceiling test is deleted rather than rewritten. It re-implemented the loop's arithmetic and would have passed through an off-by-one or a stall in the real loop; the dense integration tests carry that coverage. Its MaxVisited >= Window assertion was a style constraint in correctness clothing. - Stale text swept by grepping the mechanism nouns rather than re-reading: candidate/keyspace/ pre-filter/superset/row cap/LIKE/ESCAPE and the removed constant names, across handler, tests, record, api-conventions and the endpoint description. The two surviving "pre-filter" mentions are deliberate history. Test comments that rendered escaped non-ASCII as literal characters (which contradicted the raw-storage assertion in the same file) now show the escape text. New test List_Valued_Walk_Reads_Live_Rows_Regardless_Of_Id_Density reproduces the one-row killer and fails against attempt 4.	2026-07-27 03:10:36 +02:00
timothy	a37847e509	fix(578): a fixed row budget turns an over-matching prefilter into a false negative ... BLOCKER 1. The invariant a883e5f0 established — "the pre-filter may over-match, it must never under-match" — is only sound while the candidate set is not truncated. It was truncated. A non-ASCII or JSON-escaped prefix collapses the pattern to the bare `%"%` anchor, so every row becomes a candidate, and `ORDER BY Id LIMIT 1000` then spent the whole budget on rows that could not match. Seed 1000 songs by "zzz", put the only "éclair" in row 1001, ask for album_artist?q=é: a883e5f0 returns [], while 1b78dc9e returned "éclair" because its (separately broken) tighter pattern kept the candidate set small. Neither revision was correct — the old one under-matched at the pattern, the new one under-matched at the cap. Widening a predicate under a fixed budget starves it. So the budget is gone. Candidate rows are now walked keyset-paged on Id (`Id > @AfterId … ORDER BY Id LIMIT @Batch`), continuing past non-matching candidates and stopping on the first of: enough distinct exact matches for `limit`, a short page (source exhausted), or a 20,000-candidate-row ceiling in 2,000-row batches. The bound is on effort; it no longer silently decides the result, and the lossy case needs 20,000 rows that already passed the pre-filter before it bites. BLOCKER 2. The endpoint description and the record's rule claimed ordinal matching/dedup/ordering endpoint-wide. False for EF-backed fields: the database runs LOWER/DISTINCT/ORDER BY/LIMIT before any ordinal code, so `genre?q=é` still misses a stored "Éclair" on SQLite. Both are now scoped to the final in-memory stages, and the underlying gap is referenced as #668 rather than described as fixed. #669 (normalized SongArtist table) is referenced as the follow-up for the scan cost. Accuracy corrections to my own claims, all verified by re-running the mutations: - Only THREE of the nine Unicode cases fail 1b78dc9e (é/édith/BJÖRK — where query and stored casing differ, so the escape texts diverge); the other six pass it. The comment said all nine. They stay as continuity coverage, now labelled as such rather than as regression guards. - Ordering_Is_Best_Effort used "Zulu"/"Éclair", where DB and ordinal orderings BOTH pick "Zulu" — it could not demonstrate the divergence it claimed. Now "Zulu"/"apple", which actually diverges: ordinal ranks "Zulu" first, the DB ranks "apple" first, and limit=1 returns ["apple"]. The record sentence was false and is corrected. - The record printed literal "Édith"/"é" where it needed to show the escape TEXT (Édith, é), contradicting the very explanation it was giving. - Corrected the cost claim: the leading wildcard forces scan ACCESS, but each page stops once it has filled @Batch, so a dense query finishes early — it is not necessarily a full table scan. - The Unicode sweep is labelled a PROOF OBLIGATION: it is revision-independent and passes every revision, which is correct for what it is but must not read as regression coverage. One process note: the new record's frontmatter had a lone apostrophe inside a single-quoted YAML scalar ("SQLite's"). decisions_validate.py's hand parser accepted it; scripts/tests caught it.	2026-07-27 03:10:35 +02:00
timothy	1641ca8305	fix(578): the LIKE prefilter under-matched every accented artist; make the superset provable ... Review of 1b78dc9e found the pre-filter's correctness claim was false, and the claim was in the decision record as well as the code. F1 (high). The pattern JSON-encoded the whole query prefix on the reasoning that the stored text escapes non-ASCII, so encoding the prefix the same way would line up. It does not: SQL LOWER() lowercases the *escape text* (`É` -> `é`); it cannot case-fold the codepoint that escape denotes. So `q=é` built `%"é%`, the stored `Édith Piaf` never matched, and the row was discarded before the in-memory filter could accept it. Every accented artist — Beyoncé, Björk, Sigur Rós, Édith Piaf — was silently unsuggestable, which in a music library is the common case. The invariant that was missing, now stated in the code: the SQL pre-filter is an OPTIMIZATION. It may over-match; it must never under-match. Correctness lives in the in-memory filter. So the pattern now narrows only on the leading run of characters the JSON writer stores verbatim and stops at the first character it cannot prove — `q=Beyoncé` still narrows on `beyonc`, `q=é` narrows on nothing and leans on the row cap. Soundness rests on two facts now asserted by exhaustive computation rather than argued: no non-ASCII codepoint in U+0080..U+10FFFF OrdinalIgnoreCase-equals a printable ASCII character (false for InvariantCultureIgnoreCase, which folds ~190 — the choice of Ordinal is load-bearing), and the exact set of ASCII the encoder escapes. F1b. `UseRequestLocalization` honours Accept-Language, so the culture was caller-controlled and `ToLower()` plus the default linguistic `StartsWith(string)` let a header change the answer. Comparison is now OrdinalIgnoreCase and ordering StringComparer.Ordinal throughout — including the shared FilterSortTake that state/video_dynamic_range/content_rating also use. Sets unchanged, order now ordinal rather than culture-dependent. F2. The merge comment asserted an exactness the code does not have: sources truncate by their own ordering (DB collation / primary key), not the merge's, so a dropped value can outrank a survivor. Comment and record now say best-effort, exact only below the truncation points. F3/F4. The cap now rides `ORDER BY Id` rather than the JSON column: MySQL sorts TEXT by only max_sort_length bytes, so the old ordering was not deterministic there, and sorting the whole matching set was avoidable work. What the cap still does NOT bound is the scan — a leading-wildcard LIKE cannot seek an index — so that cost is now documented as accepted, with a normalized `SongArtist` table named as the follow-up candidate rather than left implicit. Every clause above is covered by a test verified to FAIL when that clause is mutated (old pattern builder: 5 red; culture chain: 3 red; cap=3 / cap=limit / ORDER BY json / no cap: red each). F5. Converted to a proper supersession. The old record did not merely hold a stale fact — it recorded song/music-video credits as an "intentionally-uncovered gap" and album_artist as unsupported, and this reverses that call, which `docs.decision-lifecycle` says is never a line-edit. `api.search-field-values` is archived with its original prose restored, and `api.search-field-values-sources` replaces it carrying the whole endpoint contract.	2026-07-27 03:10:35 +02:00
timothy	cd6f36185c	feat(578): artist typeahead covers music-video and song credits; album_artist stops 404ing ... `GET /api/v1/search/fields/{name}/values` sourced `artist` from `ArtistMetadata.Title` only — entity artists — so the free-text credits that `LuceneSearchIndex` also writes to the `artist` field (`MusicVideoArtist.Name`, `SongMetadata.Artists`) produced no suggestions, and `album_artist` 404'd outright. `MusicVideoArtist` turned out to be a real entity table, so it just joins the existing server-side pipeline as a `Concat` — one bounded `UNION ALL` + `LOWER(...) LIKE ... LIMIT` on both providers. `SongMetadata.Artists`/`AlbumArtists` are the hard case: EF 9 maps them as primitive collections (one JSON array per row in a single column), and neither provider can project the elements server-side — SQLite needs the SQL APPLY operator it lacks, Pomelo MySQL 9.0.0 has no primitive-collection support at all. Both failures are now pinned by a test, so a provider upgrade that fixes them shows up as a red rather than as a stale workaround. For those columns the handler pre-filters on the raw JSON (`LOWER(col) LIKE '%"<encoded-prefix>%' ESCAPE '/'` — a deliberate superset, since it matches a row and not an element), caps the rows at 1000 with an `ORDER BY` that makes the truncation deterministic, then splits and exact-filters in memory. Provider portability is by construction rather than by trusting `LIKE`: the prefix is JSON-encoded before matching (which is how it is stored, and which makes the pattern pure ASCII, so SQLite's ASCII-only `lower()` and MySQL's Unicode-aware `LOWER()` agree) and lowercased in C#, so the match is correct under a case-sensitive MySQL collation as well as a case-insensitive one. The escape character is `/`, never `\`: `ESCAPE '\'` is not a portable SQL literal. An empty `q` stays supported for these fields — the row cap already bounds it, and a non-empty-`q` carve-out would make one group of fields behave differently for the same client code. Docs: new `api.search-field-values-list-columns` record (additive sibling — the base record's rule still reads true; only its body's "intentionally-uncovered gap" claim was stale), plus `api-conventions.md`, `spa-conventions.md` §12 and the regenerated `v1.json`. Decisions-Edit: yes	2026-07-27 03:10:35 +02:00
timothy	23791c1bbb	fix(633): review fixes — the all-items upper clamp, and a cap test that couldn't fail ... Independent review (Codex) found three, two of them real defects rather than polish. 1. `search/all-items` is the one paged endpoint that clamps `pageNum` ABOVE as well as below — `Math.Clamp(pageNum, 0, MaxAllItemsPageNum)`, 2,000,000, so pageNum*pageSize cannot overflow int into a 500. The description documented only the lower clamp, so the published contract looked unbounded: a client sending pageNum=int.MaxValue is silently served page 2,000,000. Now stated, and called out as the exception it is. 2. The cap assertion could not fail in the direction that matters. `ShouldContain( "capped at 100")` is satisfied by the string "capped at 1000", so a cap-100 endpoint whose description claimed 1000 passed — precisely the wrong-cap defect the test was added to catch, and a test that cannot fail on its own subject is worse than none. Matched as a whole token instead, and mutation-verified: making /logs claim 1000 now reddens it, where before it stayed green. 3. `Description` used `First`, so a missing parameter threw "Sequence contains no matching element" — naming neither endpoint nor parameter, and reading as a broken test rather than the contract violation it is. Fails informatively now. The review confirmed independently that 12 is the complete paged set, that every other cap matches its controller, that the attributes are runtime-inert, and that the unchanged TypeScript client and endpoint index are correct rather than a missed regen. Refs #633 Decisions-Edit: yes	2026-07-26 11:23:51 +02:00
timothy	214fad2dcd	fix(633): document the 0-based paging contract on the OpenAPI parameters ... `api.paging-zero-based` says `pageNum` is 0-based across `/api/v1` and every wrapper of it. That was true of the MCP tool catalog and the docs, and not true of the generated OpenAPI document: all 24 paging parameters across the 12 paged operations were emitted with no `description` at all, so a consumer reading only `v1.json` — the intended contract, and what generated clients surface to their users — had to infer the base from `default: 0`. That is the same inference that cost #487 a verification pass on the MCP side, where the description was present but wrong. Annotates each `[FromQuery]` paging parameter with `[Description]` (`System.ComponentModel`), the mechanism `parentId` already used in ImagesController, and regenerates `v1.json`. `pageSize` states the endpoint's OWN cap, because the caps genuinely differ — 100 typical, 200 auto-tune members, 1000 search/all-items — and the record forbids documenting one global number; it also states that the offset derives from the effective (capped) size, so an over-large `pageSize` narrows the page instead of widening the offset. The generated TypeScript client covers DTOs only, not query parameters, so it is unchanged; `endpoint-index.md` carries summaries, not parameter descriptions, so it is unchanged too. Pinned by OpenApiPagingContractTests against the in-process generated document. The test NAMES the expected set of 12 paged operations rather than only filtering for parameters called `pageNum`: a filter cannot see an endpoint that should page and doesn't, which is exactly how two MCP tools escaped the equivalent check in #616. Set equality is asserted in both directions, and the caps are pinned per endpoint so a description naming the wrong cap fails — a wrong justification outlives a wrong line. Mutation-verified both ways: dropping one `[Description]` reddens the description test, and making one endpoint stop exposing `pageNum`/`pageSize` under those names reddens the set-equality test. Refs #633 Decisions-Edit: yes	2026-07-26 11:10:22 +02:00
timothyandClaude Opus 4.8	7e6861a928	fix(434): accurate endpoint description (DB-sourced) + tests for case-insensitivity, tag exclusion, distinct ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 20:55:32 +02:00
timothyandClaude Opus 4.8	b4ae0bde18	feat(434): distinct-values search endpoint (text fields, Lucene term enumeration) ... Adds GET /api/v1/search/fields/{name}/values?q=&limit= — the backend slice of the visual rule builder's facet-value typeahead (#434). Enumerates distinct Lucene term values for a text field via MultiFields.GetTerms + TermsEnum, filtered by a case-insensitive prefix, limit clamped to [1,50]. 404s when the field is absent from SearchFieldCatalog or is not type "text". ElasticSearchIndex (the optional external backend) throws NotSupportedException for this method — its text fields are analyzed, not keyword-mapped, so a terms aggregation isn't safe to guess at without verifying against a live cluster. Regenerated OpenAPI trio (v1.json, v1.d.ts, endpoint-index.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 20:43:27 +02:00
timothyandClaude Opus 4.8	ac7965dee4	feat(293): paginate GET /api/v1/search/all-items to cap DoS exposure ... The all-items endpoint fired ten index searches with limit:0 (every hit), so a broad authenticated query materialized the whole index into one response. Add optional pageNum/pageSize (clamped 1..1000; pageNum 0..2_000_000 so skip can't overflow int) and an additive per-kind Totals on the response; the SPA add-all flow now pages to completeness instead of a single unbounded fetch. - SearchController.SearchAllItems: clamp params (Logs §1 precedent), map Totals - QuerySearchIndexAllItemsHandler: skip=pageNum*pageSize, limit=pageSize, read SearchResult.TotalCount per kind - SearchResultAllItemsResponseModel: additive Totals (frozen-v1-safe) - web/src/api/search.ts: getSearchAllItems paging params + getAllSearchItemIds (pages until each kind hits its total; empty-page safety break) - tests: controller clamp/thread/totals, handler skip/limit/totals, SPA paging - docs: decisions.md 2026-07-18 (#293), api-conventions.md §5; regenerated OpenAPI Design: issue option (a) full pagination, operator-confirmed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-18 13:09:06 +02:00
timothy	ae22107d98	feat(176): GET /api/v1/search/fields endpoint + regenerated api artifacts	2026-07-18 02:41:50 +02:00
timothyandClaude Opus 4.8	ef2bd65c27	feat(api): #286 — mount the whole /api surface at /api/v1 ... Version every /api route to /api/v1 (251 controller routes + ~24 Location headers + the scanner callback URL + the Startup request-log literal), uniform across the machine API, auth, scanner and scripted-build surfaces. Add ApiVersionRewriteMiddleware: a legacy unversioned /api/* request is rewritten (NOT redirected) to /api/v1/* in-pipeline — method, body, auth headers and query survive — carrying RFC 8594 Deprecation/Sunset headers, so curl / the future MCP server / bookmarks keep working. An already- versioned path passes through; a future /api/v2 is never forced to v1. Standardize the route convention (leading-slash absolute route per method, no class-[Route] — except the two Scanner/Scripted controllers whose ~all actions share a parametrized {id} prefix), enforced by ApiRouteVersioningTests (^/api/v\d+/ over the whole Controllers.Api surface; browser-nav /auth/oidc/login is out of scope). Regenerate v1.json (160 paths, all /api/v1)/endpoint-index/v1.d.ts; sweep 945 SPA request literals + the test mocks (regex + positional URL parsers). /api/v1 is additive-only after freeze; the legacy-rewrite shim sunsets in ~2 releases (owner decision) with removal tracked as a Phase-3 follow-up. Docs: decisions.md 2026-07-13, api-conventions §1/§9, rest-api/spa-conventions/ blazor-route-parity/e2e-local/domain-model. fixes #286 refs #197 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-13 00:30:20 +02:00
timothyandClaude Opus 4.8	fecf16d3d2	feat(api): #288 wrap raw VMs, nullable-honest DTOs, search pageNum ... 24 #nullable enable flips across ErsatzTV.Core/Api; Collection/Schedule/ SmartCollection/Resolution VMs wrapped in ResponseModels (Version now header-only, SPA-verified); pageNum threaded into GET /api/search. Refs #288 #197 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-12 02:08:12 +02:00
timothy	819e751cab	feat(api): playlist add-items + search all-items endpoints (#208 #209) ... - POST /api/playlists/{id:int}/items wraps the existing AddItemsToPlaylist command (mirrors CollectionController.AddItems); controller pre-checks playlist existence for a real 404, and the handler now rejects adds to system (generated) playlists, matching the guard already applied to rename/delete/replace-items so the Blazor path gets the same protection. - GET /api/search/all-items wraps the existing QuerySearchIndexAllItems query, returning a new SearchResultAllItemsResponseModel (never expose the VM directly) so the SPA's shared "add all to collection/playlist" component can materialize ids before calling the add endpoints, same two-step flow Blazor's Search.razor already uses. - Show-detail DTO check: ShowDetailResponseModel already exposes libraryId, title, and mediaSourceKind (serialized as a string enum via the global StringEnumConverter) - no changes needed. Adds controller tests (route table + per-action) for both endpoints and regenerates the OpenAPI document, endpoint index, and SPA client types.	2026-07-09 23:56:50 +02:00
timothyandClaude Fable 5	57bb320af8	feat(api): REST endpoints for decos, deco groups, playout default deco (#144 S3 (#162)) ... Adds DecoController (groups + decos CRUD + full-state PUT), playlist and artist/multi-collection search picker wrappers, a PlaylistController for the break-content playlist cascade, and PUT /api/playouts/{id}/deco. Fixes the CreateDecoHandler missing deco-group existence check (FK 500 -> 422), matching the CreateBlock/CreateTemplate precedent. ReplaceDecoRequest.ToCommand rejects Merge on the non-mergeable sections (422). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-07 19:22:39 +02:00
timothyandClaude Fable 5	6b5bc8e29a	feat(api): REST endpoints for blocks, block groups, items, preview + scheduling search pickers ... Adds BlockController (block groups + blocks CRUD, items GET, full replace, non-persisting preview) mirroring ScheduleController, plus four scheduling search picker endpoints on SearchController (collections, television shows, television seasons, smart collections). Response DTOs in ErsatzTV.Core/Api/ Scheduling; request DTOs with ToCommand index auto-assignment. #144 S1 (#162) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-07 17:42:40 +02:00
timothyandClaude Fable 5	6c68c291a0	feat(api): browse all media kinds, grouped search, delete media-items (#141, #161) ... - Extend /api/library/browse to episodes, music videos, songs, other videos, images and remote streams (new LibraryBrowseMediaType values + hydrators); add optional Subtitle to LibraryBrowseItemResponseModel for leaf-item context - Add GET /api/search: grouped per-kind results reusing the browse query/shape; empty query -> 422 - Add DELETE /api/media-items: body { ids }, empty -> 422, success -> 204 - Tests: SearchController, MediaItemsController, security + OpenAPI contract entries - Regenerate openapi v1.json + web v1.d.ts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-07 15:23:07 +02:00