ersatztv/docs/decisions/records/api/search-field-values-sources.md

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics

key	title	status	since	supersedes	superseded-by	rule	signals	mechanics
api.search-field-values-sources	2026-07-26 — Facet-value typeahead, restated: every artist-bearing source is covered, and the JSON-column source is paged by ROW POSITION with no RESIDUAL SQL predicate (#578)	active	2026-07-26	api.search-field-values@2026-07-23	none	`GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for a narrow allow-list of catalog fields (never the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source (`title`, `show_title` only); `limit` clamped to `[1, 50]` (default 50). The FINAL filter, dedup and ordering applied to the response are ORDINAL (`OrdinalIgnoreCase` / `StringComparer.Ordinal`), never current-culture, because `UseRequestLocalization` makes the culture caller-controlled — scoped to the in-memory stages on purpose: a field sourced by a plain EF query is filtered and truncated by the DATABASE collation first (SQLite's `LOWER()` is ASCII-only), which ordinal semantics downstream cannot undo (ersatztv#668). A field whose values live in an EF **primitive collection** (one JSON array per row in a single column: `SongMetadata.Artists`, `SongMetadata.AlbumArtists`) is served, not 404'd, as bounded best-effort, and its rows are read by a keyset page carrying **NO RESIDUAL predicate** — `SELECT Id, <col> AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch`, no `LIKE`, no `LOWER`, not even `IS NOT NULL`. The cursor is itself a predicate, but a SEEKABLE one on the ordering key: it positions the scan and never discards a row. A RESIDUAL predicate discards rows the engine already produced, and `LIMIT` truncates only the survivors — so with one present it bounds the OUTPUT rather than the row count. All selectivity is in memory. The guarantee is scoped: **at most 20,000 LOGICAL rows returned/materialized and at most 10 round trips (11 for `artist`)** — NOT bounded physical work and NOT bounded bytes, because MySQL traverses deleted-but-unpurged index records and the `TEXT`/`longtext` payload width is unrestricted. The walk pages 2,000 rows at a time, stopping on the first of enough distinct matches, a short page, or the ceiling.	artist typeahead free-text credits, album_artist 404, artist suggestions missing music videos, SongMetadata.Artists, SongMetadata.AlbumArtists, MusicVideoArtist, EF primitive collection, PrimitiveCollection JSON column, SelectMany requires APPLY on SQLite, Pomelo primitive collections not enabled, LIMIT bounds output not work, seekable cursor vs residual predicate, cursor is a predicate too, MySQL purge lag traverses deleted index records, TEXT overflow pages, logical rows not physical work, keyspace is not rows, page by row position, density-independent paging, deleted rows leave Id gaps, ListValuedBatchRows, ListValuedMaxRowsRead, bounded best-effort facet values, OrdinalIgnoreCase vs InvariantCultureIgnoreCase folding, Accept-Language tr-TR dotless i, content_rating split, text field allow-list · paths: `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `ErsatzTV/Controllers/Api/SearchController.cs`, `ErsatzTV.Tests/Application/Search/GetSearchFieldValuesHandlerTests.cs`, `ErsatzTV.Tests/Application/Search/SearchFieldValuesQueryShapeTests.cs`, `web/src/api/search.ts` · issues: #578, #434, #176, #668, #669	`GetSearchFieldValuesHandler` (`GetSource`, `GetSongListValuedColumn`, `GetSongListValuedValues`, `ListValuedSql`, `ParseElements`, `FilterSortTake`); `SearchFieldValuesQueryShapeTests.List_Valued_Page_Query_Has_No_Predicate_Beyond_The_Keyset_Cursor`; api-conventions.md; spa-conventions.md §12

Supersedes api.search-field-values (#434). That record did not merely carry a stale implementation detail — it recorded a call: free-text music-video/song artist credits were "a known, intentionally-uncovered gap" and album_artist was unsupported. #578 reverses that call, so this is a supersession, not a line-edit. Everything #434 settled that still holds is restated here rather than left in the archive: enum fields ship their values inline on GET /api/v1/search/fields and need no lookup; text fields need a live one; the source is the database and never the search index; content_rating splits its compound "PG-13/TV-14" strings in memory; there is no result cache.

The three artist sources are three different problems

LuceneSearchIndex writes the artist field from three places, and only two are ordinary columns:

• ArtistMetadata.Title — a plain column. Already worked.
• MusicVideoArtist.Name — also a real entity table (MusicVideoMetadata.HasMany(m => m.Artists)), so the free-text music-video credits are directly SELECT DISTINCT-able. It just joins the existing server-side pipeline as a Concat, emitted as one bounded UNION ALL + LOWER(...) LIKE ... LIMIT on both providers.
• SongMetadata.Artists (and, for album_artist, AlbumArtists) — an IList<string> EF 9 maps as a primitive collection: no HasConversion anywhere, one JSON array per row in a single TEXT/longtext column, with no server-side projection at all. Verified against both providers: SQLite reports "Translating this query requires the SQL APPLY operation, which is not supported on SQLite", Pomelo MySQL 9.0.0 reports "Primitive collections support has not been enabled". Both failures are pinned by a test, so a provider upgrade that fixes them surfaces as a red rather than leaving a workaround in place forever.

The SQL predicate is gone, and that is the point

Three revisions tried to narrow the rows in SQL before filtering them in memory. All three were wrong, in three different ways, and the fourth was wrong too — the history is worth more than the code, so it is written out below under "four wrong quantities". The conclusion is short: there is no WHERE clause beyond the keyset cursor. No LIKE, no LOWER, not even IS NOT NULL.

That deletes an entire family of bugs along with the predicate. Gone with it: the JSON-escape reasoning (Édith is stored \u00C9dith, and SQL LOWER() folds the escape text rather than the codepoint it denotes, so a q=é pattern of \u00e9 never matched \u00C9); the "narrow only on the leading verbatim-ASCII run" rule and the exhaustive Unicode sweep that proved it sound; the ESCAPE '/' portability workaround; and the whole may-over-match-never-under-match invariant, which turned out to be conditional on something that was not true. In memory a string is just a string: element.StartsWith(query, StringComparison.OrdinalIgnoreCase).

Worth keeping one number from that history, because it is the reason the first bug survived review: the JSON-encoded pattern failed on three of nine pinned cases, not all nine — those where the query's casing differed from the stored casing, so the two escape texts diverged. When the casings agreed it worked. A bug that fires on some inputs and not others reads as "works" during a spot check.

Ordinal everywhere, because the culture is caller-controlled

UseRequestLocalization honours Accept-Language, so a caller can select tr-TR and turn q=I into ı. The old chain used ToLower() plus the default linguistic StartsWith(string), making the same library answer differently per caller. Comparison is now OrdinalIgnoreCase and ordering StringComparer.Ordinal throughout the in-memory stages, including the shared FilterSortTake that state, video_dynamic_range and content_rating also use. That is a deliberate change to shared behaviour, and not a cosmetic one: ordering happens before Take(limit), so changing the comparer can change which values survive, not merely their order. With "Zulu" and "apple", an empty q and limit=1, linguistic ordering yields "apple" and ordinal yields "Zulu". An earlier version of this record claimed the response sets were unchanged; that was false.

Scope this claim carefully — it is not endpoint-wide. A field sourced by a plain EF query runs the database's LOWER, DISTINCT, ORDER BY and LIMIT before any ordinal code executes, so the database has already decided which values survive. Store a genre "Éclair" on SQLite and ask for genre?q=é: SQLite's ASCII-only LOWER() drops it before the ordinal in-memory filter ever runs, and a case-insensitive collation's DISTINCT can likewise collapse values ordinal dedup would have kept. The endpoint description and this record's rule: therefore say "the final filter, dedup and ordering", not "matching is ordinal". That gap is now CLOSED by api.search-field-values-unicode-fold (ersatztv#668) — not by the client-side filtering guessed at here, but by a registered Unicode-correct SQL fold on a second, additive query taken only for non-ASCII queries on SQLite. The scoped wording above still stands as written: it describes what the EF stage itself does, which is unchanged.

Ordering is best-effort, and the code says so

Merging sources does not yield the exact first limit of the union. Each source truncates using its own ordering — the EF source by the database collation, the list source by primary key — and neither is the ordinal ordering the merge applies. The pair that actually demonstrates it is "Zulu" and "apple": ordinal puts every ASCII uppercase letter before every lowercase one, so the merge ranks "Zulu" first, while the case-insensitive database ordering ranks "apple" first — at limit=1 the response is ["apple"], not the ordinally-first "Zulu". Below the truncation points — the normal typeahead case — the result is exact. An earlier comment claimed exactness the code does not have; do not restore it. (An earlier version of this record used "Zulu"/"Éclair" as the example, where both orderings pick "Zulu" — it demonstrated nothing.)

What the bound bounds — four attempts, four wrong quantities

Read this before "optimizing" the query. Every one of these looked obviously correct when written, and each was caught only by someone constructing the adversarial case rather than reading the code.

#	Bounded	Why it wasn't a bound
1–2	the result — fixed LIMIT 1000 on pre-filtered rows	the pre-filter was deliberately allowed to over-match, so a widened pattern (any non-ASCII or JSON-escaped prefix collapses it to %"%) filled the budget with rows that could not match. 1,000 "zzz" songs, "éclair" at row 1,001, q=é → []
3	candidates returned — keyset paging + LIMIT	a query matching nothing must evaluate every eligible row before it can return an empty page, so the first empty page ended the walk having counted zero against the ceiling. Rows returned bounded, rows inspected unbounded
4	keyspace width — closed Id range per page	keyspace is not rows. Delete 20,000 historical rows, put one song at Id 20001, q=que → []. One row in the table, zero rows inspected. Capacity fell linearly with deletion ratio and no ratio was safe: one placed gap hides the next match
5	logical rows returned — keyset page by row position, cursor only, no residual predicate	— (physical work still unbounded; see below)

The through-line: LIMIT truncates what survives a RESIDUAL predicate. The distinction is not "predicate vs no predicate" — attempt 5's query still has Id > @AfterId. It is:

• a seekable predicate on the ordering key (the cursor) positions the scan and never discards a row, so LIMIT n yields n rows;
• a residual predicate (LIKE, LOWER, IS NOT NULL) throws away rows the engine already produced, so LIMIT bounds the survivors and says nothing about how many were produced.

Attempts 3 and 4 both kept selectivity in SQL and tried to add accounting around it. Attempt 5 drops the residual predicate and keeps only the cursor, so the accounting becomes trivial:

SELECT Id, Artists AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch

ListValuedBatchRows = 2000, ListValuedMaxRowsRead = 20000. The walk stops on the first of: enough distinct matches for limit, a short page (with no residual predicate that can only mean exhaustion — it can never mean "this stretch matched nothing", which is exactly why the residual predicate had to go), or the ceiling. Round trips: at most 10 for album_artist, at most 11 for artist, which also runs one EF query for its entity/music-video half. SearchFieldValuesQueryShapeTests pins the SQL string exactly and asserts the absence of LIKE, LOWER and IS NOT NULL, so a reviewer reintroducing "just a cheap filter" fails a test instead of silently unbounding the walk.

Exactly what is bounded — and what is NOT

State this precisely, because an earlier version of this record claimed more and the overclaim is more dangerous than the code ever was. What holds:

• at most ListValuedMaxRowsRead logical rows returned and materialized per request, and
• at most 10 (or 11) round trips.

That is the whole guarantee. It is what makes the walk terminate and what caps the number of rows and round trips. Explicitly retracted, having been asserted here in earlier revisions:

• "LIMIT n reads exactly n index entries" — false on MySQL. Deleted clustered-index records survive until purge runs, and a range scan still traverses them. Hold an old InnoDB snapshot open, delete a million early SongMetadata rows, and query from a newer snapshot with purge blocked: returning 2,000 visible rows can touch far more index records. Deletion history therefore still affects physical work — the very thing attempt 4's failure was supposed to have made irrelevant. Attempt 5 fixes the logical dependence on Id distribution; it does not make physical work independent of deletion history.
• bounded physical work / bounded I/O — row width is unbounded. Artists/AlbumArtists are unrestricted TEXT/longtext, and both SQLite and InnoDB spill large payloads to overflow pages, so a row count implies neither a byte count nor a page-read count.
• "caps what this process holds in memory" — the same overclaim one level down, and it survived the first retraction. A row count bounds neither bytes buffered nor set size: payload width is unrestricted, and one JSON array can contain arbitrarily many strings, every one of which may enter the in-memory distinct set.

Nor can the query-shape test carry more than it does: it pins the SQL string. It cannot pin an execution plan, MVCC visibility work, or payload I/O — and on MySQL, using the index to satisfy ORDER BY is an optimizer choice, not a SQL semantic.

The cost, measured

No server-side narrowing means rows are transferred that will be discarded. This is ONE data point on ONE library, not a general figure — see the row-width caveat above: these numbers hold for a library whose artist credits average ~20 bytes of JSON, and a library with long credit lists would transfer proportionally more for the same row count. Measured on a seeded 20,000-song library (in-memory SQLite, so the wall times are a floor, not a production figure):

case	rows read	round trips	payload	wall
worst case — no match, full walk	20,000	10	391.9 KiB (avg 20.1 B/row)	119 ms SQL / ~40 ms warm end-to-end
empty q (fills limit on page 1)	2,000	1	~39 KiB	~60 ms
dense prefix (rad)	2,000	1	~39 KiB	~38 ms
non-ASCII prefix (beyoncé)	2,000	1	~39 KiB	~39 ms

Judged acceptable for this shape of library: the worst case is a debounced typeahead keystroke that matches nothing, at ~392 KiB and tens of milliseconds against a local SQLite file. Dense queries — including the empty q the combobox opens with — stop on the first page. Re-measure rather than extrapolate if credit lists are long or the provider is MySQL over a network. If it ever becomes unacceptable, do not reintroduce selectivity; that is the trap this record exists to document. Go to #669.

Accepted losses

A match past row 20,000 is not found — 20,000 filler rows then "éclair" at 20,001 returns [], and a test pins exactly that rather than pretending otherwise. That is the documented bounded-best-effort contract, and unlike attempts 1–4 it now depends only on row count, not on prefix shape, deletion history or Id distribution.

Follow-up: a normalized SongArtist join table (the shape MusicVideoArtist already has) makes the predicate seekable, so there is nothing left to bound and nothing to transfer. Cost: a dual-provider schema migration plus data backfill, changes to every scanner write path populating SongMetadata.Artists, changes to the Lucene indexer, and two representations of the same fact free to drift. Tracked as ersatztv#669; rejected for #578 on blast radius, not on merit.

album_artist 404 → 200 is additive

Nothing consumes the 404 as a signal: the SPA's getSearchFieldValues (web/src/api/search.ts) treats any non-200 as "no suggestions, fall back to a free-text input", which it will now do less often. Per api.versioning-v1, widening which fields return values adds capability without removing any, so no /api/v2.

Known limitation inherited, not introduced

RESOLVED — see api.search-field-values-unicode-fold (ersatztv#668, 2026-07-27). As written for #578 this said: the EF-sourced fields (genre, studio, artist's entity half, …) still prefix-match through SQL LOWER(), which on SQLite is ASCII-only, so a stored Édith was unreachable for those fields. That predated #578 and was unchanged by it. It is now fixed — and NOT by the client-side filtering this section anticipated, which would have reintroduced the very scan #578 bounded. The surrounding scoped-ordinal wording is still load-bearing and must not be "tidied" into a broader claim: the EF stage's own behaviour is unchanged, and the defect was SQLite-only and one-sided.