timothy
f2d9c0dc8e
Final sweep confirmed the retracted MySQL over-match claim survives in no file
on the branch (only in two immutable commit messages, which stay -- rewriting
history would invalidate every sha-bound review verdict). Three nits remained.
- The fixture's class docstring said the on-MySQL claim "rests on the server's
collation", which is the one thing the decision record says it does NOT rest
on. It rests on Unicode-aware LOWER(); the executed comparison bypasses the
collation entirely. Reworded.
- The record's `rule:` enumerated the covered fields but omitted show_genre,
which GetSource and the fold both handle ("genre" or "show_genre"). Added.
- My own #688 write-up was wrong twice: the 60-line ceiling warning is
NON-blocking by design, and the calibration pytest reds at >=61, not >=60 --
main's p90 is 59, so a 60-line record makes p90 == ceiling and PASSES. The
bullet even contradicted itself, since the next sentence relies on 60 passing.
Corrected in the PR body and in the issue.
Decisions-Edit: yes
7.0 KiB
key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
| key | title | status | since | supersedes | superseded-by | rule | signals | mechanics |
|---|---|---|---|---|---|---|---|---|
| api.search-field-values-unicode-fold | 2026-07-27 — Facet-value typeahead reaches accented values: a registered Unicode fold on the SQLite non-ASCII branch, not a bounded walk (#668) | active | 2026-07-27 | none | none | The EF-sourced facet fields (`genre`, `show_genre`, `studio`, `director`, `writer`, `actor`, `tag`, `network`, `collection`, `video_codec`, `album`, and `artist`'s entity half) reach stored values whose prefix carries an uppercase non-ASCII character, on BOTH providers, with no row budget and no accepted loss. The defect was SQLite-only and ONE-SIDED: SQLite's `LOWER()` folds ASCII only (`lower('Édith')` is `'Édith'` unchanged), so the predicate UNDER-matched, which no later stage can repair. MySQL was already correct — its `LOWER()` is Unicode-aware, so `LOWER('Édith')` really is `'édith'` and the existing predicate reaches the row unaided. The fix is a SECOND, ADDITIVE query taken only when `isSqlite && q contains a non-ASCII character`: raw Dapper SQL `SELECT DISTINCT <col> AS Value FROM <table> WHERE [<discriminator> AND] etv_upper(<col>) LIKE @Pattern ESCAPE '' ORDER BY <col> LIMIT @Limit`, where `etv_upper` is a `SqliteConnection.CreateFunction` scalar implementing `ToUpperInvariant`. Every other case — all-ASCII `q`, and MySQL for all `q` — runs today's EF query BYTE-IDENTICALLY. Keeping selectivity in SQL here is NOT the refuted family from `api.search-field-values-sources`: those four attempts bounded a walk around a predicate that could not be made correct over JSON escape text, whereas this is a correct fold on a plain column in an ordinary `LIMIT`ed query. It narrows that record's "Known limitation inherited, not introduced" clause; everything else it settles still holds. | accented facet values missing, Édith not suggested, SQLite LOWER is ASCII only, etv_upper, CreateFunction custom scalar, ToUpperInvariant fold, OrdinalIgnoreCase is not invariant-upper, U+017F long s upper-folds to S, U+212A Kelvin sign, utf8mb4_0900_ai_ci accent insensitive, MySQL LOWER is unicode aware, over-match harmless under-match not, ESCAPE clause raw SQL LIKE wildcards, EF null semantics ExternalTypeId, RegisterUnicodeCaseFunctions provider static, non-sargable LOWER LIKE full table scan · paths: `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `ErsatzTV.Infrastructure.Sqlite/Data/SqliteUnicodeFunctions.cs`, `ErsatzTV.Infrastructure/Data/TvContext.cs`, `ErsatzTV/Startup.cs`, `ErsatzTV.Scanner/Program.cs` · issues: #668, #578, #434, #669 | `GetSearchFieldValuesHandler` (`ContainsNonAscii`, `IsSqlite`, `EscapeLikePrefix`, `UnicodeFoldSql`, `GetUnicodeFoldSources`, `GetUnicodeFoldedValues`, `UpperFunction`); `SqliteUnicodeFunctions.Register`; `TvContext.RegisterUnicodeCaseFunctions`; `GetSearchFieldValuesHandlerTests.Unicode_Fold_Agrees_With_The_Ordinal_Filter`; `SearchFieldValuesQueryShapeTests.Unicode_Fold_Function_Name_Matches_The_Registration`; `ProviderStaticsWiringTests` |
Narrows api.search-field-values-sources (#578), which deferred this gap; the rest of #578 stands.
The defect was one-sided, and the issue described it wrongly
The handler lowercases q with ToLowerInvariant before SQL, so both casings produce one pattern.
A stored lowercase accented value was therefore always reachable from either casing; only one whose
prefix carries an uppercase non-ASCII character was lost. ersatztv#668's body claimed q=É failed
against a stored édith; false, and a test pins the passing case beside the fixed one.
MySQL was never broken, for a reason worth recording
Verified on a live MySQL 8.4: LOWER('Édith') is édith, so the existing predicate reaches the row.
Measure the query the CODE runs, not one you type. With a LITERAL pattern LOWER(name) LIKE 'é%'
also matches Edith (the column is accent-insensitive utf8mb4_0900_ai_ci), and an earlier revision of
this record concluded from exactly that probe that MySQL over-matches and the ordinal filter corrects it.
It does not: through EF the driver binds the pattern with a BINARY collation, so the executed comparison
is accent-SENSITIVE and returns Édith alone — a driver-contingent fact, not a law. MySQL's correctness
rests on Unicode-aware LOWER(), not on the collation.
Why a fold, and not the #578 walk
Reusing #578's shape — drop SQL selectivity, keyset-walk, filter in memory — answers the wrong question.
That walk is best-effort at 20,000 rows; Genre and Actor carry one row per media item, so a large
library exceeds the budget and Édith stays unreachable — the bug restated. #578 accepts that contract
for SongMetadata.Artists because server-side projection is impossible there; these are plain
columns, where it is merely inconvenient.
The cost objection to a managed per-row fold is weak: LOWER(v) LIKE is non-sargable and no index on
any of these Name columns exists (every index is on the foreign key), so this swaps a native per-row
call for a managed one on a scan that already happens — and only on the non-ASCII branch.
The correctness property is containment, not equality
The SQL stage may over-match freely; it must never under-match. ToUpperInvariant satisfies that
because OrdinalIgnoreCase equality is a strict subset of invariant-uppercase equality.
Do not restate this as "OrdinalIgnoreCase IS invariant-uppercase-then-ordinal". It is not, and the gap
is measurable: char.ToUpperInvariant('ſ') (U+017F) is 'S', yet
"ſweet".StartsWith("S", OrdinalIgnoreCase) is false. The fold returns that row and the filter drops
it — the harmless direction. An earlier draft justified the fold by claiming the opposite;
Fold_LongS_IsNotOrdinalEqualToS pins the truth.
That same fact makes the all-ASCII fast path sound: no non-ASCII codepoint is OrdinalIgnoreCase-equal to printable ASCII (#578's sweep found 0), so an ASCII query only ever ordinal-matches an ASCII prefix.
Three traps, each guarded by a test and explained at its call site
Raw SQL gets none of EF's LIKE escaping (EscapeLikePrefix, backslash first, explicit ESCAPE).
Discriminators must mirror EF's NULL semantics — t.ExternalTypeId != X INCLUDES a NULL-typed row,
where plain SQL <> drops it. Registration is per-connection and lives at the call site, not in a
DbConnectionInterceptor: Dapper opens a closed connection itself and a direct ADO open raises no EF
interceptor, so that seam would miss exactly this query.
Residuals, stated rather than glossed
Crowding: a SQL LIMIT can fill with rows the ordinal filter then discards, under-DELIVERING the
count (never a wrong value). Not reachable on MySQL under the CURRENT driver behaviour above (a ci-collated
pattern would restore it); the SQLite fold has it when limit-many values are upper-equal but ordinal-unequal (
ſ/K/İ class), so "no accepted loss" means no unreachable VALUE, not a guaranteed count. An
over-fetch was rejected (it perturbs the pinned "apple"/"Zulu" examples). Ordering stays
best-effort per #578.