---
key: api.search-field-values-unicode-fold
title: '2026-07-27 — Facet-value typeahead reaches accented values: a registered Unicode fold on the SQLite non-ASCII branch, not a bounded walk (#668)'
status: active
since: '2026-07-27'
supersedes: none
superseded-by: none
rule: 'The EF-sourced facet fields (`genre`, `show_genre`, `studio`, `director`, `writer`, `actor`, `tag`, `network`, `collection`, `video_codec`, `album`, and `artist`''s entity half) reach stored values whose prefix carries an uppercase non-ASCII character, on BOTH providers, with no row budget and no accepted loss. The defect was SQLite-only and ONE-SIDED: SQLite''s `LOWER()` folds ASCII only (`lower(''Édith'')` is `''Édith''` unchanged), so the predicate UNDER-matched, which no later stage can repair. MySQL was already correct — its `LOWER()` is Unicode-aware, so `LOWER(''Édith'')` really is `''édith''` and the existing predicate reaches the row unaided. The fix is a SECOND, ADDITIVE query taken only when `isSqlite && q contains a non-ASCII character`: raw Dapper SQL `SELECT DISTINCT <col> AS Value FROM <table> WHERE [<discriminator> AND] etv_upper(<col>) LIKE @Pattern ESCAPE ''\'' ORDER BY <col> LIMIT @Limit`, where `etv_upper` is a `SqliteConnection.CreateFunction` scalar implementing `ToUpperInvariant`. Every other case — all-ASCII `q`, and MySQL for all `q` — runs today''s EF query BYTE-IDENTICALLY. Keeping selectivity in SQL here is NOT the refuted family from `api.search-field-values-sources`: those four attempts bounded a walk around a predicate that could not be made correct over JSON escape text, whereas this is a correct fold on a plain column in an ordinary `LIMIT`ed query. It narrows that record''s "Known limitation inherited, not introduced" clause; everything else it settles still holds.'
signals: 'accented facet values missing, Édith not suggested, SQLite LOWER is ASCII only, etv_upper, CreateFunction custom scalar, ToUpperInvariant fold, OrdinalIgnoreCase is not invariant-upper, U+017F long s upper-folds to S, U+212A Kelvin sign, utf8mb4_0900_ai_ci accent insensitive, MySQL LOWER is unicode aware, over-match harmless under-match not, ESCAPE clause raw SQL LIKE wildcards, EF null semantics ExternalTypeId, RegisterUnicodeCaseFunctions provider static, non-sargable LOWER LIKE full table scan · paths: `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `ErsatzTV.Infrastructure.Sqlite/Data/SqliteUnicodeFunctions.cs`, `ErsatzTV.Infrastructure/Data/TvContext.cs`, `ErsatzTV/Startup.cs`, `ErsatzTV.Scanner/Program.cs` · issues: #668, #578, #434, #669'
mechanics: '`GetSearchFieldValuesHandler` (`ContainsNonAscii`, `IsSqlite`, `EscapeLikePrefix`, `UnicodeFoldSql`, `GetUnicodeFoldSources`, `GetUnicodeFoldedValues`, `UpperFunction`); `SqliteUnicodeFunctions.Register`; `TvContext.RegisterUnicodeCaseFunctions`; `GetSearchFieldValuesHandlerTests.Unicode_Fold_Agrees_With_The_Ordinal_Filter`; `SearchFieldValuesQueryShapeTests.Unicode_Fold_Function_Name_Matches_The_Registration`; `ProviderStaticsWiringTests`'
---

Narrows `api.search-field-values-sources` (#578), which deferred this gap; the rest of #578 stands.

## The defect was one-sided, and the issue described it wrongly

The handler lowercases `q` with `ToLowerInvariant` **before** SQL, so both casings produce one pattern.
A stored **lowercase** accented value was therefore always reachable from either casing; only one whose
prefix carries an **uppercase** non-ASCII character was lost. ersatztv#668's body claimed `q=É` failed
against a stored `édith`; false, and a test pins the passing case beside the fixed one.

## MySQL was never broken, for a reason worth recording

Verified on a live MySQL 8.4: `LOWER('Édith')` is `édith`, so the existing predicate reaches the row.
**Measure the query the CODE runs, not one you type.** With a LITERAL pattern `LOWER(name) LIKE 'é%'`
also matches `Edith` (the column is accent-insensitive `utf8mb4_0900_ai_ci`), and an earlier revision of
this record concluded from exactly that probe that MySQL over-matches and the ordinal filter corrects it.
It does not: through EF the driver binds the pattern with a BINARY collation, so the executed comparison
is accent-SENSITIVE and returns `Édith` alone — a driver-contingent fact, not a law. MySQL's correctness
rests on Unicode-aware `LOWER()`, not on the collation.

## Why a fold, and not the #578 walk

Reusing #578's shape — drop SQL selectivity, keyset-walk, filter in memory — answers the wrong question.
That walk is best-effort at 20,000 rows; `Genre` and `Actor` carry one row per media item, so a large
library exceeds the budget and `Édith` stays unreachable — the bug restated. #578 accepts that contract
for `SongMetadata.Artists` because server-side projection is **impossible** there; these are plain
columns, where it is merely inconvenient.

The cost objection to a managed per-row fold is weak: `LOWER(v) LIKE` is non-sargable and **no index on
any of these `Name` columns exists** (every index is on the foreign key), so this swaps a native per-row
call for a managed one on a scan that already happens — and only on the non-ASCII branch.

## The correctness property is containment, not equality

The SQL stage may over-match freely; it must never under-match. `ToUpperInvariant` satisfies that
because **`OrdinalIgnoreCase` equality is a strict subset of invariant-uppercase equality**.

Do not restate this as "`OrdinalIgnoreCase` IS invariant-uppercase-then-ordinal". It is not, and the gap
is measurable: `char.ToUpperInvariant('ſ')` (U+017F) is `'S'`, yet
`"ſweet".StartsWith("S", OrdinalIgnoreCase)` is **false**. The fold returns that row and the filter drops
it — the harmless direction. An earlier draft justified the fold by claiming the opposite;
`Fold_LongS_IsNotOrdinalEqualToS` pins the truth.

That same fact makes the all-ASCII fast path sound: no non-ASCII codepoint is `OrdinalIgnoreCase`-equal to printable ASCII (#578's sweep found 0), so an ASCII query only ever ordinal-matches an ASCII prefix.

## Three traps, each guarded by a test and explained at its call site

Raw SQL gets none of EF's LIKE escaping (`EscapeLikePrefix`, backslash first, explicit `ESCAPE`).
Discriminators must mirror EF's NULL semantics — `t.ExternalTypeId != X` INCLUDES a NULL-typed row,
where plain SQL `<>` drops it. Registration is per-connection and lives at the call site, not in a
`DbConnectionInterceptor`: Dapper opens a closed connection itself and a direct ADO open raises no EF
interceptor, so that seam would miss exactly this query.

## Residuals, stated rather than glossed

**Crowding**: a SQL `LIMIT` can fill with rows the ordinal filter then discards, under-DELIVERING the
count (never a wrong value). Not reachable on MySQL under the CURRENT driver behaviour above (a ci-collated
pattern would restore it); the SQLite fold has it when limit-many values are upper-equal but ordinal-unequal (
`ſ`/`K`/`İ` class), so "no accepted loss" means no unreachable VALUE, not a guaranteed count. An
over-fetch was rejected (it perturbs the pinned `"apple"`/`"Zulu"` examples). **Ordering stays
best-effort** per #578.