fix(578): the LIKE prefilter under-matched every accented artist; make the superset provable
Review of 1b78dc9e found the pre-filter's correctness claim was false, and the claim was in the decision record as well as the code. F1 (high). The pattern JSON-encoded the whole query prefix on the reasoning that the stored text escapes non-ASCII, so encoding the prefix the same way would line up. It does not: SQL LOWER() lowercases the *escape text* (`É` -> `é`); it cannot case-fold the codepoint that escape denotes. So `q=é` built `%"é%`, the stored `Édith Piaf` never matched, and the row was discarded before the in-memory filter could accept it. Every accented artist — Beyoncé, Björk, Sigur Rós, Édith Piaf — was silently unsuggestable, which in a music library is the common case. The invariant that was missing, now stated in the code: the SQL pre-filter is an OPTIMIZATION. It may over-match; it must never under-match. Correctness lives in the in-memory filter. So the pattern now narrows only on the leading run of characters the JSON writer stores verbatim and stops at the first character it cannot prove — `q=Beyoncé` still narrows on `beyonc`, `q=é` narrows on nothing and leans on the row cap. Soundness rests on two facts now asserted by exhaustive computation rather than argued: no non-ASCII codepoint in U+0080..U+10FFFF OrdinalIgnoreCase-equals a printable ASCII character (false for InvariantCultureIgnoreCase, which folds ~190 — the choice of Ordinal is load-bearing), and the exact set of ASCII the encoder escapes. F1b. `UseRequestLocalization` honours Accept-Language, so the culture was caller-controlled and `ToLower()` plus the default linguistic `StartsWith(string)` let a header change the answer. Comparison is now OrdinalIgnoreCase and ordering StringComparer.Ordinal throughout — including the shared FilterSortTake that state/video_dynamic_range/content_rating also use. Sets unchanged, order now ordinal rather than culture-dependent. F2. The merge comment asserted an exactness the code does not have: sources truncate by their own ordering (DB collation / primary key), not the merge's, so a dropped value can outrank a survivor. Comment and record now say best-effort, exact only below the truncation points. F3/F4. The cap now rides `ORDER BY Id` rather than the JSON column: MySQL sorts TEXT by only max_sort_length bytes, so the old ordering was not deterministic there, and sorting the whole matching set was avoidable work. What the cap still does NOT bound is the scan — a leading-wildcard LIKE cannot seek an index — so that cost is now documented as accepted, with a normalized `SongArtist` table named as the follow-up candidate rather than left implicit. Every clause above is covered by a test verified to FAIL when that clause is mutated (old pattern builder: 5 red; culture chain: 3 red; cap=3 / cap=limit / ORDER BY json / no cap: red each). F5. Converted to a proper supersession. The old record did not merely hold a stale fact — it recorded song/music-video credits as an "intentionally-uncovered gap" and album_artist as unsupported, and this reverses that call, which `docs.decision-lifecycle` says is never a line-edit. `api.search-field-values` is archived with its original prose restored, and `api.search-field-values-sources` replaces it carrying the whole endpoint contract.
This commit is contained in:
@@ -18,11 +18,13 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
/// Row cap for the list-valued (JSON-array) columns on <c>SongMetadata</c>. The SQL pre-filter over the
|
||||
/// raw JSON is a deliberate superset (a matching row is fetched whole, including its non-matching
|
||||
/// elements), so the number of rows it can return has to be bounded independently of <c>limit</c> —
|
||||
/// otherwise an empty <c>q</c> degenerates into materializing the entire table. Combined with the
|
||||
/// <c>ORDER BY</c> in <see cref="ListValuedSql" />, the truncation point is deterministic rather than
|
||||
/// whatever the storage engine happened to emit first.
|
||||
/// otherwise an empty <c>q</c> degenerates into materializing the entire table. The cap is applied on
|
||||
/// <c>ORDER BY Id</c>, which is a unique integer primary key on both providers: unlike ordering on the
|
||||
/// JSON column itself, that makes the truncation point genuinely deterministic (MySQL sorts <c>TEXT</c>
|
||||
/// using only the first <c>max_sort_length</c> bytes, so long JSON rows sharing a prefix would tie
|
||||
/// arbitrarily) and lets the engine walk the primary key instead of sorting the whole matching set.
|
||||
/// </summary>
|
||||
private const int ListValuedRowCap = 1000;
|
||||
internal const int ListValuedRowCap = 1000;
|
||||
|
||||
/// <summary>
|
||||
/// Escape character for the raw-JSON <c>LIKE</c> pre-filter. Deliberately NOT a backslash: the stored
|
||||
@@ -32,6 +34,14 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
/// </summary>
|
||||
private const char LikeEscapeChar = '/';
|
||||
|
||||
/// <summary>
|
||||
/// The ASCII characters the JSON writer escapes rather than emitting verbatim (<c>&</c> and friends
|
||||
/// become <c>&</c>). A prefix character in this set cannot be matched literally against the stored
|
||||
/// text, so it terminates the narrowing run in <see cref="JsonElementPrefixPattern" />.
|
||||
/// <c>SearchFieldValuesPrefilterSupersetTests</c> pins this set against the encoder itself.
|
||||
/// </summary>
|
||||
private const string JsonEscapedAscii = "\"&'+<>\\`";
|
||||
|
||||
public async Task<Option<SearchFieldValuesResponseModel>> Handle(
|
||||
GetSearchFieldValues request,
|
||||
CancellationToken cancellationToken)
|
||||
@@ -45,17 +55,22 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
}
|
||||
|
||||
int limit = request.Limit <= 0 ? DefaultLimit : Math.Clamp(request.Limit, 1, MaxLimit);
|
||||
string qLower = (request.Query ?? string.Empty).ToLower();
|
||||
string query = request.Query ?? string.Empty;
|
||||
|
||||
// Invariant, not current-culture: UseRequestLocalization honours Accept-Language, so a caller can select
|
||||
// tr-TR and turn `q=I` into `ı` — which then matches nothing a Turkish-dotless-i-free library contains.
|
||||
// This feeds the EF-translated filter, which has no StringComparison overload EF can translate.
|
||||
string qLower = query.ToLowerInvariant();
|
||||
|
||||
// in-memory special cases (no DB query needed)
|
||||
switch (request.Name)
|
||||
{
|
||||
case "state":
|
||||
return new SearchFieldValuesResponseModel(
|
||||
FilterSortTake(Enum.GetNames<MediaItemState>(), qLower, limit));
|
||||
FilterSortTake(Enum.GetNames<MediaItemState>(), query, limit));
|
||||
case "video_dynamic_range":
|
||||
return new SearchFieldValuesResponseModel(
|
||||
FilterSortTake(["hdr", "sdr"], qLower, limit));
|
||||
FilterSortTake(["hdr", "sdr"], query, limit));
|
||||
}
|
||||
|
||||
await using TvContext dbContext = await dbContextFactory.CreateDbContextAsync(cancellationToken);
|
||||
@@ -63,7 +78,7 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
if (request.Name == "content_rating")
|
||||
{
|
||||
return new SearchFieldValuesResponseModel(
|
||||
await GetContentRatingValues(dbContext, qLower, limit, cancellationToken));
|
||||
await GetContentRatingValues(dbContext, query, limit, cancellationToken));
|
||||
}
|
||||
|
||||
IQueryable<string> source = GetSource(dbContext, request.Name);
|
||||
@@ -88,16 +103,18 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
|
||||
if (listColumn is not null)
|
||||
{
|
||||
values.AddRange(
|
||||
await GetSongListValuedValues(dbContext, listColumn, request.Query, qLower, cancellationToken));
|
||||
values.AddRange(await GetSongListValuedValues(dbContext, listColumn, query, cancellationToken));
|
||||
}
|
||||
|
||||
// Each contributing source already returned its own prefix-filtered, ascending first-`limit` values, so
|
||||
// sorting the union and taking `limit` again yields exactly the first `limit` of the combined set.
|
||||
return new SearchFieldValuesResponseModel(FilterSortTake(values.Distinct(), qLower, limit));
|
||||
// ORDERING IS BEST-EFFORT, NOT EXACT. Each source truncates using its own ordering — the EF source by the
|
||||
// database collation (SQLite's NOCASE/BINARY is ASCII-only), the list source by primary key — and neither
|
||||
// is the ordinal ordering applied here. So when a source actually truncates, a value it dropped may have
|
||||
// outranked one that survived: with "Zulu" and "Éclair" and limit=1 the EF source keeps "Zulu" and the
|
||||
// merge never sees "Éclair". Below the truncation points (the normal typeahead case) the result is exact.
|
||||
return new SearchFieldValuesResponseModel(FilterSortTake(values.Distinct(StringComparer.Ordinal), query, limit));
|
||||
}
|
||||
|
||||
private static IQueryable<string> GetSource(TvContext dbContext, string name) => name switch
|
||||
internal static IQueryable<string> GetSource(TvContext dbContext, string name) => name switch
|
||||
{
|
||||
"genre" or "show_genre" => dbContext.Set<Genre>().Select(g => g.Name),
|
||||
"studio" => dbContext.Set<Studio>().Select(s => s.Name),
|
||||
@@ -148,16 +165,17 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Instead: pre-filter on the raw JSON text in SQL (a superset — a row is matched, not an element,
|
||||
/// so its other elements come along), cap the rows, then split and exact-filter in memory. When the
|
||||
/// library has more matching rows than <see cref="ListValuedRowCap" /> the suggestion list for these
|
||||
/// fields is a deterministic prefix of the matches rather than the complete set.
|
||||
/// so its other elements come along), cap the rows by primary key, then split and exact-filter in
|
||||
/// memory. Correctness lives in the in-memory filter; the SQL pattern is only an optimization and is
|
||||
/// allowed to over-match but never to under-match. When the library has more matching rows than
|
||||
/// <see cref="ListValuedRowCap" /> the suggestion list for these fields is the lowest-id prefix of
|
||||
/// the matches rather than the complete set.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static async Task<List<string>> GetSongListValuedValues(
|
||||
TvContext dbContext,
|
||||
string column,
|
||||
string query,
|
||||
string qLower,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
IEnumerable<string> rows = await dbContext.Connection.QueryAsync<string>(
|
||||
@@ -186,7 +204,7 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
|
||||
foreach (string element in elements ?? [])
|
||||
{
|
||||
if (!string.IsNullOrEmpty(element) && element.ToLower().StartsWith(qLower))
|
||||
if (!string.IsNullOrEmpty(element) && element.StartsWith(query, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
values.Add(element);
|
||||
}
|
||||
@@ -199,45 +217,66 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
/// <summary>
|
||||
/// The bounded pre-filter query. <c>LOWER(...) LIKE ... ESCAPE</c>, <c>ORDER BY</c> and <c>LIMIT</c> are
|
||||
/// all portable between SQLite and MySQL, and the identifiers are unquoted so neither provider's
|
||||
/// quoting dialect is baked in.
|
||||
/// quoting dialect is baked in. The cap rides <c>ORDER BY Id</c> — a unique integer key both providers
|
||||
/// can walk — rather than the JSON column, so the truncation point is well-defined and no full sort of
|
||||
/// the matching set is required.
|
||||
/// </summary>
|
||||
private static string ListValuedSql(string column) =>
|
||||
internal static string ListValuedSql(string column) =>
|
||||
$"SELECT {column} FROM SongMetadata WHERE {column} IS NOT NULL " +
|
||||
$"AND LOWER({column}) LIKE @Pattern ESCAPE '{LikeEscapeChar}' " +
|
||||
$"ORDER BY {column} LIMIT @Cap";
|
||||
"ORDER BY Id LIMIT @Cap";
|
||||
|
||||
/// <summary>
|
||||
/// Builds the raw-JSON <c>LIKE</c> pattern that matches a row containing an element starting with
|
||||
/// <paramref name="query" />. Anchoring on the opening quote (<c>%"prefix%</c>) keeps the superset
|
||||
/// tight: it matches element starts, not substrings anywhere in the row.
|
||||
/// Builds the raw-JSON <c>LIKE</c> pattern used to narrow the rows fetched for a list-valued column.
|
||||
/// <para>
|
||||
/// The prefix is JSON-encoded first, because that is how it is stored — EF writes the array with the
|
||||
/// default <c>JavaScriptEncoder</c>, so <c>Beyoncé</c> lands on disk as <c>Beyoncé</c> and a
|
||||
/// literal quote as <c>"</c>. Encoding also makes the pattern pure ASCII, which is why this
|
||||
/// match means the same thing on both providers: SQLite's <c>lower()</c> is ASCII-only and MySQL's
|
||||
/// is Unicode-aware, but on ASCII input they agree. Lowercasing the pattern here (rather than
|
||||
/// relying on <c>LIKE</c>'s own case rules) is what keeps the comparison correct under a
|
||||
/// case-sensitive MySQL column collation as well as a case-insensitive one; under a
|
||||
/// case-insensitive collation it simply widens the superset, which the in-memory exact filter
|
||||
/// then narrows again.
|
||||
/// <b>This is an optimization with one hard obligation: it must never under-match.</b> Every row
|
||||
/// holding an element that starts with <paramref name="query" /> under
|
||||
/// <see cref="StringComparison.OrdinalIgnoreCase" /> — the comparison
|
||||
/// <see cref="GetSongListValuedValues" /> then applies — has to survive it. Over-matching is free;
|
||||
/// the in-memory filter discards the extras.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// So the pattern narrows on the <b>leading run of ASCII characters the JSON writer stores
|
||||
/// verbatim</b>, and stops at the first character it cannot reason about — a non-ASCII character, or
|
||||
/// one of <see cref="JsonEscapedAscii" />. A query starting with such a character narrows to the bare
|
||||
/// element-opening anchor <c>%"%</c> and leans entirely on the row cap and the in-memory filter.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Why that is sound, and why an earlier "JSON-encode the whole prefix" version was NOT: the stored
|
||||
/// text escapes non-ASCII, so <c>Édith</c> is on disk as <c>Édith</c>. SQL <c>LOWER()</c>
|
||||
/// lowercases the <i>escape text</i>, giving <c>É</c> — it cannot case-fold the codepoint that
|
||||
/// escape denotes, so a <c>q=é</c> pattern of <c>é</c> never matched and every accented artist
|
||||
/// was silently unsuggestable. Restricting the run to verbatim ASCII removes the escape from the
|
||||
/// comparison entirely. And for a run character <c>c</c>, any element character that
|
||||
/// OrdinalIgnoreCase-equals <c>c</c> is itself ASCII — no non-ASCII codepoint in all of Unicode
|
||||
/// OrdinalIgnoreCase-equals a printable ASCII character, which
|
||||
/// <c>SearchFieldValuesPrefilterSupersetTests</c> proves by exhaustive sweep rather than assertion.
|
||||
/// (That is false for <c>InvariantCultureIgnoreCase</c>, which has 190 such codepoints — the choice
|
||||
/// of Ordinal is load-bearing, not stylistic.) ASCII case-folding is exactly what both providers'
|
||||
/// <c>LOWER()</c> implements, and lowercasing the pattern here rather than trusting <c>LIKE</c>'s own
|
||||
/// case rules keeps it correct under a case-sensitive MySQL collation as well as a case-insensitive
|
||||
/// one.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static string JsonElementPrefixPattern(string query)
|
||||
internal static string JsonElementPrefixPattern(string query)
|
||||
{
|
||||
string encoded = JsonSerializer.Serialize(query ?? string.Empty);
|
||||
|
||||
// strip the quotes JsonSerializer wraps a string in
|
||||
encoded = encoded.Substring(1, encoded.Length - 2);
|
||||
|
||||
var builder = new StringBuilder("%\"");
|
||||
foreach (char c in encoded.ToLowerInvariant())
|
||||
|
||||
foreach (char c in query ?? string.Empty)
|
||||
{
|
||||
if (c is '%' or '_' or LikeEscapeChar)
|
||||
if (!char.IsAscii(c) || char.IsControl(c) || JsonEscapedAscii.Contains(c))
|
||||
{
|
||||
// cannot prove this character's stored form, so stop narrowing here
|
||||
break;
|
||||
}
|
||||
|
||||
char lower = char.ToLowerInvariant(c);
|
||||
if (lower is '%' or '_' or LikeEscapeChar)
|
||||
{
|
||||
builder.Append(LikeEscapeChar);
|
||||
}
|
||||
|
||||
builder.Append(c);
|
||||
builder.Append(lower);
|
||||
}
|
||||
|
||||
return builder.Append('%').ToString();
|
||||
@@ -245,7 +284,7 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
|
||||
private static async Task<List<string>> GetContentRatingValues(
|
||||
TvContext dbContext,
|
||||
string qLower,
|
||||
string query,
|
||||
int limit,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
@@ -264,13 +303,21 @@ public class GetSearchFieldValuesHandler(IDbContextFactory<TvContext> dbContextF
|
||||
.Where(cr => !string.IsNullOrEmpty(cr))
|
||||
.Distinct();
|
||||
|
||||
return FilterSortTake(split, qLower, limit);
|
||||
return FilterSortTake(split, query, limit);
|
||||
}
|
||||
|
||||
private static List<string> FilterSortTake(IEnumerable<string> values, string qLower, int limit) =>
|
||||
/// <summary>
|
||||
/// The one in-memory filter/sort/take every field funnels through. Both the comparison and the ordering
|
||||
/// are ORDINAL on purpose: <c>UseRequestLocalization</c> honours <c>Accept-Language</c>, so the current
|
||||
/// culture is caller-controlled, and <c>ToLower()</c> plus the default (linguistic)
|
||||
/// <c>StartsWith(string)</c> would make the result depend on it — under <c>tr-TR</c>, <c>q=I</c> lowers
|
||||
/// to <c>ı</c> and stops matching <c>Istanbul</c>. Ordinal is also what the SQL pre-filter's superset
|
||||
/// guarantee is proved against; see <see cref="JsonElementPrefixPattern" />.
|
||||
/// </summary>
|
||||
private static List<string> FilterSortTake(IEnumerable<string> values, string query, int limit) =>
|
||||
values
|
||||
.Where(v => v.ToLower().StartsWith(qLower))
|
||||
.OrderBy(v => v)
|
||||
.Where(v => v.StartsWith(query, StringComparison.OrdinalIgnoreCase))
|
||||
.OrderBy(v => v, StringComparer.Ordinal)
|
||||
.Take(limit)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using System.Globalization;
|
||||
using ErsatzTV.Application.Search.Queries;
|
||||
using ErsatzTV.Core.Api.Search;
|
||||
using ErsatzTV.Core.Domain;
|
||||
@@ -315,50 +316,95 @@ public class GetSearchFieldValuesHandlerTests
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task List_Valued_Source_Is_Bounded_By_A_Row_Cap()
|
||||
public async Task List_Valued_Source_Row_Cap_Is_Exactly_1000_Rows()
|
||||
{
|
||||
// The handler caps the raw-JSON pre-filter at 1000 rows, ordered by the raw column. Seed one row past
|
||||
// the cap whose JSON sorts last but whose element would sort FIRST in the output: an unbounded
|
||||
// implementation returns it, a capped one cannot.
|
||||
const string overCap = "aaa-past-the-row-cap";
|
||||
// Pins the cap value in BOTH directions. Asserting only that an over-cap row is dropped would also pass
|
||||
// with a cap of 3, or with the handler wrongly using `limit` as the row cap; asserting only that an
|
||||
// under-cap row survives would pass with no cap at all. The marker row's element sorts first in the
|
||||
// output, so an unbounded implementation cannot help but return it.
|
||||
const string marker = "aaa-the-marker-row";
|
||||
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
// rows are capped by ascending Id, and Id is assigned in insertion order
|
||||
async Task SeedFillerThenMarker(int fillerRows)
|
||||
{
|
||||
for (var i = 0; i < 1000; i++)
|
||||
await using TvContext context = _db.CreateContext();
|
||||
for (var i = 0; i < fillerRows; i++)
|
||||
{
|
||||
context.SongMetadata.Add(Song($"Song {i}", [$"b{i:D4}"]));
|
||||
}
|
||||
|
||||
context.SongMetadata.Add(Song("Over", ["zzz-sorts-last", overCap]));
|
||||
context.SongMetadata.Add(Song("Marker", [marker]));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var handler = new GetSearchFieldValuesHandler(_db.Factory);
|
||||
|
||||
Option<SearchFieldValuesResponseModel> result = await handler.Handle(
|
||||
// 999 filler rows -> the marker is row 1000 -> inside a cap of 1000
|
||||
await SeedFillerThenMarker(999);
|
||||
|
||||
Option<SearchFieldValuesResponseModel> atCap = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", string.Empty, 3),
|
||||
CancellationToken.None);
|
||||
|
||||
result.IsSome.ShouldBeTrue();
|
||||
result.IfSome(r => r.Values.ShouldBe(new List<string> { "b0000", "b0001", "b0002" }));
|
||||
result.IfSome(r => r.Values.ShouldNotContain(overCap));
|
||||
atCap.IsSome.ShouldBeTrue();
|
||||
atCap.IfSome(
|
||||
r => r.Values.ShouldBe(
|
||||
new List<string> { marker, "b0000", "b0001" },
|
||||
"the 1000th row must still be read - a cap below 1000 would drop it"));
|
||||
|
||||
// positive control: the same row IS reachable once the pre-filter narrows the candidate set below the cap
|
||||
// one more filler row pushes the marker to row 1001 -> outside a cap of 1000
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.SongMetadata.Add(Song("Extra", ["b9999"]));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
// move the marker to the end so it sits at Id 1002, past the cap
|
||||
SongMetadata existing = await context.SongMetadata.SingleAsync(m => m.Title == "Marker");
|
||||
context.SongMetadata.Remove(existing);
|
||||
await context.SaveChangesAsync();
|
||||
context.SongMetadata.Add(Song("Marker", [marker]));
|
||||
await context.SaveChangesAsync();
|
||||
|
||||
(await context.SongMetadata.CountAsync()).ShouldBe(1001);
|
||||
}
|
||||
|
||||
Option<SearchFieldValuesResponseModel> pastCap = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", string.Empty, 3),
|
||||
CancellationToken.None);
|
||||
|
||||
pastCap.IsSome.ShouldBeTrue();
|
||||
pastCap.IfSome(
|
||||
r => r.Values.ShouldNotContain(
|
||||
marker,
|
||||
"the 1001st row must not be read - an unbounded query would return it first"));
|
||||
pastCap.IfSome(r => r.Values.ShouldBe(new List<string> { "b0000", "b0001", "b0002" }));
|
||||
|
||||
// positive control: the same row IS reachable once the pre-filter narrows the candidates below the cap,
|
||||
// so the assertion above cannot be passing because the value is unreachable for an unrelated reason
|
||||
Option<SearchFieldValuesResponseModel> narrowed = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", "aaa", 50),
|
||||
CancellationToken.None);
|
||||
|
||||
narrowed.IsSome.ShouldBeTrue();
|
||||
narrowed.IfSome(r => r.Values.ShouldBe(new List<string> { overCap }));
|
||||
narrowed.IfSome(r => r.Values.ShouldBe(new List<string> { marker }));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task List_Valued_Columns_Are_Stored_As_Ascii_Escaped_Json_Arrays()
|
||||
public async Task Regression_Pin_List_Valued_Columns_Are_Stored_As_Ascii_Escaped_Json_Arrays()
|
||||
{
|
||||
// Pins the on-disk encoding the LIKE pre-filter is built against. If an EF upgrade changes the
|
||||
// JavaScriptEncoder (e.g. stops escaping non-ASCII), the pattern would silently stop matching —
|
||||
// this fails loudly instead. It also documents why LOWER() is provider-independent here: the
|
||||
// stored text is pure ASCII, so SQLite's ASCII-only lower() and MySQL's Unicode-aware LOWER() agree.
|
||||
// REGRESSION PIN, not coverage of #578: this asserts pre-existing EF behaviour and passes against the
|
||||
// code before this change. It is here because the pre-filter's correctness argument depends on the
|
||||
// stored form, so an encoder change must fail loudly rather than silently under-match.
|
||||
//
|
||||
// Specifically it pins that non-ASCII IS escaped — which is exactly why SQL LOWER() canNOT be used to
|
||||
// case-fold it. LOWER() lowercases the escape TEXT (\u00E9 -> \u00e9); it does not touch the codepoint
|
||||
// the escape denotes, so the stored `É` never folds to `é`. An earlier version of this change
|
||||
// claimed the opposite and JSON-encoded the whole query prefix, which silently made every accented
|
||||
// artist unsuggestable. The pattern builder now narrows only on verbatim ASCII; see
|
||||
// SearchFieldValuesPrefilterSupersetTests.
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.SongMetadata.Add(Song("One", ["Beyoncé", "\"Q\""]));
|
||||
@@ -375,6 +421,138 @@ public class GetSearchFieldValuesHandlerTests
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
[TestCase("é", "\u00C9dith Piaf")]
|
||||
[TestCase("\u00C9", "\u00C9dith Piaf")]
|
||||
[TestCase("\u00E9dith", "\u00C9dith Piaf")]
|
||||
[TestCase("bj", "Bj\u00F6rk")]
|
||||
[TestCase("bj\u00F6", "Bj\u00F6rk")]
|
||||
[TestCase("BJ\u00D6RK", "Bj\u00F6rk")]
|
||||
[TestCase("beyonc\u00E9", "Beyonc\u00E9")]
|
||||
[TestCase("sigur r", "Sigur R\u00F3s")]
|
||||
[TestCase("\u00D6", "\u00D6zdemir")]
|
||||
public async Task Prefilter_Never_Under_Matches_A_NonAscii_Value(string query, string stored)
|
||||
{
|
||||
// The SQL pre-filter narrows on raw JSON, where non-ASCII is stored escaped (\u00C9). Any narrowing
|
||||
// that includes an escaped character under-matches, because SQL LOWER() folds the escape TEXT and not
|
||||
// the codepoint — every one of these cases returned EMPTY before the pattern builder was restricted to
|
||||
// the leading verbatim-ASCII run. Accented artists are the common case in a music library, not an edge.
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.SongMetadata.AddRange(
|
||||
Song("Hit", [stored]),
|
||||
// negative control: a row that must never come back for any of these queries
|
||||
Song("Other", ["Nothing Relevant"]));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var handler = new GetSearchFieldValuesHandler(_db.Factory);
|
||||
|
||||
Option<SearchFieldValuesResponseModel> result = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", query, 50),
|
||||
CancellationToken.None);
|
||||
|
||||
result.IsSome.ShouldBeTrue();
|
||||
result.IfSome(r => r.Values.ShouldBe(new List<string> { stored }));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task Results_Do_Not_Depend_On_The_Request_Culture()
|
||||
{
|
||||
// UseRequestLocalization honours Accept-Language, so CurrentCulture is caller-controlled. Under tr-TR
|
||||
// the old `q.ToLower()` turned "I" into "\u0131" and the default linguistic StartsWith(string) compounded
|
||||
// it, so the same library answered differently per caller. The contract is ordinal: "I" matches
|
||||
// "Istanbul" and does NOT match "\u0131pek", in every culture.
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.SongMetadata.Add(Song("One", ["Istanbul Orkestrasi", "\u0131pek"]));
|
||||
context.ArtistMetadata.Add(Artist("Idil Biret"));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var handler = new GetSearchFieldValuesHandler(_db.Factory);
|
||||
var expected = new List<string> { "Idil Biret", "Istanbul Orkestrasi" };
|
||||
|
||||
CultureInfo original = CultureInfo.CurrentCulture;
|
||||
try
|
||||
{
|
||||
foreach (string culture in new[] { "en-US", "tr-TR", "az-AZ", "lt-LT" })
|
||||
{
|
||||
CultureInfo.CurrentCulture = new CultureInfo(culture);
|
||||
|
||||
Option<SearchFieldValuesResponseModel> result = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", "I", 50),
|
||||
CancellationToken.None);
|
||||
|
||||
result.IsSome.ShouldBeTrue();
|
||||
result.IfSome(r => r.Values.ShouldBe(expected, $"culture {culture} changed the result"));
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
CultureInfo.CurrentCulture = original;
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task Ordering_Is_Ordinal_And_Culture_Independent()
|
||||
{
|
||||
// The merge sorts ordinally rather than by culture, so the response order does not depend on the caller
|
||||
// either. Ordinal puts all ASCII uppercase before ASCII lowercase, and non-ASCII last.
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.SongMetadata.Add(Song("One", ["Zulu", "apple", "\u00C9clair", "Apple"]));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var handler = new GetSearchFieldValuesHandler(_db.Factory);
|
||||
var expected = new List<string> { "Apple", "Zulu", "apple", "\u00C9clair" };
|
||||
|
||||
CultureInfo original = CultureInfo.CurrentCulture;
|
||||
try
|
||||
{
|
||||
foreach (string culture in new[] { "en-US", "sv-SE" })
|
||||
{
|
||||
CultureInfo.CurrentCulture = new CultureInfo(culture);
|
||||
|
||||
Option<SearchFieldValuesResponseModel> result = await handler.Handle(
|
||||
new GetSearchFieldValues("artist", string.Empty, 50),
|
||||
CancellationToken.None);
|
||||
|
||||
result.IfSome(r => r.Values.ShouldBe(expected, $"culture {culture} changed the order"));
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
CultureInfo.CurrentCulture = original;
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task Ordering_Is_Best_Effort_When_A_Source_Truncates()
|
||||
{
|
||||
// Documents the acknowledged imprecision rather than claiming exactness the code does not have. The EF
|
||||
// source truncates by the DATABASE collation (SQLite's is ASCII-only), which is not the ordinal ordering
|
||||
// the merge then applies — so a value the database ranked outside its first `limit` never reaches the
|
||||
// merge, even if the merge would have ranked it first.
|
||||
await using (TvContext context = _db.CreateContext())
|
||||
{
|
||||
context.ArtistMetadata.Add(Artist("Zulu"));
|
||||
context.ArtistMetadata.Add(Artist("\u00C9clair"));
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
var handler = new GetSearchFieldValuesHandler(_db.Factory);
|
||||
|
||||
// with room for both, the ordinal merge ranks "Zulu" (ASCII) before "\u00C9clair" (non-ASCII)
|
||||
(await handler.Handle(new GetSearchFieldValues("artist", string.Empty, 50), CancellationToken.None))
|
||||
.IfSome(r => r.Values.ShouldBe(new List<string> { "Zulu", "\u00C9clair" }));
|
||||
|
||||
// with limit=1 the database picks the survivor by ITS collation, and the merge only sees that one
|
||||
(await handler.Handle(new GetSearchFieldValues("artist", string.Empty, 1), CancellationToken.None))
|
||||
.IfSome(r => r.Values.Count.ShouldBe(1));
|
||||
}
|
||||
|
||||
private static ArtistMetadata Artist(string title) => new()
|
||||
{
|
||||
MetadataKind = MetadataKind.External,
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
using System.Text.Json;
|
||||
using ErsatzTV.Application.Search.Queries;
|
||||
using NUnit.Framework;
|
||||
using Shouldly;
|
||||
|
||||
namespace ErsatzTV.Tests.Application.Search;
|
||||
|
||||
/// <summary>
|
||||
/// Proof obligations for the raw-JSON <c>LIKE</c> pre-filter that narrows the list-valued facet-value
|
||||
/// sources (#578). The pre-filter is an optimization layered under an in-memory exact filter, so it is
|
||||
/// allowed to over-match but must NEVER under-match — an under-match is a silent false negative, invisible
|
||||
/// to any test that only checks the values it does return.
|
||||
/// <para>
|
||||
/// The end-to-end behavioural cases live in <c>GetSearchFieldValuesHandlerTests</c>. These are the two
|
||||
/// facts the superset argument rests on, asserted directly rather than trusted.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[TestFixture]
|
||||
public class SearchFieldValuesPrefilterSupersetTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Printable ASCII, the alphabet <see cref="GetSearchFieldValuesHandler.JsonElementPrefixPattern" />
|
||||
/// narrows over (minus whatever the JSON writer escapes, which the sibling test pins).
|
||||
/// </summary>
|
||||
private static IEnumerable<char> PrintableAscii()
|
||||
{
|
||||
for (var c = ' '; c <= '~'; c++)
|
||||
{
|
||||
yield return c;
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void No_NonAscii_Codepoint_OrdinalIgnoreCase_Equals_A_Printable_Ascii_Character()
|
||||
{
|
||||
// THIS is why the narrowing run may be restricted to ASCII and still be a superset. The in-memory filter
|
||||
// compares with OrdinalIgnoreCase, so for a run character `c` the pre-filter must catch every element
|
||||
// character that OrdinalIgnoreCase-equals `c`. If any non-ASCII codepoint did, that element would be
|
||||
// stored as a \uXXXX escape, the ASCII pattern would miss the row, and the value would be unsuggestable.
|
||||
//
|
||||
// Note this is NOT true of InvariantCultureIgnoreCase, which folds ~190 codepoints (U+00AA -> a,
|
||||
// U+017F -> s, the modifier letters, ...) onto ASCII letters. The choice of Ordinal is load-bearing.
|
||||
char[] ascii = PrintableAscii().ToArray();
|
||||
var offenders = new List<string>();
|
||||
|
||||
for (var cp = 0x80; cp <= 0x10FFFF; cp++)
|
||||
{
|
||||
if (cp is >= 0xD800 and <= 0xDFFF)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
string s = char.ConvertFromUtf32(cp);
|
||||
foreach (char a in ascii)
|
||||
{
|
||||
if (string.Equals(s, a.ToString(), StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
offenders.Add($"U+{cp:X4} == '{a}'");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
offenders.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void The_Json_Writer_Escapes_Exactly_The_Ascii_Characters_The_Pattern_Builder_Refuses_To_Narrow_On()
|
||||
{
|
||||
// The narrowing run may only contain characters stored VERBATIM. Anything the writer escapes is stored
|
||||
// as \uXXXX and must terminate the run. If an encoder change adds a character to this set, the pattern
|
||||
// builder's exclusion list goes stale and starts under-matching — so pin the set, not a sample.
|
||||
var escaped = PrintableAscii()
|
||||
.Where(c => JsonSerializer.Serialize(c.ToString()) != $"\"{c}\"")
|
||||
.ToArray();
|
||||
|
||||
escaped.ShouldBe(['"', '&', '\'', '+', '<', '>', '\\', '`']);
|
||||
|
||||
// and the pattern builder must stop at every one of them (empty narrowing run -> bare anchor)
|
||||
foreach (char c in escaped)
|
||||
{
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern($"{c}abc")
|
||||
.ShouldBe("%\"%", $"expected no narrowing for the JSON-escaped '{c}'");
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void Pattern_Narrows_Only_On_The_Leading_Verbatim_Ascii_Run()
|
||||
{
|
||||
// plain ASCII: full narrowing, lowercased
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("Radio").ShouldBe("%\"radio%");
|
||||
|
||||
// non-ASCII terminates the run, keeping the ASCII head as the narrowing
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("Beyoncé").ShouldBe("%\"beyonc%");
|
||||
|
||||
// a leading non-ASCII character means no narrowing at all
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("édith").ShouldBe("%\"%");
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("Ünen").ShouldBe("%\"%");
|
||||
|
||||
// empty query: the bare element-opening anchor
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern(string.Empty).ShouldBe("%\"%");
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern(null).ShouldBe("%\"%");
|
||||
|
||||
// LIKE wildcards inside the run are escaped, not passed through
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("50%").ShouldBe("%\"50/%%");
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("a_b").ShouldBe("%\"a/_b%");
|
||||
|
||||
// the escape character itself is escaped
|
||||
GetSearchFieldValuesHandler.JsonElementPrefixPattern("AC/DC").ShouldBe("%\"ac//dc%");
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void Row_Cap_Is_Applied_On_The_Primary_Key_Not_The_Json_Column()
|
||||
{
|
||||
// MySQL sorts TEXT/longtext using only max_sort_length bytes, so ordering on the JSON column would let
|
||||
// rows sharing a long prefix tie arbitrarily and the surviving 1000 would be unspecified. Id is a unique
|
||||
// integer key on both providers.
|
||||
string sql = GetSearchFieldValuesHandler.ListValuedSql("Artists");
|
||||
|
||||
sql.ShouldContain("ORDER BY Id LIMIT @Cap");
|
||||
sql.ShouldNotContain("ORDER BY Artists");
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Application.Search.Queries;
|
||||
using ErsatzTV.Infrastructure;
|
||||
using ErsatzTV.Infrastructure.Data;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
@@ -36,9 +36,9 @@ public class SearchFieldValuesQueryShapeTests
|
||||
{
|
||||
using TvContext context = create();
|
||||
|
||||
// exactly the queryable half of GetSearchFieldValuesHandler.GetSource("artist")
|
||||
string sql = context.ArtistMetadata.Select(m => m.Title)
|
||||
.Concat(context.Set<MusicVideoArtist>().Select(a => a.Name))
|
||||
// calls the handler's own source builder (internal, via InternalsVisibleTo) rather than rebuilding
|
||||
// the LINQ here — a copy would keep passing after the handler's query changed underneath it
|
||||
string sql = GetSearchFieldValuesHandler.GetSource(context, "artist")
|
||||
.Where(v => v != null && v.ToLower().StartsWith("a"))
|
||||
.Distinct()
|
||||
.OrderBy(v => v)
|
||||
@@ -55,8 +55,11 @@ public class SearchFieldValuesQueryShapeTests
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void Song_List_Columns_Cannot_Be_Projected_Server_Side_On_Either_Provider()
|
||||
public void Regression_Pin_Song_List_Columns_Cannot_Be_Projected_Server_Side_On_Either_Provider()
|
||||
{
|
||||
// REGRESSION PIN, not coverage of #578: this asserts pre-existing EF/provider behaviour and passes
|
||||
// against the code before this change.
|
||||
//
|
||||
// Documents WHY the handler drops to raw SQL for SongMetadata.Artists / .AlbumArtists rather than
|
||||
// SelectMany-ing them: EF maps them as JSON primitive collections and neither provider can translate
|
||||
// the projection (SQLite needs APPLY; Pomelo has no primitive-collection support). If a provider
|
||||
|
||||
@@ -205,8 +205,9 @@ public class SearchController(IMediator mediator) : ControllerBase
|
||||
"Returns distinct whole values from the database for the given text field, filtered by an " +
|
||||
"optional case-insensitive prefix. Powers the visual rule builder's facet-value typeahead. " +
|
||||
"404 when the field is unknown, is not a text field, or is a text field with no distinct-value " +
|
||||
"source. The list-valued music fields (artist, album_artist) are bounded best-effort: on a very " +
|
||||
"large library they return a deterministic subset of the matches rather than every match.")]
|
||||
"source. Matching, dedup and ordering are ordinal, not culture-dependent. The list-valued music " +
|
||||
"fields (artist, album_artist) are bounded best-effort: on a very large library they return a " +
|
||||
"bounded subset of the matches rather than every match.")]
|
||||
[EndpointGroupName("general")]
|
||||
[ProducesResponseType(typeof(SearchFieldValuesResponseModel), StatusCodes.Status200OK)]
|
||||
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status404NotFound)]
|
||||
|
||||
@@ -18063,7 +18063,7 @@
|
||||
"Search"
|
||||
],
|
||||
"summary": "List distinct database values for a text search field",
|
||||
"description": "Returns distinct whole values from the database for the given text field, filtered by an optional case-insensitive prefix. Powers the visual rule builder's facet-value typeahead. 404 when the field is unknown, is not a text field, or is a text field with no distinct-value source. The list-valued music fields (artist, album_artist) are bounded best-effort: on a very large library they return a deterministic subset of the matches rather than every match.",
|
||||
"description": "Returns distinct whole values from the database for the given text field, filtered by an optional case-insensitive prefix. Powers the visual rule builder's facet-value typeahead. 404 when the field is unknown, is not a text field, or is a text field with no distinct-value source. Matching, dedup and ordering are ordinal, not culture-dependent. The list-valued music fields (artist, album_artist) are bounded best-effort: on a very large library they return a bounded subset of the matches rather than every match.",
|
||||
"operationId": "GetSearchFieldValues",
|
||||
"parameters": [
|
||||
{
|
||||
|
||||
+12
-5
@@ -456,11 +456,18 @@ No server-side caching. Powers the visual rule builder's value-input combobox fo
|
||||
or partly) by `SongMetadata.Artists`/`AlbumArtists`, which EF maps as **primitive collections** — one
|
||||
JSON array per row in a single column, with no server-side projection on either provider. `album_artist`
|
||||
therefore no longer 404s, and `artist` now also covers free-text music-video (`MusicVideoArtist`) and
|
||||
song credits, not only entity artists. For these fields the handler pre-filters on the raw JSON in SQL
|
||||
and caps the rows it reads, so on a very large library the response is a **deterministic subset** of the
|
||||
matches rather than every match — say so in the `[EndpointDescription]` of any endpoint that adopts this
|
||||
shape, and never let an unbounded scan stand in for it. Full rationale, the rejected normalized-side-table
|
||||
alternative and the dual-provider `LIKE`/`LOWER` portability rules: `api.search-field-values-list-columns`.
|
||||
song credits, not only entity artists. For these fields the handler narrows on the raw JSON in SQL and
|
||||
caps the rows it reads, so on a very large library the response is a bounded subset of the matches rather
|
||||
than every match — say so in the `[EndpointDescription]` of any endpoint that adopts this shape.
|
||||
|
||||
Two rules generalize beyond this endpoint. **A SQL pre-filter layered under an in-memory exact filter may
|
||||
over-match but must NEVER under-match** — an under-match is a silent false negative that no assertion over
|
||||
the returned values can see, so prove the superset (ours is proved by an exhaustive Unicode sweep, not by
|
||||
argument) rather than eyeballing it. And **prefix matching, dedup and ordering on any `/api/*` read must be
|
||||
ordinal, never current-culture**: `UseRequestLocalization` honours `Accept-Language`, so `ToLower()` and the
|
||||
default linguistic `StartsWith(string)` let a caller change the result by changing a header. Full rationale,
|
||||
the rejected normalized-side-table alternative, the accepted scan cost and the dual-provider `LIKE`/`LOWER`
|
||||
rules: `api.search-field-values-sources` (which supersedes `api.search-field-values`).
|
||||
|
||||
**Param + DTO expansion (#293, cap `search/all-items`)**: no new endpoint — `GET /api/v1/search/all-items`
|
||||
gained two **optional** query params (`pageSize` default 500, clamped 1–1000 via the §1 Logs `Math.Clamp`
|
||||
|
||||
+2
-2
@@ -206,7 +206,7 @@ another doc or an old issue comment should land here and then follow the link.
|
||||
- 2026-07-22 — per-schedule clock-boundary padding is a synthetic content-less Pad over the existing per-episode machinery (#392) — [`sched.clock-padding-schedule-toggle`](decisions/records/sched/clock-padding-schedule-toggle.md)
|
||||
- 2026-07-23 — Channel health = a server-derived `health` object on the channel DTOs, built-timeline detection (#415) — [`api.channel-health-object`](decisions/records/api/channel-health-object.md)
|
||||
- 2026-07-23 — Channel origin is immutable creation-provenance, stamped at insert, not a health signal (#414) — [`channel.origin-marker`](decisions/records/channel/origin-marker.md)
|
||||
- 2026-07-23 — Facet-value typeahead is a new endpoint, allow-listed to text fields, no caching (#434) — [`api.search-field-values`](decisions/records/api/search-field-values.md)
|
||||
- 2026-07-23 — Facet-value typeahead is a new endpoint, allow-listed to text fields, no caching (#434) — [`api.search-field-values`](decisions/archive/api/search-field-values.md) (superseded by `api.search-field-values-sources`)
|
||||
- 2026-07-23 — Relative-date rule builder operators are a frontend-only mapping onto existing Lucene macros (#435) — [`rulebuilder.relative-date-macros`](decisions/records/rulebuilder/relative-date-macros.md)
|
||||
- 2026-07-25 — A media-server sweep also refuses when the api client silently dropped items whose projection threw; the ratio threshold is rejected (#484) — [`scan.projection-failure-sweep-guard`](decisions/records/scan/projection-failure-sweep-guard.md)
|
||||
- 2026-07-25 — LibraryFolder identity is enforced by a unique index on `(LibraryPathId, PathHash)`, not an in-process lock (#491) — [`scan.libraryfolder-unique-identity`](decisions/records/scan/libraryfolder-unique-identity.md)
|
||||
@@ -215,4 +215,4 @@ another doc or an old issue comment should land here and then follow the link.
|
||||
- 2026-07-25 — Rule-builder group nesting is bounded-arbitrary depth (`MAX_GROUP_DEPTH`), not one level (#436) — [`spa.rulebuilder-nesting`](decisions/records/spa/rulebuilder-nesting.md)
|
||||
- 2026-07-25 — The rationale-edit marker is a git trailer, not a substring anywhere in the commit range (#609) — [`ci.decisions-edit-trailer`](decisions/records/ci/decisions-edit-trailer.md)
|
||||
- 2026-07-25 — UI-E2E: headless Playwright flows in the existing `functional-e2e` job, browser baked into the CI image (#445) — [`ci.ui-e2e-harness`](decisions/records/ci/ui-e2e-harness.md)
|
||||
- 2026-07-26 — List-valued (JSON primitive collection) facet fields get bounded best-effort values via a raw-JSON LIKE pre-filter, not a normalized side table (#578) — [`api.search-field-values-list-columns`](decisions/records/api/search-field-values-list-columns.md)
|
||||
- 2026-07-26 — Facet-value typeahead restated: every artist source covered; list-valued columns via a superset LIKE pre-filter that may over-match but never under-match (#578) — [`api.search-field-values-sources`](decisions/records/api/search-field-values-sources.md)
|
||||
|
||||
@@ -28,8 +28,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `api.schedule-item-flat-dto` | Schedule-item GET/POST/PUT use a flat, non-polymorphic `ScheduleItemResponseModel` (every subtype field promoted to a nullable top-level member) instead of the polymorphic Application VM hierarchy, with mutation field names matching `ScheduleItemRequest` 1:1 for a lossless round-trip. | 2026-07-10 | [link](records/api/schedule-item-flat-dto.md) |
|
||||
| `api.scheduling-hardening` | Create/Replace handlers guard against null/whitespace `name` (`IsNullOrWhiteSpace`, not just `Length`) to prevent NRE-500s, template-item overlap validation compares by index (not record value-equality) to catch exact-duplicate items, and unreachable 404 `ProducesResponseType` attributes on create-only actions are trimmed. | 2026-07-13 | [link](records/api/scheduling-hardening.md) |
|
||||
| `api.search-allitems-paging` | `GET /api/v1/search/all-items` is paginated (capped page size, `Totals` field) to bound DoS exposure; the SPA add-all flow pages to completeness instead of relying on an unbounded response. | 2026-07-18 | [link](records/api/search-allitems-paging.md) |
|
||||
| `api.search-field-values` | `GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for one of a narrow allow-list of catalog fields (not the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a typeahead suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source; case-insensitive prefix-filtered on `q`, `limit` clamped to `[1, 50]` (default 50). | 2026-07-23 | [link](records/api/search-field-values.md) |
|
||||
| `api.search-field-values-list-columns` | A `text` search field whose values live in an EF **primitive collection** (one JSON array per row in a single column — `SongMetadata.Artists`, `SongMetadata.AlbumArtists`) IS served by `GET /api/v1/search/fields/{name}/values`, not 404'd, but as **bounded best-effort**: SQL pre-filters on the raw JSON with `LOWER(col) LIKE '%"<json-encoded-lowercased-prefix>%' ESCAPE '/'` — a deliberate superset, since it matches a row, not an element — capped at a fixed row cap (1000) with an `ORDER BY` on the raw column so the truncation point is deterministic; the rows are then split, exact-prefix-filtered, deduped, sorted and `limit`ed in memory. An empty `q` is supported (it degenerates to `%"%`, still capped), so these fields need no special-case client contract. The prefix is JSON-encoded before matching because that is how it is stored, which also makes the pattern pure ASCII and therefore makes `LOWER()` mean the same thing on SQLite (ASCII-only) and MySQL (Unicode-aware); the pattern is lowercased in C# rather than relying on `LIKE`'s own case rules, so the match is correct under a case-sensitive MySQL collation as well as a case-insensitive one. The escape character is `/`, never `\` — `ESCAPE '\'` is not a portable SQL literal. | 2026-07-26 | [link](records/api/search-field-values-list-columns.md) |
|
||||
| `api.search-field-values-sources` | `GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for a narrow allow-list of catalog fields (never the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source (`title`, `show_title` only); `limit` clamped to `[1, 50]` (default 50). Prefix matching, dedup and ordering are ORDINAL (`OrdinalIgnoreCase` / `StringComparer.Ordinal`), never current-culture, because `UseRequestLocalization` makes the culture caller-controlled. A field whose values live in an EF **primitive collection** (one JSON array per row in a single column: `SongMetadata.Artists`, `SongMetadata.AlbumArtists`) is served, not 404'd, as bounded best-effort: SQL narrows on the raw JSON with a `LIKE` pattern built ONLY from the leading run of characters the JSON writer stores verbatim (printable ASCII minus `"&'+<>\`), stopping at the first character it cannot prove — this pre-filter is an optimization that MAY over-match and MUST NEVER under-match, with correctness living in the in-memory exact filter; the rows are capped at 1000 on `ORDER BY Id` (a unique integer key, not the JSON column) and then split, filtered, deduped, sorted and `limit`ed in memory. Ordering across merged sources is best-effort, exact only below the truncation points. | 2026-07-26 | [link](records/api/search-field-values-sources.md) |
|
||||
| `api.search-paging-cap` | Search stays capped at 100 items per media kind; an overflowing kind's "See all" reuses library-browse paging instead of adding new API surface. | 2026-07-11 | [link](records/api/search-paging-cap.md) |
|
||||
| `api.versioning-v1` | The entire `/api` surface is versioned to `/api/v1` uniformly (no unversioned corner); legacy unversioned callers are rewritten in-pipeline (not redirected) with Deprecation/Link/Sunset headers, and post-freeze `/api/v1` is additive-only — a breaking change requires `/api/v2`. | 2026-07-13 | [link](records/api/versioning-v1.md) |
|
||||
| `blazor.rollback-tag` | The commit immediately preceding the Blazor-removal merge is tagged `blazor-final` (not a `v*` tag, so it doesn't trigger a prod release build) as the documented rollback/restore path. | 2026-07-11 | [link](records/blazor/rollback-tag.md) |
|
||||
|
||||
+10
-12
@@ -1,13 +1,13 @@
|
||||
---
|
||||
key: api.search-field-values
|
||||
title: 2026-07-23 — Facet-value typeahead is a new endpoint, allow-listed to text fields, no caching (#434)
|
||||
status: active
|
||||
status: superseded
|
||||
since: '2026-07-23'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: '`GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for one of a narrow allow-list of catalog fields (not the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a typeahead suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source; case-insensitive prefix-filtered on `q`, `limit` clamped to `[1, 50]` (default 50).'
|
||||
superseded-by: api.search-field-values-sources@2026-07-26
|
||||
rule: '(superseded) `GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for one of a narrow allow-list of catalog fields (not the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a typeahead suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source; case-insensitive prefix-filtered on `q`, `limit` clamped to `[1, 50]` (default 50).'
|
||||
signals: 'facet-value typeahead, rule builder value combobox, distinct field values, GetSearchFieldValues, text field allow-list, DB-sourced distinct values, content_rating split · paths: `ErsatzTV/Controllers/Api/SearchController.cs`, `ErsatzTV.Application/Search/Queries/GetSearchFieldValues.cs`, `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `web/src/api/search.ts` · issues: #434, #176'
|
||||
mechanics: '`SearchController.GetSearchFieldValues`; `GetSearchFieldValuesHandler`; api-conventions.md; spa-conventions.md §12'
|
||||
mechanics: superseded by `api.search-field-values-sources` (ersatztv#578), which keeps this endpoint contract and reverses the "no distinct-value source" call for the list-valued music fields
|
||||
---
|
||||
|
||||
Enum fields (e.g. `type`, `content_rating` group) already ship their allowed values inline on
|
||||
@@ -23,9 +23,8 @@ an empty list for a field that will never have values — a 404 tells a caller "
|
||||
**DB-sourced, not the search index.** The handler injects `IDbContextFactory<TvContext>` and resolves an
|
||||
explicit per-field-name `IQueryable<string>` (or, for a few special cases, an in-memory list) rather than
|
||||
querying `ISearchIndex`: `genre`/`show_genre` → `Set<Genre>()`, `studio` → `Set<Studio>()`, `director` →
|
||||
`Set<Director>()`, `writer` → `Set<Writer>()`, `actor` → `Actors`, `artist` → `ArtistMetadata.Title`
|
||||
concatenated with `Set<MusicVideoArtist>().Name` (plus `SongMetadata.Artists` via the list-valued path —
|
||||
see `api.search-field-values-list-columns`, #578), `tag` →
|
||||
`Set<Director>()`, `writer` → `Set<Writer>()`, `actor` → `Actors`, `artist` → `ArtistMetadata.Title` (entity
|
||||
artists only — free-text music-video/song artist credits are a known, intentionally-uncovered gap), `tag` →
|
||||
`Set<Tag>()` excluding `Tag.NfoCountryTypeId`/`Tag.PlexNetworkTypeId` (reapplying the indexer's own
|
||||
exclusions so country/network strings don't leak in as tags), `network` → `Set<Tag>()` filtered to
|
||||
`Tag.PlexNetworkTypeId`, `collection` → `Collections`, `video_codec` → `MediaStreams` filtered to
|
||||
@@ -37,11 +36,10 @@ of queried: `state` (the fixed 4-value `MediaItemState` enum) and `video_dynamic
|
||||
`MovieMetadata`/`ShowMetadata`/`OtherVideoMetadata`/`RemoteStreamMetadata`, so the handler pulls the distinct
|
||||
raw strings then `Split('/')`s, trims, and dedupes in memory before the same prefix-filter/sort/take — this
|
||||
matches what search actually matches on, rather than surfacing the compound string as one facet value.
|
||||
**`title` and `show_title` are explicitly NOT supported** (404, free-text fallback): they are near-unique
|
||||
free-text fields spanning ~9 metadata tables where a distinct list of every title isn't a useful facet.
|
||||
`album_artist` was originally 404 here for a different reason — it backs onto `SongMetadata.AlbumArtists`,
|
||||
which EF can't translate into a server-side distinct query — and since #578 it returns bounded best-effort
|
||||
values instead; see `api.search-field-values-list-columns`.
|
||||
**`title`, `show_title`, `album_artist` are explicitly NOT supported** (404, free-text fallback): `title`/
|
||||
`show_title` are near-unique free-text fields spanning ~9 metadata tables where a distinct list of every
|
||||
title isn't a useful facet; `album_artist` backs onto `SongMetadata.AlbumArtists`, a value-converted
|
||||
`IList<string>` column EF can't translate into a server-side distinct query.
|
||||
|
||||
**Why a thin query, not a cache.** No result cache, no debounce on the server side (the SPA combobox
|
||||
debounces the keystroke) — each per-field query is a bounded, indexed `Distinct`/`Take`; adding a cache
|
||||
@@ -1,75 +0,0 @@
|
||||
---
|
||||
key: api.search-field-values-list-columns
|
||||
title: '2026-07-26 — list-valued (JSON primitive collection) fields get bounded best-effort facet values via a raw-JSON LIKE pre-filter, not a normalized side table (#578)'
|
||||
status: active
|
||||
since: '2026-07-26'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'A `text` search field whose values live in an EF **primitive collection** (one JSON array per row in a single column — `SongMetadata.Artists`, `SongMetadata.AlbumArtists`) IS served by `GET /api/v1/search/fields/{name}/values`, not 404''d, but as **bounded best-effort**: SQL pre-filters on the raw JSON with `LOWER(col) LIKE ''%"<json-encoded-lowercased-prefix>%'' ESCAPE ''/''` — a deliberate superset, since it matches a row, not an element — capped at a fixed row cap (1000) with an `ORDER BY` on the raw column so the truncation point is deterministic; the rows are then split, exact-prefix-filtered, deduped, sorted and `limit`ed in memory. An empty `q` is supported (it degenerates to `%"%`, still capped), so these fields need no special-case client contract. The prefix is JSON-encoded before matching because that is how it is stored, which also makes the pattern pure ASCII and therefore makes `LOWER()` mean the same thing on SQLite (ASCII-only) and MySQL (Unicode-aware); the pattern is lowercased in C# rather than relying on `LIKE`''s own case rules, so the match is correct under a case-sensitive MySQL collation as well as a case-insensitive one. The escape character is `/`, never `\` — `ESCAPE ''\''` is not a portable SQL literal.'
|
||||
signals: 'artist typeahead free-text credits, album_artist 404, SongMetadata.Artists, SongMetadata.AlbumArtists, MusicVideoArtist suggestions, EF primitive collection, PrimitiveCollection JSON column, SelectMany requires APPLY on SQLite, Pomelo primitive collections not enabled, raw JSON LIKE pre-filter, ListValuedRowCap, bounded best-effort facet values, LIKE ESCAPE portability · paths: `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `ErsatzTV/Controllers/Api/SearchController.cs`, `ErsatzTV.Tests/Application/Search/GetSearchFieldValuesHandlerTests.cs`, `ErsatzTV.Tests/Application/Search/SearchFieldValuesQueryShapeTests.cs` · issues: #578, #434'
|
||||
mechanics: '`GetSearchFieldValuesHandler.GetSongListValuedColumn` / `.GetSongListValuedValues` / `.ListValuedSql` / `.JsonElementPrefixPattern`; `SearchFieldValuesQueryShapeTests` (dual-provider `ToQueryString` guard); `api.search-field-values` (the base endpoint contract this extends)'
|
||||
---
|
||||
|
||||
Extends `api.search-field-values` rather than superseding it: that record's rule still reads true —
|
||||
whole values, from the database, not the Lucene term dictionary, 404 for a field with no
|
||||
distinct-value source. What changed is only **which** fields have a source. Its body's claim that
|
||||
music-video/song artist credits are "a known, intentionally-uncovered gap" and that `album_artist`
|
||||
has no server-side source was corrected in place.
|
||||
|
||||
**The three `artist` sources are not the same problem.** `LuceneSearchIndex` writes the `artist`
|
||||
field from three places, and only two of them are ordinary columns:
|
||||
|
||||
- `ArtistMetadata.Title` — a plain column on a plain table.
|
||||
- `MusicVideoArtist.Name` — also a real entity table (`MusicVideoMetadata.HasMany(m => m.Artists)`),
|
||||
so the free-text music-video credits are directly `SELECT DISTINCT`-able. These two are simply
|
||||
`Concat`ed into the existing server-side pipeline
|
||||
(`.Where(v => v.ToLower().StartsWith(qLower)).Distinct().OrderBy(v => v).Take(limit)`), which EF
|
||||
emits as one bounded `UNION ALL` + `LOWER(...) LIKE ... LIMIT` on both providers.
|
||||
- `SongMetadata.Artists` (and, for `album_artist`, `SongMetadata.AlbumArtists`) — an
|
||||
`IList<string>` that EF 9 maps as a **primitive collection**: no `HasConversion` anywhere, one
|
||||
JSON array per row in a single `TEXT`/`longtext` column. This one has no server-side projection at
|
||||
all. Verified empirically against both providers: SQLite reports *"Translating this query requires
|
||||
the SQL APPLY operation, which is not supported on SQLite"*, and Pomelo MySQL 9.0.0 reports
|
||||
*"Primitive collections support has not been enabled"*. `SearchFieldValuesQueryShapeTests` pins
|
||||
both failures, so if a provider upgrade ever makes the projection translate, the raw-SQL path can
|
||||
be retired on a red test rather than on a hunch.
|
||||
|
||||
**Why the LIKE superset and not a normalized `SongArtist` table.** A real join table (the shape
|
||||
`MusicVideoArtist` already has) would make this exact and indexable forever, but it costs a
|
||||
dual-provider schema migration plus a data backfill, changes to every scanner write path that
|
||||
populates `SongMetadata.Artists`, changes to the Lucene indexer, and leaves two representations of
|
||||
the same fact to drift apart — a large blast radius for a typeahead convenience. Rejected on cost.
|
||||
The raw-JSON pre-filter is confined to one private method, needs no migration, and is a strictly
|
||||
additive read path. Also rejected: querying the Lucene term dictionary, which is the thing
|
||||
`api.search-field-values` exists to avoid (analyzed `TextField`s store lowercased word tokens).
|
||||
|
||||
**The bound is the point, and it is tested behaviourally.** `ListValuedRowCap = 1000` bounds rows
|
||||
fetched, not values returned, because the pre-filter matches rows whole. The `ORDER BY` on the raw
|
||||
column is what makes the truncation deterministic — and therefore testable:
|
||||
`List_Valued_Source_Is_Bounded_By_A_Row_Cap` seeds 1001 rows where the over-cap row's JSON sorts
|
||||
*last* but its element would sort *first* in the output, so an unbounded implementation returns it
|
||||
and a capped one cannot. A positive control in the same test narrows `q` until that row is back
|
||||
under the cap and asserts it reappears, so the test cannot pass by the value being unreachable for
|
||||
some unrelated reason.
|
||||
|
||||
**`album_artist` 404 → 200 is additive.** Nothing consumes the 404 as a signal: the SPA's
|
||||
`getSearchFieldValues` (`web/src/api/search.ts`) treats any non-200 as "no suggestions, fall back to
|
||||
a free-text input", which is exactly what it will now do less often. Per `api.versioning-v1`,
|
||||
widening which fields return values — and which values a field returns — adds capability without
|
||||
removing any, so it needs no `/api/v2`.
|
||||
|
||||
**Known imprecision, deliberately accepted.** On a library with more than 1000 rows matching the
|
||||
pre-filter, the suggestions for `artist`/`album_artist` are a deterministic prefix of the matches
|
||||
rather than every match. This is stated on the endpoint's `[EndpointDescription]` so it reaches the
|
||||
OpenAPI document. The alternative — requiring a non-empty `q` for list-valued fields — was rejected:
|
||||
it would make one group of fields behave differently from every other field for the same client
|
||||
code, for a case the row cap already handles.
|
||||
|
||||
**Dual-provider verification, and its limit.** The queryable half is pinned on both providers by
|
||||
`SearchFieldValuesQueryShapeTests`, which compiles the query with `ToQueryString()` — no server
|
||||
needed. The raw-SQL half cannot be executed against MySQL from the unit-test suite; it is defended
|
||||
by construction instead (portable `LOWER`/`LIKE`/`ESCAPE`/`ORDER BY`/`LIMIT`, unquoted identifiers,
|
||||
a non-backslash escape character, and an ASCII-only pattern so collation and `LOWER()` semantics
|
||||
cannot diverge) plus `List_Valued_Columns_Are_Stored_As_Ascii_Escaped_Json_Arrays`, which pins the
|
||||
on-disk JSON encoding the pattern is built against so an EF encoder change fails loudly instead of
|
||||
silently returning nothing.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
key: api.search-field-values-sources
|
||||
title: '2026-07-26 — Facet-value typeahead, restated: every artist-bearing source is covered, list-valued columns via a superset LIKE pre-filter that may over-match but never under-match (#578)'
|
||||
status: active
|
||||
since: '2026-07-26'
|
||||
supersedes: api.search-field-values@2026-07-23
|
||||
superseded-by: none
|
||||
rule: '`GET /api/v1/search/fields/{name}/values?q=&limit=` returns distinct WHOLE values from the database for a narrow allow-list of catalog fields (never the Lucene term dictionary — analyzed `TextField`s store lowercased word tokens, e.g. "Science Fiction" → `science`/`fiction`, useless as a suggestion), 404 for an unknown field, a non-`text` field, or a `text` field with no distinct-value source (`title`, `show_title` only); `limit` clamped to `[1, 50]` (default 50). Prefix matching, dedup and ordering are ORDINAL (`OrdinalIgnoreCase` / `StringComparer.Ordinal`), never current-culture, because `UseRequestLocalization` makes the culture caller-controlled. A field whose values live in an EF **primitive collection** (one JSON array per row in a single column: `SongMetadata.Artists`, `SongMetadata.AlbumArtists`) is served, not 404''d, as bounded best-effort: SQL narrows on the raw JSON with a `LIKE` pattern built ONLY from the leading run of characters the JSON writer stores verbatim (printable ASCII minus `"&''+<>\`), stopping at the first character it cannot prove — this pre-filter is an optimization that MAY over-match and MUST NEVER under-match, with correctness living in the in-memory exact filter; the rows are capped at 1000 on `ORDER BY Id` (a unique integer key, not the JSON column) and then split, filtered, deduped, sorted and `limit`ed in memory. Ordering across merged sources is best-effort, exact only below the truncation points.'
|
||||
signals: 'artist typeahead free-text credits, album_artist 404, artist suggestions missing music videos, SongMetadata.Artists, SongMetadata.AlbumArtists, MusicVideoArtist, EF primitive collection, PrimitiveCollection JSON column, SelectMany requires APPLY on SQLite, Pomelo primitive collections not enabled, raw JSON LIKE pre-filter, superset never under-match, accented artist unsuggestable, LOWER cannot fold a \u00XX escape, OrdinalIgnoreCase vs InvariantCultureIgnoreCase folding, Accept-Language tr-TR dotless i, ListValuedRowCap, ORDER BY Id not TEXT, max_sort_length, content_rating split, text field allow-list · paths: `ErsatzTV.Application/Search/Queries/GetSearchFieldValuesHandler.cs`, `ErsatzTV/Controllers/Api/SearchController.cs`, `ErsatzTV.Tests/Application/Search/SearchFieldValuesPrefilterSupersetTests.cs`, `ErsatzTV.Tests/Application/Search/GetSearchFieldValuesHandlerTests.cs`, `web/src/api/search.ts` · issues: #578, #434, #176'
|
||||
mechanics: '`GetSearchFieldValuesHandler` (`GetSource`, `GetSongListValuedColumn`, `GetSongListValuedValues`, `ListValuedSql`, `JsonElementPrefixPattern`, `FilterSortTake`); `SearchFieldValuesPrefilterSupersetTests` (the superset proof); api-conventions.md; spa-conventions.md §12'
|
||||
---
|
||||
|
||||
Supersedes `api.search-field-values` (#434). That record did not merely carry a stale implementation
|
||||
detail — it recorded a **call**: free-text music-video/song artist credits were "a known,
|
||||
intentionally-uncovered gap" and `album_artist` was unsupported. #578 reverses that call, so this is
|
||||
a supersession, not a line-edit. Everything #434 settled that still holds is restated here rather
|
||||
than left in the archive: enum fields ship their values inline on `GET /api/v1/search/fields` and
|
||||
need no lookup; text fields need a live one; the source is the database and never the search index;
|
||||
`content_rating` splits its compound `"PG-13/TV-14"` strings in memory; there is no result cache.
|
||||
|
||||
## The three `artist` sources are three different problems
|
||||
|
||||
`LuceneSearchIndex` writes the `artist` field from three places, and only two are ordinary columns:
|
||||
|
||||
- `ArtistMetadata.Title` — a plain column. Already worked.
|
||||
- `MusicVideoArtist.Name` — also a real entity table (`MusicVideoMetadata.HasMany(m => m.Artists)`),
|
||||
so the free-text music-video credits are directly `SELECT DISTINCT`-able. It just joins the
|
||||
existing server-side pipeline as a `Concat`, emitted as one bounded `UNION ALL` +
|
||||
`LOWER(...) LIKE ... LIMIT` on both providers.
|
||||
- `SongMetadata.Artists` (and, for `album_artist`, `AlbumArtists`) — an `IList<string>` EF 9 maps as
|
||||
a **primitive collection**: no `HasConversion` anywhere, one JSON array per row in a single
|
||||
`TEXT`/`longtext` column, with no server-side projection at all. Verified against both providers:
|
||||
SQLite reports *"Translating this query requires the SQL APPLY operation, which is not supported on
|
||||
SQLite"*, Pomelo MySQL 9.0.0 reports *"Primitive collections support has not been enabled"*. Both
|
||||
failures are pinned by a test, so a provider upgrade that fixes them surfaces as a red rather than
|
||||
leaving a workaround in place forever.
|
||||
|
||||
## The pre-filter is an optimization; it may over-match, it must never under-match
|
||||
|
||||
This is the load-bearing invariant, and the first implementation got it wrong in a way that no test
|
||||
of the returned values could see. It JSON-encoded the whole query prefix, reasoning that since the
|
||||
stored text escapes non-ASCII (`Édith` is on disk as `Édith`), encoding the prefix the same way
|
||||
would line up. **It does not.** SQL `LOWER()` lowercases the *escape text* — `É` — it cannot
|
||||
case-fold the codepoint that escape denotes. So `q=é` produced a pattern of `é` that never
|
||||
matched, the row was discarded before the in-memory filter ever saw it, and **every accented artist
|
||||
was silently unsuggestable** — in a music library, the common case, not an edge case.
|
||||
|
||||
The fix is to narrow only on ground that can be proved: the leading run of characters the JSON
|
||||
writer stores **verbatim**, stopping at the first non-ASCII or JSON-escaped character. A query
|
||||
starting with such a character narrows to the bare element-opening anchor `%"%` and leans on the row
|
||||
cap and the in-memory filter. `q=Beyoncé` still narrows on `beyonc`; `q=é` narrows on nothing.
|
||||
|
||||
Soundness rests on two facts, both **asserted by exhaustive computation** in
|
||||
`SearchFieldValuesPrefilterSupersetTests` rather than argued:
|
||||
|
||||
1. **No non-ASCII codepoint in all of Unicode `OrdinalIgnoreCase`-equals a printable ASCII
|
||||
character** (swept over `U+0080`–`U+10FFFF`). So for a run character `c`, every element character
|
||||
that can match `c` is itself ASCII, hence stored verbatim, hence foldable by both providers'
|
||||
`LOWER()`. This is *false* for `InvariantCultureIgnoreCase`, which folds ~190 codepoints
|
||||
(`U+00AA`→`a`, `U+017F`→`s`, the modifier letters) onto ASCII letters — **the choice of Ordinal is
|
||||
load-bearing, not stylistic**.
|
||||
2. **The exact set of printable-ASCII characters the JSON writer escapes** is `"`, `&`, `'`, `+`,
|
||||
`<`, `>`, `\`, `` ` ``, pinned against the encoder itself so an encoder change fails the build
|
||||
instead of quietly shrinking the superset.
|
||||
|
||||
## Ordinal everywhere, because the culture is caller-controlled
|
||||
|
||||
`UseRequestLocalization` honours `Accept-Language`, so a caller can select `tr-TR` and turn `q=I`
|
||||
into `ı`. The old chain used `ToLower()` plus the default *linguistic* `StartsWith(string)`, making
|
||||
the same library answer differently per caller. Comparison is now `OrdinalIgnoreCase` and ordering
|
||||
`StringComparer.Ordinal` throughout, including the shared `FilterSortTake` that `state`,
|
||||
`video_dynamic_range` and `content_rating` also use. That is a deliberate change to shared behaviour:
|
||||
response *sets* are unchanged, response *order* is now ordinal rather than culture-dependent.
|
||||
|
||||
## Ordering is best-effort, and the code says so
|
||||
|
||||
Merging sources does **not** yield the exact first `limit` of the union. Each source truncates using
|
||||
its own ordering — the EF source by the database collation (SQLite's is ASCII-only), the list source
|
||||
by primary key — and neither is the ordinal ordering the merge applies. With `"Zulu"` and `"Éclair"`
|
||||
and `limit=1`, the EF source keeps `"Zulu"` and the merge never sees `"Éclair"`, which it would have
|
||||
ranked first. Below the truncation points — the normal typeahead case — the result is exact. An
|
||||
earlier comment claimed exactness the code does not have; do not restore it.
|
||||
|
||||
## What the row cap does and does not bound
|
||||
|
||||
`ListValuedRowCap = 1000` bounds **rows fetched**, not values returned, because the pre-filter
|
||||
matches rows whole. The cap rides `ORDER BY Id` — a unique integer primary key both providers can
|
||||
walk — rather than the JSON column: ordering on `TEXT` would not be deterministic on MySQL, which
|
||||
sorts using only the first `max_sort_length` (default 1024) bytes, so long rows sharing a prefix
|
||||
would tie arbitrarily; and it avoids sorting the whole matching set.
|
||||
|
||||
**It does not bound database work, and that is a known cost.** `LOWER(col) LIKE '%…'` has a leading
|
||||
wildcard, so no index can be sought and every `SongMetadata` row is scanned before the cap applies.
|
||||
An empty `q` matches nearly every row. Every `artist` request now scans the song table's JSON column
|
||||
in addition to its previous work, once per debounced keystroke; the endpoint is authenticated by
|
||||
default but `Api:RequireKeyForReads=false` makes it anonymous. Accepted for now because the scan is
|
||||
bounded in *memory* and the affected table is one of the smaller ones, and because the alternative is
|
||||
the migration below. **Follow-up candidate: a normalized `SongArtist` join table** (the shape
|
||||
`MusicVideoArtist` already has) would make this exact, indexable and seekable — at the cost of a
|
||||
dual-provider schema migration plus data backfill, changes to every scanner write path populating
|
||||
`SongMetadata.Artists`, changes to the Lucene indexer, and two representations of the same fact free
|
||||
to drift. Rejected for #578 on blast radius, not on merit.
|
||||
|
||||
## `album_artist` 404 → 200 is additive
|
||||
|
||||
Nothing consumes the 404 as a signal: the SPA's `getSearchFieldValues` (`web/src/api/search.ts`)
|
||||
treats any non-200 as "no suggestions, fall back to a free-text input", which it will now do less
|
||||
often. Per `api.versioning-v1`, widening which fields return values adds capability without removing
|
||||
any, so no `/api/v2`.
|
||||
|
||||
## Known limitation inherited, not introduced
|
||||
|
||||
The **EF-sourced** fields (`genre`, `studio`, `artist`'s entity half, …) still prefix-match through
|
||||
SQL `LOWER()`, which on SQLite is ASCII-only — so `q=é` does not match a stored `Édith` for those
|
||||
fields either. That predates #578 and is unchanged by it; fixing it would mean filtering those tables
|
||||
client-side. Noted here so the next reader does not mistake the list-valued fix for a global one.
|
||||
@@ -619,12 +619,12 @@ not wired here.
|
||||
for these two fields — the compiled query is the existing `released_inthelast:"7 day"`-style
|
||||
`CustomMultiFieldQueryParser` macro, so nothing downstream changes. `validation.ts`'s `ruleError`
|
||||
requires the value to parse as a positive integer before it's compiled.
|
||||
- **Facet-value typeahead** (#434, `api.search-field-values`) — the value input for a `text` field
|
||||
- **Facet-value typeahead** (#434/#578, `api.search-field-values-sources`) — the value input for a `text` field
|
||||
(not enum) is a combobox backed by `getSearchFieldValues` (`web/src/api/search.ts` →
|
||||
`GET /api/v1/search/fields/{name}/values?q=&limit=`), debounced on keystroke, prefix-matching the
|
||||
in-progress value against distinct terms already in the index. It always allows free-text entry as a
|
||||
fallback — a 404 (non-text field) or an empty result list (e.g. ElasticSearch backend) degrades to a
|
||||
plain text input rather than blocking the rule. Since #578 (`api.search-field-values-list-columns`)
|
||||
plain text input rather than blocking the rule. Since #578 (`api.search-field-values-sources`)
|
||||
`album_artist` returns values instead of 404ing, and `artist` covers free-text music-video/song credits
|
||||
as well as entity artists; for those two the server's list is **bounded best-effort** on a very large
|
||||
library, so the free-text fallback stays load-bearing — never treat an absent suggestion as an invalid
|
||||
|
||||
Reference in New Issue
Block a user