Files
ersatztv/ErsatzTV/Controllers/Api/SearchController.cs
T
timothy 373956fcee fix(578): delete the SQL predicate — LIMIT only bounds work when there is nothing to discard
BLOCKER. Attempt 4 bounded the Id KEYSPACE, and keyspace is not rows. Delete 20,000 historical rows,
put one song at Id 20001, query artist?q=que: the walk burned all ten windows on empty ranges and
returned [] for a table containing exactly one row. Capacity fell linearly with deletion ratio and no
ratio was safe — one placed gap hides the next match. My record called that "heavily fragmented" and
the endpoint description said loss happens "on a very large library"; the one-row example disproves
both.

Option A. The query now carries NO predicate at all — no LIKE, no LOWER, not even IS NOT NULL:

  SELECT Id, Artists AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch

That is the whole fix, and it is the point. LIMIT truncates what survives a predicate, so with any
predicate present it bounds the OUTPUT and says nothing about the WORK; the engine may evaluate and
discard arbitrarily many rows first. Stripped to a bare primary-key range, LIMIT n reads exactly n
index entries and n rows — independent of sparsity, deletion history or where the gaps fall. All
selectivity moves into memory. A short page can now only mean exhaustion, which is precisely what it
could not mean while a predicate was present.

Four attempts, four wrong quantities: the result (a fixed budget the over-matching pre-filter
starved), candidates returned (a no-match query must evaluate every eligible row before returning an
empty page), keyspace width (above), and finally actual rows. The record carries the table; it is
worth more than the code.

Deleting the predicate deletes a whole bug family with it: the JSON-escape reasoning, the
narrow-only-on-verbatim-ASCII rule, the exhaustive Unicode sweep that proved it sound, the ESCAPE '/'
portability workaround, and the may-over-match-never-under-match invariant that turned out to be
conditional on something untrue. SearchFieldValuesPrefilterSupersetTests is deleted entirely; the one
assertion worth keeping — that the SQL has no predicate — moved to the query-shape suite, which pins
the SQL string exactly so "just a cheap filter" fails a test instead of silently unbounding the walk.

Measured cost of no server-side narrowing, on a seeded 20,000-song library (in-memory SQLite):
worst case (no match, full walk) 20,000 rows / 10 round trips / 391.9 KiB / 119ms SQL, ~40ms warm
end-to-end. Empty q, dense and non-ASCII prefixes all stop on page 1 at ~39 KiB and ~40ms. Judged
acceptable for a debounced typeahead against a local file. If it ever is not, the answer is #669, not
reintroducing selectivity — the record says so explicitly.

Also fixed:
- Round-trip count was advertised as 10; it is at most 10 for album_artist and 11 for artist, which
  also runs its EF query. The MAX(Id) probe is gone with the keyspace scheme, so there is no extra
  scalar call.
- The duplicated-formula ceiling test is deleted rather than rewritten. It re-implemented the loop's
  arithmetic and would have passed through an off-by-one or a stall in the real loop; the dense
  integration tests carry that coverage. Its MaxVisited >= Window assertion was a style constraint in
  correctness clothing.
- Stale text swept by grepping the mechanism nouns rather than re-reading: candidate/keyspace/
  pre-filter/superset/row cap/LIKE/ESCAPE and the removed constant names, across handler, tests,
  record, api-conventions and the endpoint description. The two surviving "pre-filter" mentions are
  deliberate history. Test comments that rendered escaped non-ASCII as literal characters (which
  contradicted the raw-storage assertion in the same file) now show the escape text.

New test List_Valued_Walk_Reads_Live_Rows_Regardless_Of_Id_Density reproduces the one-row killer and
fails against attempt 4.
2026-07-27 03:10:36 +02:00

252 lines
13 KiB
C#

using System.ComponentModel;
using ErsatzTV.Application.MediaCollections;
using ErsatzTV.Application.MediaItems;
using ErsatzTV.Application.Search;
using ErsatzTV.Application.Search.Queries;
using ErsatzTV.Core;
using ErsatzTV.Core.Api.Scheduling;
using ErsatzTV.Core.Api.Search;
using ErsatzTV.Extensions;
using MediatR;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
namespace ErsatzTV.Controllers.Api;
[ApiController]
public class SearchController(IMediator mediator) : ControllerBase
{
private const int MaxPageSize = 100;
// all-items returns bare ids (cheap), so it tolerates a larger page than the item-returning
// /search endpoint; the clamp still bounds one response to <= 10 kinds * MaxAllItemsPageSize ids
// so a broad query can't materialize the whole index in one request (issue #293).
private const int DefaultAllItemsPageSize = 500;
private const int MaxAllItemsPageSize = 1000;
// Upper-bound the page number so pageNum * pageSize (the search skip) can't overflow int and 500 —
// no legitimate client pages past this, and it keeps skip + limit inside int range at MaxAllItemsPageSize.
private const int MaxAllItemsPageNum = 2_000_000;
[HttpGet("/api/v1/search", Name = "Search")]
[Tags("Search")]
[EndpointSummary("Search library items across all media kinds")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(SearchResultsResponseModel), StatusCodes.Status200OK)]
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status422UnprocessableEntity)]
public async Task<IActionResult> Search(
[FromQuery] string query = "",
[FromQuery]
[Description("0-based page index: the first page is 0, not 1. A negative value is clamped to 0.")]
int pageNum = 0,
[FromQuery]
[Description("Rows per page (default 50); capped at 100 for this endpoint. The page offset is derived from the effective (capped) size, so a larger value narrows the page instead of widening the offset.")]
int pageSize = 50,
CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(query))
{
return BaseError.New("A non-empty query is required").ToErrorResult();
}
int clampedPageNum = Math.Max(0, pageNum);
int clampedPageSize = Math.Clamp(pageSize, 1, MaxPageSize);
SearchResultsResponseModel result = await mediator.Send(
new GetSearchResults(query, clampedPageNum, clampedPageSize),
cancellationToken);
return new OkObjectResult(result);
}
[HttpGet("/api/v1/search/all-items", Name = "SearchAllItems")]
[Tags("Search")]
[EndpointSummary("Search library items across all media kinds and return raw id lists")]
[EndpointDescription(
"Returns matching item ids, grouped by media kind, one clamped page per kind plus per-kind " +
"total counts. Used by the SPA's \"add all to collection/playlist\" flow, which pages to " +
"completeness. Paging bounds a broad query so it can't materialize the whole index in one " +
"request (issue #293).")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(SearchResultAllItemsResponseModel), StatusCodes.Status200OK)]
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status422UnprocessableEntity)]
public async Task<IActionResult> SearchAllItems(
[FromQuery] string query = "",
[FromQuery]
[Description("0-based page index: the first page is 0, not 1. A negative value is clamped to 0. Unlike the other paged endpoints this one is also bounded ABOVE, at 2000000, so that pageNum * pageSize cannot overflow; a larger value is clamped down to that maximum rather than rejected.")]
int pageNum = 0,
[FromQuery]
[Description("Rows per page (default 500); capped at 1000 for this endpoint. The page offset is derived from the effective (capped) size, so a larger value narrows the page instead of widening the offset.")]
int pageSize = DefaultAllItemsPageSize,
CancellationToken cancellationToken = default)
{
if (string.IsNullOrWhiteSpace(query))
{
return BaseError.New("A non-empty query is required").ToErrorResult();
}
int clampedPageNum = Math.Clamp(pageNum, 0, MaxAllItemsPageNum);
int clampedPageSize = Math.Clamp(pageSize, 1, MaxAllItemsPageSize);
SearchResultAllItemsViewModel result = await mediator.Send(
new QuerySearchIndexAllItems(query, clampedPageNum, clampedPageSize),
cancellationToken);
return new OkObjectResult(Project(result));
}
[HttpGet("/api/v1/search/collections", Name = "SearchCollections")]
[Tags("Search")]
[EndpointSummary("Search collections by name")]
[EndpointDescription("Returns matching collections as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchCollections(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<MediaCollectionViewModel> results = await mediator.Send(
new SearchCollections(query ?? string.Empty),
cancellationToken);
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
}
[HttpGet("/api/v1/search/television-shows", Name = "SearchTelevisionShows")]
[Tags("Search")]
[EndpointSummary("Search television shows by name")]
[EndpointDescription("Returns matching television shows as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchTelevisionShows(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<NamedMediaItemViewModel> results = await mediator.Send(
new SearchTelevisionShows(query ?? string.Empty),
cancellationToken);
return results.Map(s => new SchedulingPickerOptionResponseModel(s.MediaItemId, s.Name)).ToList();
}
[HttpGet("/api/v1/search/television-seasons", Name = "SearchTelevisionSeasons")]
[Tags("Search")]
[EndpointSummary("Search television seasons by name")]
[EndpointDescription("Returns matching television seasons as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchTelevisionSeasons(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<NamedMediaItemViewModel> results = await mediator.Send(
new SearchTelevisionSeasons(query ?? string.Empty),
cancellationToken);
return results.Map(s => new SchedulingPickerOptionResponseModel(s.MediaItemId, s.Name)).ToList();
}
[HttpGet("/api/v1/search/smart-collections", Name = "SearchSmartCollections")]
[Tags("Search")]
[EndpointSummary("Search smart collections by name")]
[EndpointDescription("Returns matching smart collections as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchSmartCollections(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<SmartCollectionViewModel> results = await mediator.Send(
new SearchSmartCollections(query ?? string.Empty),
cancellationToken);
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
}
[HttpGet("/api/v1/search/artists", Name = "SearchArtists")]
[Tags("Search")]
[EndpointSummary("Search artists by name")]
[EndpointDescription("Returns matching artists as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchArtists(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<NamedMediaItemViewModel> results = await mediator.Send(
new SearchArtists(query ?? string.Empty),
cancellationToken);
return results.Map(a => new SchedulingPickerOptionResponseModel(a.MediaItemId, a.Name)).ToList();
}
[HttpGet("/api/v1/search/multi-collections", Name = "SearchMultiCollections")]
[Tags("Search")]
[EndpointSummary("Search multi collections by name")]
[EndpointDescription("Returns matching multi collections as {id, name} options for scheduling editors.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
public async Task<List<SchedulingPickerOptionResponseModel>> SearchMultiCollections(
[FromQuery] string query = "",
CancellationToken cancellationToken = default)
{
List<MultiCollectionViewModel> results = await mediator.Send(
new SearchMultiCollections(query ?? string.Empty),
cancellationToken);
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
}
[HttpGet("/api/v1/search/fields", Name = "GetSearchFields")]
[Tags("Search")]
[EndpointSummary("List the filterable fields for the visual rule builder")]
[EndpointDescription(
"Returns the curated catalog of searchable fields (name, friendly label, type, UI group, and " +
"allowed values for enum fields). Drives the SmartCollection rule builder and is introspectable by MCP.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(List<SearchFieldResponseModel>), StatusCodes.Status200OK)]
public Task<List<SearchFieldResponseModel>> GetSearchFields(CancellationToken cancellationToken) =>
mediator.Send(new GetSearchFieldCatalog(), cancellationToken);
[HttpGet("/api/v1/search/fields/{name}/values", Name = "GetSearchFieldValues")]
[Tags("Search")]
[EndpointSummary("List distinct database values for a text search field")]
[EndpointDescription(
"Returns distinct whole values from the database for the given text field, filtered by an " +
"optional case-insensitive prefix. Powers the visual rule builder's facet-value typeahead. " +
"404 when the field is unknown, is not a text field, or is a text field with no distinct-value " +
"source. The final filter, dedup and ordering applied to the response are ordinal and not " +
"culture-dependent; note that fields sourced by a plain database query are additionally " +
"pre-filtered by the database collation first, which on SQLite is ASCII-only. The list-valued " +
"music fields (artist, album_artist) are bounded best-effort: the server reads a bounded number " +
"of song rows per request, so a library larger than that bound may yield a subset of the matches.")]
[EndpointGroupName("general")]
[ProducesResponseType(typeof(SearchFieldValuesResponseModel), StatusCodes.Status200OK)]
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status404NotFound)]
public async Task<IActionResult> GetSearchFieldValues(
[FromRoute] string name,
[FromQuery] string q = "",
[FromQuery] int limit = 50,
CancellationToken cancellationToken = default)
{
Option<SearchFieldValuesResponseModel> result = await mediator.Send(
new GetSearchFieldValues(name, q ?? string.Empty, limit),
cancellationToken);
return result.ToGetResult();
}
private static SearchResultAllItemsResponseModel Project(SearchResultAllItemsViewModel vm) =>
new(
vm.MovieIds,
vm.ShowIds,
vm.SeasonIds,
vm.EpisodeIds,
vm.ArtistIds,
vm.MusicVideoIds,
vm.OtherVideoIds,
vm.SongIds,
vm.ImageIds,
vm.RemoteStreamIds,
new SearchResultAllItemsTotalsResponseModel(
vm.Totals.MovieCount,
vm.Totals.ShowCount,
vm.Totals.SeasonCount,
vm.Totals.EpisodeCount,
vm.Totals.ArtistCount,
vm.Totals.MusicVideoCount,
vm.Totals.OtherVideoCount,
vm.Totals.SongCount,
vm.Totals.ImageCount,
vm.Totals.RemoteStreamCount));
}