BLOCKER. Attempt 4 bounded the Id KEYSPACE, and keyspace is not rows. Delete 20,000 historical rows, put one song at Id 20001, query artist?q=que: the walk burned all ten windows on empty ranges and returned [] for a table containing exactly one row. Capacity fell linearly with deletion ratio and no ratio was safe — one placed gap hides the next match. My record called that "heavily fragmented" and the endpoint description said loss happens "on a very large library"; the one-row example disproves both. Option A. The query now carries NO predicate at all — no LIKE, no LOWER, not even IS NOT NULL: SELECT Id, Artists AS Payload FROM SongMetadata WHERE Id > @AfterId ORDER BY Id LIMIT @Batch That is the whole fix, and it is the point. LIMIT truncates what survives a predicate, so with any predicate present it bounds the OUTPUT and says nothing about the WORK; the engine may evaluate and discard arbitrarily many rows first. Stripped to a bare primary-key range, LIMIT n reads exactly n index entries and n rows — independent of sparsity, deletion history or where the gaps fall. All selectivity moves into memory. A short page can now only mean exhaustion, which is precisely what it could not mean while a predicate was present. Four attempts, four wrong quantities: the result (a fixed budget the over-matching pre-filter starved), candidates returned (a no-match query must evaluate every eligible row before returning an empty page), keyspace width (above), and finally actual rows. The record carries the table; it is worth more than the code. Deleting the predicate deletes a whole bug family with it: the JSON-escape reasoning, the narrow-only-on-verbatim-ASCII rule, the exhaustive Unicode sweep that proved it sound, the ESCAPE '/' portability workaround, and the may-over-match-never-under-match invariant that turned out to be conditional on something untrue. SearchFieldValuesPrefilterSupersetTests is deleted entirely; the one assertion worth keeping — that the SQL has no predicate — moved to the query-shape suite, which pins the SQL string exactly so "just a cheap filter" fails a test instead of silently unbounding the walk. Measured cost of no server-side narrowing, on a seeded 20,000-song library (in-memory SQLite): worst case (no match, full walk) 20,000 rows / 10 round trips / 391.9 KiB / 119ms SQL, ~40ms warm end-to-end. Empty q, dense and non-ASCII prefixes all stop on page 1 at ~39 KiB and ~40ms. Judged acceptable for a debounced typeahead against a local file. If it ever is not, the answer is #669, not reintroducing selectivity — the record says so explicitly. Also fixed: - Round-trip count was advertised as 10; it is at most 10 for album_artist and 11 for artist, which also runs its EF query. The MAX(Id) probe is gone with the keyspace scheme, so there is no extra scalar call. - The duplicated-formula ceiling test is deleted rather than rewritten. It re-implemented the loop's arithmetic and would have passed through an off-by-one or a stall in the real loop; the dense integration tests carry that coverage. Its MaxVisited >= Window assertion was a style constraint in correctness clothing. - Stale text swept by grepping the mechanism nouns rather than re-reading: candidate/keyspace/ pre-filter/superset/row cap/LIKE/ESCAPE and the removed constant names, across handler, tests, record, api-conventions and the endpoint description. The two surviving "pre-filter" mentions are deliberate history. Test comments that rendered escaped non-ASCII as literal characters (which contradicted the raw-storage assertion in the same file) now show the escape text. New test List_Valued_Walk_Reads_Live_Rows_Regardless_Of_Id_Density reproduces the one-row killer and fails against attempt 4.
252 lines
13 KiB
C#
252 lines
13 KiB
C#
using System.ComponentModel;
|
|
using ErsatzTV.Application.MediaCollections;
|
|
using ErsatzTV.Application.MediaItems;
|
|
using ErsatzTV.Application.Search;
|
|
using ErsatzTV.Application.Search.Queries;
|
|
using ErsatzTV.Core;
|
|
using ErsatzTV.Core.Api.Scheduling;
|
|
using ErsatzTV.Core.Api.Search;
|
|
using ErsatzTV.Extensions;
|
|
using MediatR;
|
|
using Microsoft.AspNetCore.Http;
|
|
using Microsoft.AspNetCore.Mvc;
|
|
|
|
namespace ErsatzTV.Controllers.Api;
|
|
|
|
[ApiController]
|
|
public class SearchController(IMediator mediator) : ControllerBase
|
|
{
|
|
private const int MaxPageSize = 100;
|
|
|
|
// all-items returns bare ids (cheap), so it tolerates a larger page than the item-returning
|
|
// /search endpoint; the clamp still bounds one response to <= 10 kinds * MaxAllItemsPageSize ids
|
|
// so a broad query can't materialize the whole index in one request (issue #293).
|
|
private const int DefaultAllItemsPageSize = 500;
|
|
private const int MaxAllItemsPageSize = 1000;
|
|
|
|
// Upper-bound the page number so pageNum * pageSize (the search skip) can't overflow int and 500 —
|
|
// no legitimate client pages past this, and it keeps skip + limit inside int range at MaxAllItemsPageSize.
|
|
private const int MaxAllItemsPageNum = 2_000_000;
|
|
|
|
[HttpGet("/api/v1/search", Name = "Search")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search library items across all media kinds")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(SearchResultsResponseModel), StatusCodes.Status200OK)]
|
|
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status422UnprocessableEntity)]
|
|
public async Task<IActionResult> Search(
|
|
[FromQuery] string query = "",
|
|
[FromQuery]
|
|
[Description("0-based page index: the first page is 0, not 1. A negative value is clamped to 0.")]
|
|
int pageNum = 0,
|
|
[FromQuery]
|
|
[Description("Rows per page (default 50); capped at 100 for this endpoint. The page offset is derived from the effective (capped) size, so a larger value narrows the page instead of widening the offset.")]
|
|
int pageSize = 50,
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
if (string.IsNullOrWhiteSpace(query))
|
|
{
|
|
return BaseError.New("A non-empty query is required").ToErrorResult();
|
|
}
|
|
|
|
int clampedPageNum = Math.Max(0, pageNum);
|
|
int clampedPageSize = Math.Clamp(pageSize, 1, MaxPageSize);
|
|
SearchResultsResponseModel result = await mediator.Send(
|
|
new GetSearchResults(query, clampedPageNum, clampedPageSize),
|
|
cancellationToken);
|
|
return new OkObjectResult(result);
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/all-items", Name = "SearchAllItems")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search library items across all media kinds and return raw id lists")]
|
|
[EndpointDescription(
|
|
"Returns matching item ids, grouped by media kind, one clamped page per kind plus per-kind " +
|
|
"total counts. Used by the SPA's \"add all to collection/playlist\" flow, which pages to " +
|
|
"completeness. Paging bounds a broad query so it can't materialize the whole index in one " +
|
|
"request (issue #293).")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(SearchResultAllItemsResponseModel), StatusCodes.Status200OK)]
|
|
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status422UnprocessableEntity)]
|
|
public async Task<IActionResult> SearchAllItems(
|
|
[FromQuery] string query = "",
|
|
[FromQuery]
|
|
[Description("0-based page index: the first page is 0, not 1. A negative value is clamped to 0. Unlike the other paged endpoints this one is also bounded ABOVE, at 2000000, so that pageNum * pageSize cannot overflow; a larger value is clamped down to that maximum rather than rejected.")]
|
|
int pageNum = 0,
|
|
[FromQuery]
|
|
[Description("Rows per page (default 500); capped at 1000 for this endpoint. The page offset is derived from the effective (capped) size, so a larger value narrows the page instead of widening the offset.")]
|
|
int pageSize = DefaultAllItemsPageSize,
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
if (string.IsNullOrWhiteSpace(query))
|
|
{
|
|
return BaseError.New("A non-empty query is required").ToErrorResult();
|
|
}
|
|
|
|
int clampedPageNum = Math.Clamp(pageNum, 0, MaxAllItemsPageNum);
|
|
int clampedPageSize = Math.Clamp(pageSize, 1, MaxAllItemsPageSize);
|
|
SearchResultAllItemsViewModel result = await mediator.Send(
|
|
new QuerySearchIndexAllItems(query, clampedPageNum, clampedPageSize),
|
|
cancellationToken);
|
|
return new OkObjectResult(Project(result));
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/collections", Name = "SearchCollections")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search collections by name")]
|
|
[EndpointDescription("Returns matching collections as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchCollections(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<MediaCollectionViewModel> results = await mediator.Send(
|
|
new SearchCollections(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/television-shows", Name = "SearchTelevisionShows")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search television shows by name")]
|
|
[EndpointDescription("Returns matching television shows as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchTelevisionShows(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<NamedMediaItemViewModel> results = await mediator.Send(
|
|
new SearchTelevisionShows(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(s => new SchedulingPickerOptionResponseModel(s.MediaItemId, s.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/television-seasons", Name = "SearchTelevisionSeasons")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search television seasons by name")]
|
|
[EndpointDescription("Returns matching television seasons as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchTelevisionSeasons(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<NamedMediaItemViewModel> results = await mediator.Send(
|
|
new SearchTelevisionSeasons(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(s => new SchedulingPickerOptionResponseModel(s.MediaItemId, s.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/smart-collections", Name = "SearchSmartCollections")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search smart collections by name")]
|
|
[EndpointDescription("Returns matching smart collections as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchSmartCollections(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<SmartCollectionViewModel> results = await mediator.Send(
|
|
new SearchSmartCollections(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/artists", Name = "SearchArtists")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search artists by name")]
|
|
[EndpointDescription("Returns matching artists as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchArtists(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<NamedMediaItemViewModel> results = await mediator.Send(
|
|
new SearchArtists(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(a => new SchedulingPickerOptionResponseModel(a.MediaItemId, a.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/multi-collections", Name = "SearchMultiCollections")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("Search multi collections by name")]
|
|
[EndpointDescription("Returns matching multi collections as {id, name} options for scheduling editors.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SchedulingPickerOptionResponseModel>), StatusCodes.Status200OK)]
|
|
public async Task<List<SchedulingPickerOptionResponseModel>> SearchMultiCollections(
|
|
[FromQuery] string query = "",
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
List<MultiCollectionViewModel> results = await mediator.Send(
|
|
new SearchMultiCollections(query ?? string.Empty),
|
|
cancellationToken);
|
|
return results.Map(c => new SchedulingPickerOptionResponseModel(c.Id, c.Name)).ToList();
|
|
}
|
|
|
|
[HttpGet("/api/v1/search/fields", Name = "GetSearchFields")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("List the filterable fields for the visual rule builder")]
|
|
[EndpointDescription(
|
|
"Returns the curated catalog of searchable fields (name, friendly label, type, UI group, and " +
|
|
"allowed values for enum fields). Drives the SmartCollection rule builder and is introspectable by MCP.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(List<SearchFieldResponseModel>), StatusCodes.Status200OK)]
|
|
public Task<List<SearchFieldResponseModel>> GetSearchFields(CancellationToken cancellationToken) =>
|
|
mediator.Send(new GetSearchFieldCatalog(), cancellationToken);
|
|
|
|
[HttpGet("/api/v1/search/fields/{name}/values", Name = "GetSearchFieldValues")]
|
|
[Tags("Search")]
|
|
[EndpointSummary("List distinct database values for a text search field")]
|
|
[EndpointDescription(
|
|
"Returns distinct whole values from the database for the given text field, filtered by an " +
|
|
"optional case-insensitive prefix. Powers the visual rule builder's facet-value typeahead. " +
|
|
"404 when the field is unknown, is not a text field, or is a text field with no distinct-value " +
|
|
"source. The final filter, dedup and ordering applied to the response are ordinal and not " +
|
|
"culture-dependent; note that fields sourced by a plain database query are additionally " +
|
|
"pre-filtered by the database collation first, which on SQLite is ASCII-only. The list-valued " +
|
|
"music fields (artist, album_artist) are bounded best-effort: the server reads a bounded number " +
|
|
"of song rows per request, so a library larger than that bound may yield a subset of the matches.")]
|
|
[EndpointGroupName("general")]
|
|
[ProducesResponseType(typeof(SearchFieldValuesResponseModel), StatusCodes.Status200OK)]
|
|
[ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status404NotFound)]
|
|
public async Task<IActionResult> GetSearchFieldValues(
|
|
[FromRoute] string name,
|
|
[FromQuery] string q = "",
|
|
[FromQuery] int limit = 50,
|
|
CancellationToken cancellationToken = default)
|
|
{
|
|
Option<SearchFieldValuesResponseModel> result = await mediator.Send(
|
|
new GetSearchFieldValues(name, q ?? string.Empty, limit),
|
|
cancellationToken);
|
|
return result.ToGetResult();
|
|
}
|
|
|
|
private static SearchResultAllItemsResponseModel Project(SearchResultAllItemsViewModel vm) =>
|
|
new(
|
|
vm.MovieIds,
|
|
vm.ShowIds,
|
|
vm.SeasonIds,
|
|
vm.EpisodeIds,
|
|
vm.ArtistIds,
|
|
vm.MusicVideoIds,
|
|
vm.OtherVideoIds,
|
|
vm.SongIds,
|
|
vm.ImageIds,
|
|
vm.RemoteStreamIds,
|
|
new SearchResultAllItemsTotalsResponseModel(
|
|
vm.Totals.MovieCount,
|
|
vm.Totals.ShowCount,
|
|
vm.Totals.SeasonCount,
|
|
vm.Totals.EpisodeCount,
|
|
vm.Totals.ArtistCount,
|
|
vm.Totals.MusicVideoCount,
|
|
vm.Totals.OtherVideoCount,
|
|
vm.Totals.SongCount,
|
|
vm.Totals.ImageCount,
|
|
vm.Totals.RemoteStreamCount));
|
|
}
|