Seven more SPA list loads silently truncate at the server cap (same class as #634, one screen over) #644

Closed
opened 2026-07-26 11:16:48 +02:00 by timothy · 2 comments
Owner

Found by the cold review of the #634 PR, and confirmed by grep. #634 fixed one call site; the same defect shape survives at seven more, all in web/src/screens/.

Each asks for a pageSize far above what the endpoint actually clamps to, takes result.page as if it were the whole list, and has no paging UI — so past the cap the rows simply vanish. No error, no truncation indicator. api.paging-zero-based is explicit that the offset derives from the effective (capped) size, so the over-large pageSize buys nothing at all.

Call site Endpoint Asks for Server cap
RerunCollectionsScreen.tsx:136 /api/v1/rerun-collections 1000 100
RerunCollectionsScreen.tsx:105 /api/v1/multi-collections 1000 100
RerunCollectionsScreen.tsx:111 /api/v1/library/browse 500 100
MultiCollectionsScreen.tsx:40 /api/v1/multi-collections 1000 100
PlaylistsScreen.tsx:152 /api/v1/multi-collections 1000 100
PlaylistsScreen.tsx:158 /api/v1/library/browse 500 100
FillerPresetsScreen.tsx:489 /api/v1/library/browse 500 100

Why this is worth its own issue

RerunCollectionsScreen.tsx:136 is the sharp one: #634 made the schedule picker able to reference rerun collection #120, while the screen that creates and edits rerun collections still cannot show it. Shipping one without the other leaves a state a user can reach but not manage.

Fix

Same pattern #634 used: page to completeness against totalCount, mirroring CollectionsScreen.enterReorder. Do not raise the server caps — api.search-allitems-paging set the precedent that the client pages and the server stays bounded.

Worth extracting the loop once rather than writing it a seventh time; #634's loadAllRerunCollections is a reasonable starting shape but is currently local to SchedulesScreen.

Done-when

  • Each of the seven call sites is no longer a SILENT truncation - see the scope amendment below (completeness for the four bounded lists; a bounded load with a visible truncation hint for the three library/browse pickers, tracked to #651)
  • The paging loop exists in ONE shared place rather than copied per screen
  • A test covers the >cap case for at least the getMultiCollections and getLibraryBrowseItems loaders, pinning the exact expected set (not a non-empty check)
  • A grep confirms no remaining pageSize: above an endpoint's cap in web/src
  • Adversarial review passed

Scope amendment (2026-07-26, after the cold review)

The original box "returns the complete list regardless of count" is withdrawn for three of the
seven sites
, because doing it would have been worse than the bug.

RerunCollectionsScreen.tsx:111, PlaylistsScreen.tsx:158 and FillerPresetsScreen.tsx:489 are
getLibraryBrowseItems pickers whose mediaType can be Episode/Song/Image/Movie/
MusicVideo. Paging those to completeness on a large library means ~200 serial requests on mount,
each more expensive than the last (LuceneSearchIndex.Search collects skip + limit sorted
top-docs), feeding tens of thousands of <option> nodes into a native <select>.

The defect this issue actually names is silent truncation. So those three keep a bounded load
and gain a visible truncation hint; making them genuinely usable needs a searchable source, which
is #651. The four bounded, admin-created lists do page to completeness as planned.

Also filed from this review: #650 (two more at-cap truncations the over-cap grep could not see).

Found by the cold review of the #634 PR, and confirmed by grep. #634 fixed **one** call site; the same defect shape survives at seven more, all in `web/src/screens/`. Each asks for a `pageSize` far above what the endpoint actually clamps to, takes `result.page` as if it were the whole list, and has no paging UI — so past the cap the rows simply vanish. No error, no truncation indicator. `api.paging-zero-based` is explicit that the offset derives from the **effective** (capped) size, so the over-large `pageSize` buys nothing at all. | Call site | Endpoint | Asks for | Server cap | |---|---|---|---| | `RerunCollectionsScreen.tsx:136` | `/api/v1/rerun-collections` | 1000 | 100 | | `RerunCollectionsScreen.tsx:105` | `/api/v1/multi-collections` | 1000 | 100 | | `RerunCollectionsScreen.tsx:111` | `/api/v1/library/browse` | 500 | 100 | | `MultiCollectionsScreen.tsx:40` | `/api/v1/multi-collections` | 1000 | 100 | | `PlaylistsScreen.tsx:152` | `/api/v1/multi-collections` | 1000 | 100 | | `PlaylistsScreen.tsx:158` | `/api/v1/library/browse` | 500 | 100 | | `FillerPresetsScreen.tsx:489` | `/api/v1/library/browse` | 500 | 100 | ## Why this is worth its own issue `RerunCollectionsScreen.tsx:136` is the sharp one: #634 made the *schedule picker* able to reference rerun collection #120, while the screen that **creates and edits** rerun collections still cannot show it. Shipping one without the other leaves a state a user can reach but not manage. ## Fix Same pattern #634 used: page to completeness against `totalCount`, mirroring `CollectionsScreen.enterReorder`. Do **not** raise the server caps — `api.search-allitems-paging` set the precedent that the client pages and the server stays bounded. Worth extracting the loop once rather than writing it a seventh time; #634's `loadAllRerunCollections` is a reasonable starting shape but is currently local to `SchedulesScreen`. ## Done-when - [x] Each of the seven call sites is no longer a SILENT truncation - see the scope amendment below (completeness for the four bounded lists; a bounded load with a visible truncation hint for the three `library/browse` pickers, tracked to #651) - [x] The paging loop exists in ONE shared place rather than copied per screen - [x] A test covers the >cap case for at least the `getMultiCollections` and `getLibraryBrowseItems` loaders, pinning the exact expected set (not a non-empty check) - [x] A grep confirms no remaining `pageSize:` above an endpoint's cap in `web/src` - [x] Adversarial review passed --- ## Scope amendment (2026-07-26, after the cold review) The original box "returns the complete list regardless of count" is **withdrawn for three of the seven sites**, because doing it would have been worse than the bug. `RerunCollectionsScreen.tsx:111`, `PlaylistsScreen.tsx:158` and `FillerPresetsScreen.tsx:489` are `getLibraryBrowseItems` pickers whose `mediaType` can be `Episode`/`Song`/`Image`/`Movie`/ `MusicVideo`. Paging those to completeness on a large library means ~200 serial requests on mount, each more expensive than the last (`LuceneSearchIndex.Search` collects `skip + limit` sorted top-docs), feeding tens of thousands of `<option>` nodes into a native `<select>`. The defect this issue actually names is **silent** truncation. So those three keep a bounded load and gain a *visible* truncation hint; making them genuinely usable needs a searchable source, which is #651. The four bounded, admin-created lists do page to completeness as planned. Also filed from this review: **#650** (two more at-cap truncations the over-cap grep could not see).
timothy added the bugfrontendpriority: medium labels 2026-07-26 11:17:14 +02:00
timothy added the in-progress label 2026-07-26 20:22:59 +02:00
Author
Owner

Claiming — picked up as the top-ranked eligible backlog issue by scripts/select-queue.sh (prio-medium, no open blockers).

Claiming — picked up as the top-ranked eligible backlog issue by `scripts/select-queue.sh` (prio-medium, no open blockers).
Author
Owner

Closing record

Outcome: Merged as PR #656. web/src/api/paging.ts (loadAllPages) is the single paging
implementation; #634's loop in SchedulesScreen is now a delegate, so no copy survives. The seven
call sites are split into two classes rather than all paged to completeness — see Root cause.

Root cause: Two distinct causes, and only the first is what the issue described.

  1. The reported defect. Each site asked for a pageSize far above the endpoint's clamp
    (MaxPageSize = 100, confirmed by reading RerunCollectionController, MultiCollectionController
    and LibraryBrowseController — not taken from the issue), then treated result.page as the whole
    list with no paging UI. The inflated pageSize bought nothing, because api.paging-zero-based
    derives the offset from the effective (capped) size.

  2. Why the prescribed fix was wrong for three of them. The issue's "page all seven to completeness"
    instruction did not account for what those three pickers are over. RerunCollectionsScreen:111,
    PlaylistsScreen:158 and FillerPresetsScreen:489 call getLibraryBrowseItems with a mediaType
    that can be Episode/Song/Image/Movie/MusicVideo — the largest tables in an install — and
    feed a native <select>. Paging those to completeness on a 20k-episode library means ~200 serial
    requests on mount, each more expensive than the last (LuceneSearchIndex.Search computes
    hitsLimit = skip + limit and collects that many sorted top-docs, then runs a second search for the
    page map), rendering tens of thousands of <option> nodes. That is worse than the truncation.

    The defect the issue actually names is silent truncation. So those three keep a bounded page and
    gained a visible truncation hint; the first Done-when box was amended rather than ticked as written.

Decisions/conventions changed: added spa.list-completeness-vs-bounded-pickers — page to
completeness only for lists bounded by construction; a picker over a media library stays bounded and
shows its truncation.

Reusable knowledge:

  • A completeness guard that keys on the symptom misses the defect. This PR added a grep for
    pageSize: above an endpoint's cap. Two more truncations request exactly the cap and were
    therefore structurally invisible to it (#650) — TraktListsScreen renders "101 lists" above 100
    rows; ChannelBuilder.loadCollections returns totalCount: merged.length, so canLoadMore can
    never fire. The defect is "treating one page as the whole list", not "an inflated number".
  • The fix for a silent-truncation bug reintroduced silent truncation one layer down. The paging
    helper's empty-page break returned a partial list indistinguishable from a complete one — and it
    is reachable in normal operation, since HydrateMediaItems drops Lucene hits whose DB rows
    vanished and the handler's own comment admits totalCount can be stale. Now { items, complete }.
  • One boolean carrying two meanings produced a self-contradictory UI: an incomplete Class A load
    rendered "Showing the first 47 of 47 — use search to narrow", pointing at a search box that does not
    exist for that picker. Split into a hint: 'truncated' | 'incomplete' | 'none' discriminator.
  • A multi-request loop invalidates guards that were adequate for one request. activeRef
    ("still mounted") never answered "still current"; the window widened from one RTT to N, so a stale
    loop could resurrect deleted rows. Needed seqRef + AbortController.

Verification: 107 test files / 1019 vitest tests, npm run lint clean, npm run build
(tsc -b && vite build) clean, decisions-validate: OK. Full CI green on daedf003 (Build & test,
EF migration integrity, Functional E2E, Formatting, API docs in sync, script tests, decisions
lifecycle). Three cold adversarial review rounds; the round-2 reviewer ran a negative control on the
generic-typing fix, reverting the conditional rest-tuple and confirming tsc then reports
TS2578: Unused '@ts-expect-error' directive — proving that guard is load-bearing, not decorative.

Deferred:

  • #651 — the three library/browse pickers need a searchable source; a bounded 100-row window is
    usable only by accident. The query param already exists and two other call sites use it.
  • #650 — the two at-cap truncations above.
  • The AbortSignal stops the loop issuing further pages; it is not threaded into the domain API
    loaders, so an already-in-flight request still completes. Threading it would touch every module in
    web/src/api/. The docs say "stops issuing further page requests", which is accurate.

Docs updated: docs/spa-conventions.md (§3b, rewritten for the Class A/B split and the hint
discriminator), docs/decisions/records/spa/list-completeness-vs-bounded-pickers.md (new),
docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Merged as PR #656. `web/src/api/paging.ts` (`loadAllPages`) is the single paging implementation; #634's loop in `SchedulesScreen` is now a delegate, so no copy survives. The seven call sites are split into two classes rather than all paged to completeness — see Root cause. **Root cause:** Two distinct causes, and only the first is what the issue described. 1. *The reported defect.* Each site asked for a `pageSize` far above the endpoint's clamp (`MaxPageSize = 100`, confirmed by reading `RerunCollectionController`, `MultiCollectionController` and `LibraryBrowseController` — not taken from the issue), then treated `result.page` as the whole list with no paging UI. The inflated `pageSize` bought nothing, because `api.paging-zero-based` derives the offset from the *effective* (capped) size. 2. *Why the prescribed fix was wrong for three of them.* The issue's "page all seven to completeness" instruction did not account for what those three pickers are over. `RerunCollectionsScreen:111`, `PlaylistsScreen:158` and `FillerPresetsScreen:489` call `getLibraryBrowseItems` with a `mediaType` that can be `Episode`/`Song`/`Image`/`Movie`/`MusicVideo` — the largest tables in an install — and feed a native `<select>`. Paging those to completeness on a 20k-episode library means ~200 serial requests on mount, each more expensive than the last (`LuceneSearchIndex.Search` computes `hitsLimit = skip + limit` and collects that many sorted top-docs, then runs a second search for the page map), rendering tens of thousands of `<option>` nodes. That is worse than the truncation. The defect the issue actually names is **silent** truncation. So those three keep a bounded page and gained a visible truncation hint; the first Done-when box was amended rather than ticked as written. **Decisions/conventions changed:** added `spa.list-completeness-vs-bounded-pickers` — page to completeness only for lists bounded by construction; a picker over a media library stays bounded and shows its truncation. **Reusable knowledge:** - **A completeness guard that keys on the symptom misses the defect.** This PR added a grep for `pageSize:` *above* an endpoint's cap. Two more truncations request *exactly* the cap and were therefore structurally invisible to it (#650) — `TraktListsScreen` renders "101 lists" above 100 rows; `ChannelBuilder.loadCollections` returns `totalCount: merged.length`, so `canLoadMore` can never fire. The defect is "treating one page as the whole list", not "an inflated number". - **The fix for a silent-truncation bug reintroduced silent truncation one layer down.** The paging helper's empty-page `break` returned a partial list indistinguishable from a complete one — and it is reachable in *normal* operation, since `HydrateMediaItems` drops Lucene hits whose DB rows vanished and the handler's own comment admits `totalCount` can be stale. Now `{ items, complete }`. - **One boolean carrying two meanings produced a self-contradictory UI**: an incomplete Class A load rendered "Showing the first 47 of 47 — use search to narrow", pointing at a search box that does not exist for that picker. Split into a `hint: 'truncated' | 'incomplete' | 'none'` discriminator. - **A multi-request loop invalidates guards that were adequate for one request.** `activeRef` ("still mounted") never answered "still current"; the window widened from one RTT to N, so a stale loop could resurrect deleted rows. Needed `seqRef` + `AbortController`. **Verification:** 107 test files / 1019 vitest tests, `npm run lint` clean, `npm run build` (`tsc -b && vite build`) clean, `decisions-validate: OK`. Full CI green on `daedf003` (Build & test, EF migration integrity, Functional E2E, Formatting, API docs in sync, script tests, decisions lifecycle). Three cold adversarial review rounds; the round-2 reviewer ran a negative control on the generic-typing fix, reverting the conditional rest-tuple and confirming `tsc` then reports `TS2578: Unused '@ts-expect-error' directive` — proving that guard is load-bearing, not decorative. **Deferred:** - #651 — the three `library/browse` pickers need a *searchable* source; a bounded 100-row window is usable only by accident. The `query` param already exists and two other call sites use it. - #650 — the two at-cap truncations above. - The `AbortSignal` stops the loop issuing *further* pages; it is not threaded into the domain API loaders, so an already-in-flight request still completes. Threading it would touch every module in `web/src/api/`. The docs say "stops issuing further page requests", which is accurate. **Docs updated:** `docs/spa-conventions.md` (§3b, rewritten for the Class A/B split and the `hint` discriminator), `docs/decisions/records/spa/list-completeness-vs-bounded-pickers.md` (new), `docs/decisions/README.md` (regenerated).
timothy removed the in-progress label 2026-07-26 22:12:37 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#644