feat(610): split the decision corpus into one YAML-frontmatter file per record
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Failing after 23s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 1m17s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m29s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m5s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 16m5s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m6s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped

168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and
docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key,
so one-active-record-per-key becomes a filesystem property rather than a
validator check, and supersession becomes a `git mv`.

WHY: the monolith was a concurrency problem before an aesthetic one. A
3,900-line append target made parallel sessions collide -- PR #605 and PR #614
both hit append-vs-append conflicts during routine rebases, and hand-resolving
those inside the corpus is exactly the operation the rationale-rewrite guard
exists to police.

HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness
does not rest on reading it. The parser was taught BOTH formats first, so the
body-diff guard parses the old form at the merge-base and the new form at head --
the migration validates itself, no bypass. The proof is a field-level equivalence
harness: 168 records before and after, zero lost, zero gained, zero field
mismatches, zero rationale bodies differing. Reviewers should scrutinise the
harness; it is the actual evidence.

What measuring caught that reading would not have:

- ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each
  topic file's preamble, mostly the only copy. Source files are kept and
  stripped, never deleted. They also cannot be filed per-area: topic files hold
  several areas and 4 of 23 areas span several files.
- Archive discovery was a non-recursive glob; after the split it found ZERO
  archived records, surfacing as four bogus "supersedes points to unknown key"
  errors rather than an obvious failure.
- ~32 live docs point into the corpus BY DATE, which the split dangles. Each
  stripped file now ends with a generated "Records formerly in this file" index,
  which also rescues the identical breadcrumbs in old issue comments.
- decisions.md's "In this file:" list was 97 same-file anchor bullets that the
  split makes WRONG, not merely stale. Dropped; the generated index replaces
  them with links that resolve.

The equivalence harness now runs against a checked-in FIXTURE, not the live
corpus. The earlier version migrated the real tree, which made it a one-shot:
the moment the migration landed there was nothing left to move and the tests
failed for reasons unrelated to the code. A fixture keeps them testing the
SCRIPT rather than the repo's current state.

Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain
two directories for one concept. Renaming a key is not a move -- it changes
identity, breaks the equivalence proof, and invalidates MemPalace's per-key
drawers. Taxonomy normalisation is separate work.

refs #610
This commit is contained in:
2026-07-25 19:45:09 +02:00
parent 8578dc1ca7
commit fba5233caf
190 changed files with 7193 additions and 5987 deletions
@@ -0,0 +1,124 @@
---
key: scan.projection-failure-sweep-guard
title: 2026-07-25 — A media-server sweep also refuses when the api client silently dropped items whose projection threw; the ratio threshold is rejected (#484)
status: active
since: '2026-07-25'
supersedes: none
superseded-by: none
rule: '`MediaServerReconciliationGuard` takes a per-enumeration projection-failure count and refuses the file-not-found sweep (logged loudly) whenever it is non-zero against a non-empty existing set — at all six sweeps, including the nested per-show season and per-season episode ones #477 left unguarded (via `ShouldFlagMissingDescendants`, which applies the failure refusal but not #477''s empty-fetch branch). Deliberate guard-clause skips (STRM, virtual, unsupported type) are explicitly **not** failures and never suppress a sweep. The missing-fraction / ratio threshold floated by #477 is **rejected**, not deferred.'
signals: 'projection failure, silently dropped item, deferred ratio threshold rejected, STRM skip vs failure, library sweep anti-nuke follow-up, nested season/episode sweep guard · paths: `MediaServerProjectionResult`, `MediaServerProjectionFailureCounter`, `MediaServerReconciliationGuard`, `JellyfinApiClient.GetPagedLibraryItems`, `EmbyApiClient.GetPagedLibraryContents` · issues: #484, #477, #476'
mechanics: '`MediaServerReconciliationGuardTests` policy table · `JellyfinApiClientTests.ProjectionFailureCounter` · `MediaServerTelevisionLibraryScannerTests.Projection_Failure_Suppresses_The_Partial_Deletion_Sweep`'
---
This resolves both items the `scan.zero-item-fetch-guard` record (#477) left deferred. That record's rule
is unchanged and still in force: a zero-item fetch against a non-empty library still refuses the sweep.
This is a **second, independent** refusal on the same guard, plus a decision not to build the first.
- **The two ways a projection produces nothing are NOT the same thing, and conflating them is the main
way to get this wrong.** Each media-server api client maps every item the server returned through a
private `ProjectTo*` and drops the ones that yield nothing. A *deliberate skip* is a guard clause at
the top of the projection — Jellyfin `ProjectToMovie`/`ProjectToMusicVideo` (`LocationType !=
"FileSystem"`, `.strm`), `ProjectToEpisode` (same two), `ProjectToCollectionMediaItem` (both plus an
unmatched `item.Type`); Emby `ProjectToMovie` (no `MediaSources`), `ProjectToEpisode` (`LocationType
== "Virtual"`), `ProjectToCollectionMediaItem` (unmatched `item.Type`). These are permanent and
expected: a library holding one STRM file emits one on **every** scan, forever. A *failure drop* is
the `catch (Exception ex) { LogWarning(ex, "Error projecting …"); }` that every `ProjectTo*` in all
three clients ends with — the server DID return that item, we just could not build it, and at the
reconcile step that is indistinguishable from a deletion. Only failure drops suppress the sweep.
Counting deliberate skips would permanently disable reconciliation for any library containing a single
STRM file, so stale rows would accumulate forever — a regression, not a safe default.
`Jellyfin`/`Emby` `ProjectToShow`, `ProjectToSeason` and `ProjectToCollection` have **no** guard
clause at all, so for them every drop is a failure.
- **Rejected — the missing-fraction / ratio threshold.** It is a two-sided heuristic. Set low it
silently suppresses legitimate bulk deletions (stale rows persist invisibly, and the user's only
signal is a warning nobody reads); set high it misses the partial-fetch case it exists for. Choosing
the number needs per-install telemetry we do not collect, and no default is defensible for both a
20-item library and a 20,000-item one. Decisively: the failure it approximates is **exactly
observable** by the mechanism above, so accepting an unbounded false-suppression risk to approximate
it is a bad trade. A genuine bulk deletion produces **zero** projection failures (and a
correspondingly smaller server-reported total), so the deterministic signal has no false positives on
the very case the ratio threshold would have broken.
- **A three-state projection result, not a wider tuple.** The seam is deliberately narrow.
`IAsyncEnumerable<Tuple<TItem, int>>` appears in ~90 signatures across `ErsatzTV.Core/Interfaces/
{Jellyfin,Emby,Plex}`, the three api clients and ~15 scanners; widening it for this would be a
disproportionate, risky refactor. Instead the **private** mapper contract inside each client changed
from `Option<TItem>` to `MediaServerProjectionResult<TItem>` (projected / skipped / failed) — private,
so zero public churn — and the paged helper counts `IsFailure` in exactly one place per client. The
scanner reads the count through an **optional trailing parameter** on only the library-level methods
that actually feed a sweep (`IJellyfinApiClient.GetMovieLibraryItems` /
`GetMusicVideoLibraryItems` / `GetShowLibraryItemsWithoutPeople`, `IEmbyApiClient.GetMovieLibraryItems`
/ `GetShowLibraryItems`) **and** the nested season/episode ones (`GetSeasonLibraryItems`,
`GetEpisodeLibraryItems`, `GetEpisodeLibraryItemsWithoutPeople`), so every other call site is untouched.
A library that cannot be resolved (`Option<TLibrary>.None` inside a mapper) is classed as a **failure**
rather than a skip; that branch is defensive and currently **unreachable** (every counted path passes a
concrete library, and the only `None` caller — collections — does not route through it), kept only so
the default classification is right if a future caller does pass `None`.
- **The counter is per-enumeration state, never ambient.** `MediaServerProjectionFailureCounter` is
created by the scanner that owns the sweep, handed to the single enumeration whose result it will
diff, and read only after that enumeration completes. It is never a field on an api client (those are
long-lived and shared) and never static, so concurrent scans of different libraries cannot leak
failures into each other's sweep decision; increments are interlocked so a paginator that ever fans
out stays correct.
- **Wired into all six sweeps, not #477's four.** The four library-level ones — the three base scanners
(`MediaServerMovieLibraryScanner`, `MediaServerTelevisionLibraryScanner`,
`MediaServerOtherVideoLibraryScanner`) plus `JellyfinMusicVideoLibraryScanner`**and** the two nested
TV ones, `ScanSeasons`' `FlagFileNotFoundSeasons` and `ScanEpisodes`' `FlagFileNotFoundEpisodes`.
`ScanLibraryWithoutCleanup` (single-show rescan) passes no counter because it runs no sweep.
- **The nested season/episode sweeps get the failure refusal but NOT #477's empty-fetch branch.**
`scan.zero-item-fetch-guard` left them unguarded because "the blast radius is one show's seasons / one
season's episodes, not the whole library." That reasoning is sound for a per-parent **empty fetch** — a
plausible legitimate state, bounded to one parent — but it does **not** transfer to a projection
failure, which is systematic by construction: one bad code path fires on every parent, so a
`ProjectToEpisode` regression makes *every* season enumerate zero episodes and
`existing.Except([])` sweeps the **entire** episode library in a single scan, which `EmptyTrashHandler`
then deletes permanently. Same shape for `ProjectToSeason` plus #476's cascade. So the nested sweeps
call `MediaServerReconciliationGuard.ShouldFlagMissingDescendants` — the **same class and the same
private failure predicate** as `ShouldFlagMissing`, deliberately without the empty-fetch branch. A
separate entry point rather than reusing `ShouldFlagMissing` wholesale, because importing #477's
zero-count branch here would silently change per-parent behaviour and break #476's cascade, which
depends on an emptied parent still sweeping. One class still owns the invariant; only the empty-fetch
policy differs, and the difference is pinned by a test.
- **Known residual: a mass *Skip* is not covered, by design, and one skip site is response-shape
dependent.** The whole split rests on skips being permanent and item-intrinsic. Emby's
`ProjectToMovie` guard (`MediaSources is null || Count == 0`) does not fully satisfy that: it is
response-shape dependent, so a Refit/DTO drift or a `fields`-parameter regression would present as a
mass **Skip**, which by construction leaves the sweep enabled. A *total* such regression is caught by
#477's zero-incoming branch; a *partial* one (say 1,200 incoming vs 2,000 existing, zero failures)
would sweep 800 healthy movies into `FileNotFound`. Plex's pre-projection
`.Filter(m => m.Media.Count > 0 …)` is the same class. This is stated rather than engineered around:
re-classifying shape-dependent skips as failures would reintroduce exactly the STRM-style permanent
suppression the split exists to prevent. Treat it as the known next question if a mass-skip incident
ever occurs.
- **Log-contract note.** When both refusals apply, only the #484 "silently dropped …" warning is emitted,
not #477's "returned zero items" line — the failure names the actual cause, and emitting both would
imply two independent problems. Operator alerting that greps the #477 string will not fire in that
case. The message property is `{Scope}` (e.g. `library Movies`, `show Keeper seasons`,
`season 1 of show Keeper episodes`), not `{Library}`, so one alert pattern covers all six sweeps.
- **Plex reports zero failures because it has none.** `PlexServerApiClient`'s movie/show/other-video
projections return a bare entity with no `Option` and no `catch`, so a bad item throws and unwinds the
whole scan — the pre-existing "protection by accident of control flow" that #477 named. The Plex
scanner overrides therefore accept the counter and ignore it; that is honest, not a gap. If those
projections ever grow a swallowing `catch`, they must report into the counter at the same time.
- **Tests.** `MediaServerReconciliationGuardTests` pins the extended policy table: a failure count skips
and warns; a short incoming set with **zero** failures (the deliberate-skip case) still sweeps; a
1-of-500 bulk deletion with zero failures still sweeps; failures against an empty local set stay a
silent no-op; the failure branch is reported when both refusals apply. `JellyfinApiClientTests
.ProjectionFailureCounter` drives the real client over real JSON and proves the split at the layer
that makes it — a STRM plus a virtual item leave the counter at 0 while the healthy item still flows,
a projection made to throw yields no items and a count of 1, and two enumerations on one client
instance keep separate counts. `MediaServerTelevisionLibraryScannerTests
.Projection_Failure_Suppresses_The_Partial_Deletion_Sweep` is the integration pair to #476's cascade
test: identical arrangement, only the counter differs, so the cascade test is the positive control
that keeps the new test from passing vacuously. The nested levels get
`Season_Sweep_Respects_Projection_Failures` and `Episode_Sweep_Respects_Projection_Failures`, each
parameterised with its own zero-failure positive control, plus guard tests asserting the descendant
entry point still sweeps a per-parent **empty** fetch (the #477-scope invariant) and still sweeps when
only deliberate skips shortened the incoming set.
- **Tests pin the JOIN, not just the two halves.** A client test proving the counter fills, plus a
scanner test proving the guard honours a counter it is handed, would both stay green if a refactor
handed the api client a *fresh* counter while the sweep read the local one — silently killing the
protection. `MediaServerMovieLibraryScannerTests.Projection_Failure_From_The_Api_Suppresses_The_Sweep_
End_To_End`, `MediaServerTelevisionLibraryScannerTests.Public_ScanLibrary_Joins_The_Api_Counter_To_The_
Sweep`, the two nested TV cases, and `JellyfinMusicVideoLibraryScannerTests
.MusicVideo_Sweep_Respects_Projection_Failures` all drive the real `ScanLibrary` entry point and record
the failure from *inside* the enumeration, so same-instance wiring is what makes them pass.