ersatztv/docs/decisions/retrieval-eval.md

Retrieval-eval question bank (decision-lifecycle corpus, #521)

This bank measures correct active-record selection + citation, not mere semantic proximity. The migrated decision corpus (docs/decisions.md + docs/decisions/*.md, catalog at docs/decisions/README.md, archive at docs/decisions/archive/) intentionally keeps a superseded or retired record's prose verbatim next to a live superseded-by/supersedes pointer. A retrieval approach that matches on wording alone can walk straight past that pointer into the archived record and return a decision this project no longer follows. For each question below, the only correct answer is the cited active key (status: active, in a file active_files() globs) — landing on an archived/superseded key, or missing a key that already answers the question (and thus proposing a reimplementation), is scored a miss, regardless of how relevant the returned text reads.

This bank is run as the Task 10 cold-agent retrieval sim: a fresh agent is given the catalog (docs/decisions/README.md) plus the active docs/decisions/*.md wing and asked each question with no other context, then graded on whether it names the expected key and cites the right file.

Verification: every key cited below was confirmed present and status: active via PYTHONPATH=. python3 -c "import scripts.decisions_lib as dl; [print(r.key, r.status) for f in dl.active_files() for r in dl.parse_file(f) if r.key]" against this worktree's corpus on 2026-07-21.

---

1. Paraphrased task → decision discovery

Q1. "I'm adding a new REST endpoint for a feature that touches three tables — do I need to stand up a service/business-logic layer, or can the controller call MediatR directly?"

• Expected key: api.mediatr-passthrough
• File: docs/decisions.md
• Why: The rule is literally "thin controllers over existing MediatR handlers, no new service/business-logic layer" — answer is NO, don't add one. A naive search might instead surface mcp.server-foundation (also MediatR-adjacent, wrong layer) or nothing at all if it only matches on "REST endpoint" rather than the layering question.

Q2. "Should the playout-build-finished notification go out over a websocket/SignalR push channel so the SPA doesn't have to poll?"

• Expected key: api.async-op-contract
• File: docs/decisions.md
• Why: The active rule explicitly rejects a live push channel in favor of an isLocked HTTP-observable flag on list/detail GETs as the substitute. A naive search keying on "poll" alone might return spa.playback-troubleshoot-poll (a different, narrower polling decision for the troubleshoot screen) instead of the general async-op contract that actually governs new queue-triggering endpoints.

2. Exact code/path lookup

Q3. "Where do new API response DTOs live, and what nullable pragma convention do they follow?"

• Expected key: api.response-dtos
• File: docs/decisions.md
• Why: Direct hit — rule gives the exact path pattern (ErsatzTV.Core/Api/<Domain>/*ResponseModel.cs) and the file-scoped #nullable enable convention. No plausible superseded alternative exists for this key.

Q4. "Where does the EntityLocker implementation live, and how does it avoid a torn check-then-set race on its lock flags?"

• Expected key: locking.entitylocker-atomic-flags
• File: docs/decisions.md
• Why: Rule + Mechanics cite ErsatzTV.Infrastructure/Locking/EntityLocker.cs and the Interlocked.CompareExchange-guarded atomic-flag fix directly; no other record touches this file.

3. Active-vs-superseded (selecting the archived record here is a FAILURE)

Q5 (verbatim, coordinated with server-management#642). "Is #390's small-lane CI move current?"

• Expected answer: No — superseded by ci.runner-placement.
• Expected key: ci.runner-placement
• File: docs/decisions.md
• Why: #390 moved docker build jobs onto the small runner lane to dodge queue time; #406 reversed that (worst-case memory, not runtime, was the real constraint) and the durable rule now lives under ci.runner-placement (and its sibling ci.small-lane-git-only, which defines small by job kind, not usual runtime). #390 itself was never given its own ## record (it's prose-only, referenced inside ci.runner-placement's own Signals line) — answering as if #390's move still holds, or citing #390 as a standalone current decision, is the failure mode this question targets.

Q6. "Is docs/decisions.md still kept append-only by a line-deletion-diff CI/hook guard?"

• Expected answer: No — that mechanism is superseded.
• Expected key: docs.decision-lifecycle
• File: docs/decisions.md
• Superseded record a naive search might return instead: docs.append-only-guard (docs/decisions/archive/release-ci-governance.md, status: superseded, superseded-by: docs.decision-lifecycle@2026-07-21)
• Why: docs.append-only-guard's prose (numstat-deleted-count hook + CI job) still reads as a perfectly good, on-topic answer to "is decisions.md append-only" — that's exactly the trap: the mechanism it describes was replaced by the lifecycle-schema validator (scripts/decisions_validate.py) that this migration introduced. Selecting the archived record instead of following its superseded-by pointer to docs.decision-lifecycle is the scored failure.

Q6b. "What is the current source of live queue state at session start?"

• Expected answer: Two concurrent tracks — Orientation (docs/README.md task-signal map → the active decisions catalog) and, only when no issue is named, Selection (scripts/select-queue.sh N, a deterministic query over live Gitea state). Selecting the archived docs.queue-state-gitea-tracker ("queue state lives in pinned tracker issue #237") is a FAILURE — #237 closed 2026-07-13 and is now a single archival breadcrumb that MUST NOT be read for live state.
• Expected key: startup.parallel-orientation
• File: docs/decisions.md
• Superseded record a naive search might return instead: docs.queue-state-gitea-tracker (docs/decisions/archive/startup.md, status: superseded, superseded-by: startup.parallel-orientation@2026-07-21)
• Why: docs.queue-state-gitea-tracker's prose ("Pinned tracker issue #237 holds the goal + ordered arc") still reads as a plausible, on-topic answer — that's the trap: the arc completed and #237 closed, so scripts/select-queue.sh (2026-07-19) replaced the mechanical selection logic with live Gitea queries, and #520 formalized the two-track protocol that retires #237 entirely. Following docs.queue-state-gitea-tracker's own superseded-by pointer to startup.parallel-orientation (2026-07-21) rather than citing the archived record as current is the scored behavior.

4. Retired feature

No status: retired record exists yet anywhere in the migrated corpus (confirmed via grep -rn "status: retired" docs/decisions.md docs/decisions/*.md docs/decisions/archive/*.md — zero hits). Using the one available superseded record as the closest analog instead:

Q7. "Has the Husky commit-msg decisions-guard hook (the one that blocks any line-deletion diff to decisions.md) been removed now that decisions carry a lifecycle schema?"

• Expected answer: Superseded, not simply "removed" — the append-only-by-diff mechanism is replaced by scripts/decisions_validate.py's lifecycle-field checks; the sibling H3 root-screenshot guard from the same original record was split out and is still active today.
• Expected key: docs.decision-lifecycle (supersedes docs.append-only-guard); sibling active key ci.root-screenshot-guard (docs/decisions/release-ci-governance.md) must NOT be reported as superseded — it's a distinct, still-live record split from the same legacy heading.
• Why: Tests that an agent doesn't over-generalize "the old #303 H9/H3 heading was archived" into wrongly retiring the H3 half too — the migration-map explicitly documents the split.

5. Rationale / rejected-alternative

Q8. "Why doesn't EntityLocker use owner tokens or lease objects instead of plain unlock calls?"

• Expected key: locking.entitylocker-atomic-flags
• File: docs/decisions.md
• Why: The record's body explicitly documents the single-owner-release discipline as the chosen design and that Unlock* on an already-unlocked slot returns false + logs a Warning rather than throwing — the rejected alternative (owner tokens/leases) is named in the Rule line itself.

Q9. "Why did the project reject a CI build-once shared-compile-artifact approach?"

• Expected key: ci.build-once-rejected
• File: docs/decisions.md
• Why: Rule states it plainly: measured and rejected for a 40-85% wall-clock regression, keeping the #420 cross-run tree-identity skip instead. A search for "CI build speed" alone might surface ci.docs-only-skip-steps or ci.peak-anon-measurement (both real, both wrong for "why was build-once rejected").

6. Convention already implemented — do NOT reimplement

Q10. "We need seasonal/holiday scheduling (different programming around Christmas, say) — should I design a new date-conditional scheduling feature?"

• Expected key: sched.seasonal-scheduling-existing
• File: docs/decisions.md
• Why: Already ships via IAlternateScheduleItem (Classic ProgramScheduleAlternate, Block PlayoutTemplate) evaluated by AlternateScheduleSelector.GetScheduleForDate; #73 was closed as already-implemented with a docs-only recipe added. Building a new feature here duplicates existing, shipped functionality — the correct answer is "use alternate schedules," not a design doc.

Q11. "Should I add a way to pad content out to the next clock boundary (e.g. keep a channel's 7:00 PM start exact) per channel?"

• Expected key: sched.clock-padding-existing
• File: docs/decisions.md
• Why: Already exists via FillerPreset's FillerMode.Pad (Classic) and pad_to_next/pad_until (Sequential/YAML); #77 was closed as verified+documented, not built new. Only the one-click per-channel UI toggle is deferred (behind the #388 design-system epic) — the underlying capability is not missing.

Q12. "Do we need to build a fair-share / weighted rotation mode for multi-collections, or does ShuffleInOrder already cover that?"

• Expected key: sched.weighted-shuffle
• File: docs/decisions.md
• Why: This one cuts the other way on purpose — ShuffleInOrder only anti-clumps (its padding spacers emit nothing), so it does NOT already cover fair-share; the correct answer is that a new PlaybackOrder.WeightedShuffle = 9 order was deliberately added rather than retrofitting ShuffleInOrder. Tests that "already exists, don't rebuild" isn't over-applied to a superficially-similar existing feature that actually doesn't do the job.

---

Class coverage summary

#	Class	Key(s)
Q1	paraphrased-discovery	api.mediatr-passthrough
Q2	paraphrased-discovery	api.async-op-contract
Q3	exact-lookup	api.response-dtos
Q4	exact-lookup	locking.entitylocker-atomic-flags
Q5	active-vs-superseded (verbatim #390)	ci.runner-placement
Q6	active-vs-superseded	docs.decision-lifecycle (vs archived docs.append-only-guard)
Q6b	active-vs-superseded	startup.parallel-orientation (vs archived docs.queue-state-gitea-tracker)
Q7	retired-feature (no retired record exists; superseded used as analog)	docs.decision-lifecycle / ci.root-screenshot-guard (must stay active)
Q8	rationale/rejected-alternative	locking.entitylocker-atomic-flags
Q9	rationale/rejected-alternative	ci.build-once-rejected
Q10	convention-already-implemented	sched.seasonal-scheduling-existing
Q11	convention-already-implemented	sched.clock-padding-existing
Q12	convention-already-implemented (negative control)	sched.weighted-shuffle

13 scored questions (Q1–Q12 plus Q6b) across all six required classes.