ersatztv/docs/decisions/retrieval-eval.md

# Retrieval-eval question bank (decision-lifecycle corpus, #521)

This bank measures **correct active-record selection + citation**, not mere semantic proximity.
The migrated decision corpus (`docs/decisions.md` + `docs/decisions/*.md`, catalog at
`docs/decisions/README.md`, archive at `docs/decisions/archive/`) intentionally keeps a superseded
or retired record's prose *verbatim* next to a live `superseded-by`/`supersedes` pointer. A retrieval
approach that matches on wording alone can walk straight past that pointer into the archived record
and return a decision this project no longer follows. For each question below, the only correct
answer is the cited active `key` (status: `active`, in a file `active_files()` globs) — landing on
an archived/superseded key, or missing a key that already answers the question (and thus proposing a
reimplementation), is scored a **miss**, regardless of how relevant the returned text reads.

This bank is run as the **Task 10 cold-agent retrieval sim**: a fresh agent is given the catalog
(`docs/decisions/README.md`) plus the active `docs/decisions/*.md` wing and asked each question with
no other context, then graded on whether it names the expected `key` and cites the right file.

Verification: every `key` cited below was confirmed present and `status: active` via
`PYTHONPATH=. python3 -c "import scripts.decisions_lib as dl; [print(r.key, r.status) for f in dl.active_files() for r in dl.parse_file(f) if r.key]"`
against this worktree's corpus on 2026-07-21.

---

## 1. Paraphrased task → decision discovery

**Q1.** "I'm adding a new REST endpoint for a feature that touches three tables — do I need to stand
up a service/business-logic layer, or can the controller call MediatR directly?"
- **Expected key:** `api.mediatr-passthrough`
- **File:** `docs/decisions.md`
- **Why:** The rule is literally "thin controllers over existing MediatR handlers, no new
  service/business-logic layer" — answer is NO, don't add one. A naive search might instead surface
  `mcp.server-foundation` (also MediatR-adjacent, wrong layer) or nothing at all if it only matches
  on "REST endpoint" rather than the layering question.

**Q2.** "Should the playout-build-finished notification go out over a websocket/SignalR push channel
so the SPA doesn't have to poll?"
- **Expected key:** `api.async-op-contract`
- **File:** `docs/decisions.md`
- **Why:** The active rule explicitly rejects a live push channel in favor of an `isLocked`
  HTTP-observable flag on list/detail GETs as the substitute. A naive search keying on "poll" alone
  might return `spa.playback-troubleshoot-poll` (a different, narrower polling decision for the
  troubleshoot screen) instead of the general async-op contract that actually governs new
  queue-triggering endpoints.

## 2. Exact code/path lookup

**Q3.** "Where do new API response DTOs live, and what nullable pragma convention do they follow?"
- **Expected key:** `api.response-dtos`
- **File:** `docs/decisions.md`
- **Why:** Direct hit — rule gives the exact path pattern
  (`ErsatzTV.Core/Api/<Domain>/*ResponseModel.cs`) and the file-scoped `#nullable enable` convention.
  No plausible superseded alternative exists for this key.

**Q4.** "Where does the `EntityLocker` implementation live, and how does it avoid a torn
check-then-set race on its lock flags?"
- **Expected key:** `locking.entitylocker-atomic-flags`
- **File:** `docs/decisions.md`
- **Why:** Rule + Mechanics cite `ErsatzTV.Infrastructure/Locking/EntityLocker.cs` and the
  `Interlocked.CompareExchange`-guarded atomic-flag fix directly; no other record touches this file.

## 3. Active-vs-superseded (selecting the archived record here is a FAILURE)

**Q5** *(verbatim, coordinated with server-management#642).* "Is #390's small-lane CI move current?"
- **Expected answer:** **No** — superseded by `ci.runner-placement`.
- **Expected key:** `ci.runner-placement`
- **File:** `docs/decisions.md`
- **Why:** #390 moved `docker build` jobs onto the `small` runner lane to dodge queue time; #406
  reversed that (worst-case memory, not runtime, was the real constraint) and the durable rule now
  lives under `ci.runner-placement` (and its sibling `ci.small-lane-git-only`, which defines
  `small` by job *kind*, not usual runtime). #390 itself was never given its own `##` record (it's
  prose-only, referenced inside `ci.runner-placement`'s own Signals line) — answering as if #390's
  move still holds, or citing #390 as a standalone current decision, is the failure mode this
  question targets.

**Q6.** "Is `docs/decisions.md` still kept append-only by a line-deletion-diff CI/hook guard?"
- **Expected answer:** **No** — that mechanism is superseded.
- **Expected key:** `docs.decision-lifecycle`
- **File:** `docs/decisions.md`
- **Superseded record a naive search might return instead:** `docs.append-only-guard`
  (`docs/decisions/archive/release-ci-governance.md`, `status: superseded`,
  `superseded-by: docs.decision-lifecycle@2026-07-21`)
- **Why:** `docs.append-only-guard`'s prose (numstat-deleted-count hook + CI job) still reads as a
  perfectly good, on-topic answer to "is decisions.md append-only" — that's exactly the trap: the
  *mechanism* it describes was replaced by the lifecycle-schema validator
  (`scripts/decisions_validate.py`) that this migration introduced. Selecting the archived record
  instead of following its `superseded-by` pointer to `docs.decision-lifecycle` is the scored
  failure.

**Q6b** *(deferred — flag, do not score yet).* "Does volatile session/queue state still live in the
pinned Gitea tracker issue #237 handoff pattern, or has that moved to a parallel-orientation
mechanism?"
- **Current state in this corpus snapshot:** `docs.queue-state-gitea-tracker` is still `status:
  active` (`docs/decisions.md`) — Task 8's `startup.parallel-orientation` record had not landed in
  this worktree as of this bank's authoring (2026-07-21). Once Task 8 merges and supersedes it, this
  becomes a third scored active-vs-superseded pair (expected key: `startup.parallel-orientation`;
  superseded key: `docs.queue-state-gitea-tracker`). Left in the bank as a placeholder per the Task 9
  brief; re-verify both keys' `status` before scoring it.

## 4. Retired feature

No `status: retired` record exists yet anywhere in the migrated corpus (confirmed via
`grep -rn "status: retired" docs/decisions.md docs/decisions/*.md docs/decisions/archive/*.md` —
zero hits). Using the one available superseded record as the closest analog instead:

**Q7.** "Has the Husky `commit-msg` decisions-guard hook (the one that blocks any line-deletion diff
to `decisions.md`) been removed now that decisions carry a lifecycle schema?"
- **Expected answer:** Superseded, not simply "removed" — the append-only-by-diff mechanism is
  replaced by `scripts/decisions_validate.py`'s lifecycle-field checks; the sibling H3
  root-screenshot guard from the same original record was split out and is **still active** today.
- **Expected key:** `docs.decision-lifecycle` (supersedes `docs.append-only-guard`); sibling active
  key `ci.root-screenshot-guard` (`docs/decisions/release-ci-governance.md`) must NOT be reported as
  superseded — it's a distinct, still-live record split from the same legacy heading.
- **Why:** Tests that an agent doesn't over-generalize "the old #303 H9/H3 heading was archived" into
  wrongly retiring the H3 half too — the migration-map explicitly documents the split.

## 5. Rationale / rejected-alternative

**Q8.** "Why doesn't `EntityLocker` use owner tokens or lease objects instead of plain unlock calls?"
- **Expected key:** `locking.entitylocker-atomic-flags`
- **File:** `docs/decisions.md`
- **Why:** The record's body explicitly documents the single-owner-release discipline as the chosen
  design and that `Unlock*` on an already-unlocked slot returns `false` + logs a Warning rather than
  throwing — the rejected alternative (owner tokens/leases) is named in the Rule line itself.

**Q9.** "Why did the project reject a CI build-once shared-compile-artifact approach?"
- **Expected key:** `ci.build-once-rejected`
- **File:** `docs/decisions.md`
- **Why:** Rule states it plainly: measured and rejected for a 40-85% wall-clock regression, keeping
  the #420 cross-run tree-identity skip instead. A search for "CI build speed" alone might surface
  `ci.docs-only-skip-steps` or `ci.peak-anon-measurement` (both real, both wrong for "why was
  build-once rejected").

## 6. Convention already implemented — do NOT reimplement

**Q10.** "We need seasonal/holiday scheduling (different programming around Christmas, say) — should
I design a new date-conditional scheduling feature?"
- **Expected key:** `sched.seasonal-scheduling-existing`
- **File:** `docs/decisions.md`
- **Why:** Already ships via `IAlternateScheduleItem` (Classic `ProgramScheduleAlternate`, Block
  `PlayoutTemplate`) evaluated by `AlternateScheduleSelector.GetScheduleForDate`; #73 was closed as
  already-implemented with a docs-only recipe added. Building a new feature here duplicates existing,
  shipped functionality — the correct answer is "use alternate schedules," not a design doc.

**Q11.** "Should I add a way to pad content out to the next clock boundary (e.g. keep a channel's
7:00 PM start exact) per channel?"
- **Expected key:** `sched.clock-padding-existing`
- **File:** `docs/decisions.md`
- **Why:** Already exists via `FillerPreset`'s `FillerMode.Pad` (Classic) and
  `pad_to_next`/`pad_until` (Sequential/YAML); #77 was closed as verified+documented, not built new.
  Only the one-click per-channel UI toggle is deferred (behind the #388 design-system epic) — the
  underlying capability is not missing.

**Q12.** "Do we need to build a fair-share / weighted rotation mode for multi-collections, or does
`ShuffleInOrder` already cover that?"
- **Expected key:** `sched.weighted-shuffle`
- **File:** `docs/decisions.md`
- **Why:** This one cuts the other way on purpose — `ShuffleInOrder` only anti-clumps (its padding
  spacers emit nothing), so it does NOT already cover fair-share; the correct answer is that a new
  `PlaybackOrder.WeightedShuffle = 9` order was deliberately added rather than retrofitting
  `ShuffleInOrder`. Tests that "already exists, don't rebuild" isn't over-applied to a
  superficially-similar existing feature that actually doesn't do the job.

---

## Class coverage summary

| # | Class | Key(s) |
| - | ----- | ---- |
| Q1 | paraphrased-discovery | `api.mediatr-passthrough` |
| Q2 | paraphrased-discovery | `api.async-op-contract` |
| Q3 | exact-lookup | `api.response-dtos` |
| Q4 | exact-lookup | `locking.entitylocker-atomic-flags` |
| Q5 | active-vs-superseded (verbatim #390) | `ci.runner-placement` |
| Q6 | active-vs-superseded | `docs.decision-lifecycle` (vs archived `docs.append-only-guard`) |
| Q6b | active-vs-superseded (deferred, unscored) | `startup.parallel-orientation` (pending Task 8) vs `docs.queue-state-gitea-tracker` |
| Q7 | retired-feature (no `retired` record exists; superseded used as analog) | `docs.decision-lifecycle` / `ci.root-screenshot-guard` (must stay active) |
| Q8 | rationale/rejected-alternative | `locking.entitylocker-atomic-flags` |
| Q9 | rationale/rejected-alternative | `ci.build-once-rejected` |
| Q10 | convention-already-implemented | `sched.seasonal-scheduling-existing` |
| Q11 | convention-already-implemented | `sched.clock-padding-existing` |
| Q12 | convention-already-implemented (negative control) | `sched.weighted-shuffle` |

12 scored questions (Q1–Q12, excluding the deferred Q6b placeholder) across all six required classes.