# Retrieval-eval question bank (decision-lifecycle corpus, #521) This bank measures **correct active-record selection + citation**, not mere semantic proximity. The migrated decision corpus (`docs/decisions.md` + `docs/decisions/*.md`, catalog at `docs/decisions/README.md`, archive at `docs/decisions/archive/`) intentionally keeps a superseded or retired record's prose *verbatim* next to a live `superseded-by`/`supersedes` pointer. A retrieval approach that matches on wording alone can walk straight past that pointer into the archived record and return a decision this project no longer follows. For each question below, the only correct answer is the cited active `key` (status: `active`, in a file `active_files()` globs) — landing on an archived/superseded key, or missing a key that already answers the question (and thus proposing a reimplementation), is scored a **miss**, regardless of how relevant the returned text reads. This bank is run as the **Task 10 cold-agent retrieval sim**: a fresh agent is given the catalog (`docs/decisions/README.md`) plus the active `docs/decisions/*.md` wing and asked each question with no other context, then graded on whether it names the expected `key` and cites the right file. Verification: every `key` cited below was confirmed present and `status: active` via `PYTHONPATH=. python3 -c "import scripts.decisions_lib as dl; [print(r.key, r.status) for f in dl.active_files() for r in dl.parse_file(f) if r.key]"` against this worktree's corpus on 2026-07-21. --- ## 1. Paraphrased task → decision discovery **Q1.** "I'm adding a new REST endpoint for a feature that touches three tables — do I need to stand up a service/business-logic layer, or can the controller call MediatR directly?" - **Expected key:** `api.mediatr-passthrough` - **File:** `docs/decisions.md` - **Why:** The rule is literally "thin controllers over existing MediatR handlers, no new service/business-logic layer" — answer is NO, don't add one. A naive search might instead surface `mcp.server-foundation` (also MediatR-adjacent, wrong layer) or nothing at all if it only matches on "REST endpoint" rather than the layering question. **Q2.** "Should the playout-build-finished notification go out over a websocket/SignalR push channel so the SPA doesn't have to poll?" - **Expected key:** `api.async-op-contract` - **File:** `docs/decisions.md` - **Why:** The active rule explicitly rejects a live push channel in favor of an `isLocked` HTTP-observable flag on list/detail GETs as the substitute. A naive search keying on "poll" alone might return `spa.playback-troubleshoot-poll` (a different, narrower polling decision for the troubleshoot screen) instead of the general async-op contract that actually governs new queue-triggering endpoints. ## 2. Exact code/path lookup **Q3.** "Where do new API response DTOs live, and what nullable pragma convention do they follow?" - **Expected key:** `api.response-dtos` - **File:** `docs/decisions.md` - **Why:** Direct hit — rule gives the exact path pattern (`ErsatzTV.Core/Api//*ResponseModel.cs`) and the file-scoped `#nullable enable` convention. No plausible superseded alternative exists for this key. **Q4.** "Where does the `EntityLocker` implementation live, and how does it avoid a torn check-then-set race on its lock flags?" - **Expected key:** `locking.entitylocker-atomic-flags` - **File:** `docs/decisions.md` - **Why:** Rule + Mechanics cite `ErsatzTV.Infrastructure/Locking/EntityLocker.cs` and the `Interlocked.CompareExchange`-guarded atomic-flag fix directly; no other record touches this file. ## 3. Active-vs-superseded (selecting the archived record here is a FAILURE) **Q5** *(verbatim, coordinated with server-management#642).* "Is #390's small-lane CI move current?" - **Expected answer:** **No** — superseded by `ci.runner-placement`. - **Expected key:** `ci.runner-placement` - **File:** `docs/decisions.md` - **Why:** #390 moved `docker build` jobs onto the `small` runner lane to dodge queue time; #406 reversed that (worst-case memory, not runtime, was the real constraint) and the durable rule now lives under `ci.runner-placement` (and its sibling `ci.small-lane-git-only`, which defines `small` by job *kind*, not usual runtime). #390 itself was never given its own `##` record (it's prose-only, referenced inside `ci.runner-placement`'s own Signals line) — answering as if #390's move still holds, or citing #390 as a standalone current decision, is the failure mode this question targets. **Q6.** "Is `docs/decisions.md` still kept append-only by a line-deletion-diff CI/hook guard?" - **Expected answer:** **No** — that mechanism is superseded. - **Expected key:** `docs.decision-lifecycle` - **File:** `docs/decisions.md` - **Superseded record a naive search might return instead:** `docs.append-only-guard` (`docs/decisions/archive/release-ci-governance.md`, `status: superseded`, `superseded-by: docs.decision-lifecycle@2026-07-21`) - **Why:** `docs.append-only-guard`'s prose (numstat-deleted-count hook + CI job) still reads as a perfectly good, on-topic answer to "is decisions.md append-only" — that's exactly the trap: the *mechanism* it describes was replaced by the lifecycle-schema validator (`scripts/decisions_validate.py`) that this migration introduced. Selecting the archived record instead of following its `superseded-by` pointer to `docs.decision-lifecycle` is the scored failure. **Q6b.** "What is the current source of live queue state at session start?" - **Expected answer:** Two concurrent tracks — Orientation (docs/README.md task-signal map → the active decisions catalog) and, only when no issue is named, Selection (`scripts/select-queue.sh N`, a deterministic query over live Gitea state). Selecting the archived `docs.queue-state-gitea-tracker` ("queue state lives in pinned tracker issue #237") is a **FAILURE** — #237 closed 2026-07-13 and is now a single archival breadcrumb that MUST NOT be read for live state. - **Expected key:** `startup.parallel-orientation` - **File:** `docs/decisions.md` - **Superseded record a naive search might return instead:** `docs.queue-state-gitea-tracker` (`docs/decisions/archive/startup.md`, `status: superseded`, `superseded-by: startup.parallel-orientation@2026-07-21`) - **Why:** `docs.queue-state-gitea-tracker`'s prose ("Pinned tracker issue #237 holds the goal + ordered arc") still reads as a plausible, on-topic answer — that's the trap: the arc completed and #237 closed, so `scripts/select-queue.sh` (2026-07-19) replaced the *mechanical* selection logic with live Gitea queries, and #520 formalized the two-track protocol that retires #237 entirely. Following `docs.queue-state-gitea-tracker`'s own `superseded-by` pointer to `startup.parallel-orientation` (2026-07-21) rather than citing the archived record as current is the scored behavior. ## 4. Retired feature No `status: retired` record exists yet anywhere in the migrated corpus (confirmed via `grep -rn "status: retired" docs/decisions.md docs/decisions/*.md docs/decisions/archive/*.md` — zero hits). Using the one available superseded record as the closest analog instead: **Q7.** "Has the Husky `commit-msg` decisions-guard hook (the one that blocks any line-deletion diff to `decisions.md`) been removed now that decisions carry a lifecycle schema?" - **Expected answer:** Superseded, not simply "removed" — the append-only-by-diff mechanism is replaced by `scripts/decisions_validate.py`'s lifecycle-field checks; the sibling H3 root-screenshot guard from the same original record was split out and is **still active** today. - **Expected key:** `docs.decision-lifecycle` (supersedes `docs.append-only-guard`); sibling active key `ci.root-screenshot-guard` (`docs/decisions/release-ci-governance.md`) must NOT be reported as superseded — it's a distinct, still-live record split from the same legacy heading. - **Why:** Tests that an agent doesn't over-generalize "the old #303 H9/H3 heading was archived" into wrongly retiring the H3 half too — the migration-map explicitly documents the split. ## 5. Rationale / rejected-alternative **Q8.** "Why doesn't `EntityLocker` use owner tokens or lease objects instead of plain unlock calls?" - **Expected key:** `locking.entitylocker-atomic-flags` - **File:** `docs/decisions.md` - **Why:** The record's body explicitly documents the single-owner-release discipline as the chosen design and that `Unlock*` on an already-unlocked slot returns `false` + logs a Warning rather than throwing — the rejected alternative (owner tokens/leases) is named in the Rule line itself. **Q9.** "Why did the project reject a CI build-once shared-compile-artifact approach?" - **Expected key:** `ci.build-once-rejected` - **File:** `docs/decisions.md` - **Why:** Rule states it plainly: measured and rejected for a 40-85% wall-clock regression, keeping the #420 cross-run tree-identity skip instead. A search for "CI build speed" alone might surface `ci.docs-only-skip-steps` or `ci.peak-anon-measurement` (both real, both wrong for "why was build-once rejected"). ## 6. Convention already implemented — do NOT reimplement **Q10.** "We need seasonal/holiday scheduling (different programming around Christmas, say) — should I design a new date-conditional scheduling feature?" - **Expected key:** `sched.seasonal-scheduling-existing` - **File:** `docs/decisions.md` - **Why:** Already ships via `IAlternateScheduleItem` (Classic `ProgramScheduleAlternate`, Block `PlayoutTemplate`) evaluated by `AlternateScheduleSelector.GetScheduleForDate`; #73 was closed as already-implemented with a docs-only recipe added. Building a new feature here duplicates existing, shipped functionality — the correct answer is "use alternate schedules," not a design doc. **Q11.** "Should I add a way to pad content out to the next clock boundary (e.g. keep a channel's 7:00 PM start exact) per channel?" - **Expected key:** `sched.clock-padding-existing` - **File:** `docs/decisions.md` - **Why:** Already exists via `FillerPreset`'s `FillerMode.Pad` (Classic) and `pad_to_next`/`pad_until` (Sequential/YAML); #77 was closed as verified+documented, not built new. Only the one-click per-channel UI toggle is deferred (behind the #388 design-system epic) — the underlying capability is not missing. **Q12.** "Do we need to build a fair-share / weighted rotation mode for multi-collections, or does `ShuffleInOrder` already cover that?" - **Expected key:** `sched.weighted-shuffle` - **File:** `docs/decisions.md` - **Why:** This one cuts the other way on purpose — `ShuffleInOrder` only anti-clumps (its padding spacers emit nothing), so it does NOT already cover fair-share; the correct answer is that a new `PlaybackOrder.WeightedShuffle = 9` order was deliberately added rather than retrofitting `ShuffleInOrder`. Tests that "already exists, don't rebuild" isn't over-applied to a superficially-similar existing feature that actually doesn't do the job. --- ## Class coverage summary | # | Class | Key(s) | | - | ----- | ---- | | Q1 | paraphrased-discovery | `api.mediatr-passthrough` | | Q2 | paraphrased-discovery | `api.async-op-contract` | | Q3 | exact-lookup | `api.response-dtos` | | Q4 | exact-lookup | `locking.entitylocker-atomic-flags` | | Q5 | active-vs-superseded (verbatim #390) | `ci.runner-placement` | | Q6 | active-vs-superseded | `docs.decision-lifecycle` (vs archived `docs.append-only-guard`) | | Q6b | active-vs-superseded | `startup.parallel-orientation` (vs archived `docs.queue-state-gitea-tracker`) | | Q7 | retired-feature (no `retired` record exists; superseded used as analog) | `docs.decision-lifecycle` / `ci.root-screenshot-guard` (must stay active) | | Q8 | rationale/rejected-alternative | `locking.entitylocker-atomic-flags` | | Q9 | rationale/rejected-alternative | `ci.build-once-rejected` | | Q10 | convention-already-implemented | `sched.seasonal-scheduling-existing` | | Q11 | convention-already-implemented | `sched.clock-padding-existing` | | Q12 | convention-already-implemented (negative control) | `sched.weighted-shuffle` | 13 scored questions (Q1–Q12 plus Q6b) across all six required classes.