timothyandClaude Opus 4.8
fb6720ea27
- Exclude docs/decisions/retrieval-eval.md from active decision parsing (_NON_DECISION_FILES); its `## N.` eval-question headings were being miscounted as 7 legacy-unmigrated records. - Add decisions_lib.metadata_line_count() + a decisions_validate guard that fails a record with more than one `key:` metadata line, so a stacked-metadata-block migration bug (which the parser silently tolerated by reading only the first block) can't recur unnoticed. TDD: test_duplicate_metadata_block_fails / test_single_metadata_block_passes. - De-duplicate the 6 docs/decisions.md records left with two stacked metadata blocks (scan.getoraddfolder-db-lookup #488, scan.musicvideo-reconciliation #494, scan.jellyfin-mixed-content-library #489, iptv.logo-drives-bug-preset #67, ffmpeg.qsv-decode-encode-split #498, ci.small-lane-git-only server-management#639), merging the union of Signals/paths/issues/Mechanics from both blocks and keeping the richer Rule wording; rationale prose untouched. - Fill in the deferred Q6b row in docs/decisions/retrieval-eval.md now that startup.parallel-orientation is active in docs/decisions.md, scoring it as a real active-vs-superseded question against the archived docs.queue-state-gitea-tracker. - Regenerate docs/decisions/README.md via build_decisions_catalog.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
193 lines
12 KiB
Markdown
193 lines
12 KiB
Markdown
# Retrieval-eval question bank (decision-lifecycle corpus, #521)
|
||
|
||
This bank measures **correct active-record selection + citation**, not mere semantic proximity.
|
||
The migrated decision corpus (`docs/decisions.md` + `docs/decisions/*.md`, catalog at
|
||
`docs/decisions/README.md`, archive at `docs/decisions/archive/`) intentionally keeps a superseded
|
||
or retired record's prose *verbatim* next to a live `superseded-by`/`supersedes` pointer. A retrieval
|
||
approach that matches on wording alone can walk straight past that pointer into the archived record
|
||
and return a decision this project no longer follows. For each question below, the only correct
|
||
answer is the cited active `key` (status: `active`, in a file `active_files()` globs) — landing on
|
||
an archived/superseded key, or missing a key that already answers the question (and thus proposing a
|
||
reimplementation), is scored a **miss**, regardless of how relevant the returned text reads.
|
||
|
||
This bank is run as the **Task 10 cold-agent retrieval sim**: a fresh agent is given the catalog
|
||
(`docs/decisions/README.md`) plus the active `docs/decisions/*.md` wing and asked each question with
|
||
no other context, then graded on whether it names the expected `key` and cites the right file.
|
||
|
||
Verification: every `key` cited below was confirmed present and `status: active` via
|
||
`PYTHONPATH=. python3 -c "import scripts.decisions_lib as dl; [print(r.key, r.status) for f in dl.active_files() for r in dl.parse_file(f) if r.key]"`
|
||
against this worktree's corpus on 2026-07-21.
|
||
|
||
---
|
||
|
||
## 1. Paraphrased task → decision discovery
|
||
|
||
**Q1.** "I'm adding a new REST endpoint for a feature that touches three tables — do I need to stand
|
||
up a service/business-logic layer, or can the controller call MediatR directly?"
|
||
- **Expected key:** `api.mediatr-passthrough`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** The rule is literally "thin controllers over existing MediatR handlers, no new
|
||
service/business-logic layer" — answer is NO, don't add one. A naive search might instead surface
|
||
`mcp.server-foundation` (also MediatR-adjacent, wrong layer) or nothing at all if it only matches
|
||
on "REST endpoint" rather than the layering question.
|
||
|
||
**Q2.** "Should the playout-build-finished notification go out over a websocket/SignalR push channel
|
||
so the SPA doesn't have to poll?"
|
||
- **Expected key:** `api.async-op-contract`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** The active rule explicitly rejects a live push channel in favor of an `isLocked`
|
||
HTTP-observable flag on list/detail GETs as the substitute. A naive search keying on "poll" alone
|
||
might return `spa.playback-troubleshoot-poll` (a different, narrower polling decision for the
|
||
troubleshoot screen) instead of the general async-op contract that actually governs new
|
||
queue-triggering endpoints.
|
||
|
||
## 2. Exact code/path lookup
|
||
|
||
**Q3.** "Where do new API response DTOs live, and what nullable pragma convention do they follow?"
|
||
- **Expected key:** `api.response-dtos`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** Direct hit — rule gives the exact path pattern
|
||
(`ErsatzTV.Core/Api/<Domain>/*ResponseModel.cs`) and the file-scoped `#nullable enable` convention.
|
||
No plausible superseded alternative exists for this key.
|
||
|
||
**Q4.** "Where does the `EntityLocker` implementation live, and how does it avoid a torn
|
||
check-then-set race on its lock flags?"
|
||
- **Expected key:** `locking.entitylocker-atomic-flags`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** Rule + Mechanics cite `ErsatzTV.Infrastructure/Locking/EntityLocker.cs` and the
|
||
`Interlocked.CompareExchange`-guarded atomic-flag fix directly; no other record touches this file.
|
||
|
||
## 3. Active-vs-superseded (selecting the archived record here is a FAILURE)
|
||
|
||
**Q5** *(verbatim, coordinated with server-management#642).* "Is #390's small-lane CI move current?"
|
||
- **Expected answer:** **No** — superseded by `ci.runner-placement`.
|
||
- **Expected key:** `ci.runner-placement`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** #390 moved `docker build` jobs onto the `small` runner lane to dodge queue time; #406
|
||
reversed that (worst-case memory, not runtime, was the real constraint) and the durable rule now
|
||
lives under `ci.runner-placement` (and its sibling `ci.small-lane-git-only`, which defines
|
||
`small` by job *kind*, not usual runtime). #390 itself was never given its own `##` record (it's
|
||
prose-only, referenced inside `ci.runner-placement`'s own Signals line) — answering as if #390's
|
||
move still holds, or citing #390 as a standalone current decision, is the failure mode this
|
||
question targets.
|
||
|
||
**Q6.** "Is `docs/decisions.md` still kept append-only by a line-deletion-diff CI/hook guard?"
|
||
- **Expected answer:** **No** — that mechanism is superseded.
|
||
- **Expected key:** `docs.decision-lifecycle`
|
||
- **File:** `docs/decisions.md`
|
||
- **Superseded record a naive search might return instead:** `docs.append-only-guard`
|
||
(`docs/decisions/archive/release-ci-governance.md`, `status: superseded`,
|
||
`superseded-by: docs.decision-lifecycle@2026-07-21`)
|
||
- **Why:** `docs.append-only-guard`'s prose (numstat-deleted-count hook + CI job) still reads as a
|
||
perfectly good, on-topic answer to "is decisions.md append-only" — that's exactly the trap: the
|
||
*mechanism* it describes was replaced by the lifecycle-schema validator
|
||
(`scripts/decisions_validate.py`) that this migration introduced. Selecting the archived record
|
||
instead of following its `superseded-by` pointer to `docs.decision-lifecycle` is the scored
|
||
failure.
|
||
|
||
**Q6b.** "What is the current source of live queue state at session start?"
|
||
- **Expected answer:** Two concurrent tracks — Orientation (docs/README.md task-signal map → the
|
||
active decisions catalog) and, only when no issue is named, Selection
|
||
(`scripts/select-queue.sh N`, a deterministic query over live Gitea state). Selecting the archived
|
||
`docs.queue-state-gitea-tracker` ("queue state lives in pinned tracker issue #237") is a **FAILURE**
|
||
— #237 closed 2026-07-13 and is now a single archival breadcrumb that MUST NOT be read for live
|
||
state.
|
||
- **Expected key:** `startup.parallel-orientation`
|
||
- **File:** `docs/decisions.md`
|
||
- **Superseded record a naive search might return instead:** `docs.queue-state-gitea-tracker`
|
||
(`docs/decisions/archive/startup.md`, `status: superseded`,
|
||
`superseded-by: startup.parallel-orientation@2026-07-21`)
|
||
- **Why:** `docs.queue-state-gitea-tracker`'s prose ("Pinned tracker issue #237 holds the goal +
|
||
ordered arc") still reads as a plausible, on-topic answer — that's the trap: the arc completed and
|
||
#237 closed, so `scripts/select-queue.sh` (2026-07-19) replaced the *mechanical* selection logic
|
||
with live Gitea queries, and #520 formalized the two-track protocol that retires #237 entirely.
|
||
Following `docs.queue-state-gitea-tracker`'s own `superseded-by` pointer to
|
||
`startup.parallel-orientation` (2026-07-21) rather than citing the archived record as current is
|
||
the scored behavior.
|
||
|
||
## 4. Retired feature
|
||
|
||
No `status: retired` record exists yet anywhere in the migrated corpus (confirmed via
|
||
`grep -rn "status: retired" docs/decisions.md docs/decisions/*.md docs/decisions/archive/*.md` —
|
||
zero hits). Using the one available superseded record as the closest analog instead:
|
||
|
||
**Q7.** "Has the Husky `commit-msg` decisions-guard hook (the one that blocks any line-deletion diff
|
||
to `decisions.md`) been removed now that decisions carry a lifecycle schema?"
|
||
- **Expected answer:** Superseded, not simply "removed" — the append-only-by-diff mechanism is
|
||
replaced by `scripts/decisions_validate.py`'s lifecycle-field checks; the sibling H3
|
||
root-screenshot guard from the same original record was split out and is **still active** today.
|
||
- **Expected key:** `docs.decision-lifecycle` (supersedes `docs.append-only-guard`); sibling active
|
||
key `ci.root-screenshot-guard` (`docs/decisions/release-ci-governance.md`) must NOT be reported as
|
||
superseded — it's a distinct, still-live record split from the same legacy heading.
|
||
- **Why:** Tests that an agent doesn't over-generalize "the old #303 H9/H3 heading was archived" into
|
||
wrongly retiring the H3 half too — the migration-map explicitly documents the split.
|
||
|
||
## 5. Rationale / rejected-alternative
|
||
|
||
**Q8.** "Why doesn't `EntityLocker` use owner tokens or lease objects instead of plain unlock calls?"
|
||
- **Expected key:** `locking.entitylocker-atomic-flags`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** The record's body explicitly documents the single-owner-release discipline as the chosen
|
||
design and that `Unlock*` on an already-unlocked slot returns `false` + logs a Warning rather than
|
||
throwing — the rejected alternative (owner tokens/leases) is named in the Rule line itself.
|
||
|
||
**Q9.** "Why did the project reject a CI build-once shared-compile-artifact approach?"
|
||
- **Expected key:** `ci.build-once-rejected`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** Rule states it plainly: measured and rejected for a 40-85% wall-clock regression, keeping
|
||
the #420 cross-run tree-identity skip instead. A search for "CI build speed" alone might surface
|
||
`ci.docs-only-skip-steps` or `ci.peak-anon-measurement` (both real, both wrong for "why was
|
||
build-once rejected").
|
||
|
||
## 6. Convention already implemented — do NOT reimplement
|
||
|
||
**Q10.** "We need seasonal/holiday scheduling (different programming around Christmas, say) — should
|
||
I design a new date-conditional scheduling feature?"
|
||
- **Expected key:** `sched.seasonal-scheduling-existing`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** Already ships via `IAlternateScheduleItem` (Classic `ProgramScheduleAlternate`, Block
|
||
`PlayoutTemplate`) evaluated by `AlternateScheduleSelector.GetScheduleForDate`; #73 was closed as
|
||
already-implemented with a docs-only recipe added. Building a new feature here duplicates existing,
|
||
shipped functionality — the correct answer is "use alternate schedules," not a design doc.
|
||
|
||
**Q11.** "Should I add a way to pad content out to the next clock boundary (e.g. keep a channel's
|
||
7:00 PM start exact) per channel?"
|
||
- **Expected key:** `sched.clock-padding-existing`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** Already exists via `FillerPreset`'s `FillerMode.Pad` (Classic) and
|
||
`pad_to_next`/`pad_until` (Sequential/YAML); #77 was closed as verified+documented, not built new.
|
||
Only the one-click per-channel UI toggle is deferred (behind the #388 design-system epic) — the
|
||
underlying capability is not missing.
|
||
|
||
**Q12.** "Do we need to build a fair-share / weighted rotation mode for multi-collections, or does
|
||
`ShuffleInOrder` already cover that?"
|
||
- **Expected key:** `sched.weighted-shuffle`
|
||
- **File:** `docs/decisions.md`
|
||
- **Why:** This one cuts the other way on purpose — `ShuffleInOrder` only anti-clumps (its padding
|
||
spacers emit nothing), so it does NOT already cover fair-share; the correct answer is that a new
|
||
`PlaybackOrder.WeightedShuffle = 9` order was deliberately added rather than retrofitting
|
||
`ShuffleInOrder`. Tests that "already exists, don't rebuild" isn't over-applied to a
|
||
superficially-similar existing feature that actually doesn't do the job.
|
||
|
||
---
|
||
|
||
## Class coverage summary
|
||
|
||
| # | Class | Key(s) |
|
||
| - | ----- | ---- |
|
||
| Q1 | paraphrased-discovery | `api.mediatr-passthrough` |
|
||
| Q2 | paraphrased-discovery | `api.async-op-contract` |
|
||
| Q3 | exact-lookup | `api.response-dtos` |
|
||
| Q4 | exact-lookup | `locking.entitylocker-atomic-flags` |
|
||
| Q5 | active-vs-superseded (verbatim #390) | `ci.runner-placement` |
|
||
| Q6 | active-vs-superseded | `docs.decision-lifecycle` (vs archived `docs.append-only-guard`) |
|
||
| Q6b | active-vs-superseded | `startup.parallel-orientation` (vs archived `docs.queue-state-gitea-tracker`) |
|
||
| Q7 | retired-feature (no `retired` record exists; superseded used as analog) | `docs.decision-lifecycle` / `ci.root-screenshot-guard` (must stay active) |
|
||
| Q8 | rationale/rejected-alternative | `locking.entitylocker-atomic-flags` |
|
||
| Q9 | rationale/rejected-alternative | `ci.build-once-rejected` |
|
||
| Q10 | convention-already-implemented | `sched.seasonal-scheduling-existing` |
|
||
| Q11 | convention-already-implemented | `sched.clock-padding-existing` |
|
||
| Q12 | convention-already-implemented (negative control) | `sched.weighted-shuffle` |
|
||
|
||
13 scored questions (Q1–Q12 plus Q6b) across all six required classes.
|