docs(773): root-cause the recurring defect shapes across the full closed-issue corpus #782

Merged
timothy merged 8 commits from issue/773-defect-shapes into main 2026-08-13 21:16:51 +02:00
2 changed files with 537 additions and 0 deletions
+7
View File
@@ -24,6 +24,7 @@ doc below, or that changes which sections a task signal points to.**
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible) |
| Live local run / Playwright-MCP verification | `docs/e2e-local.md` + `scripts/e2e-local.sh` |
| Adding/changing a UI-E2E browser flow | `docs/e2e-local.md` → "UI-E2E harness" + `scripts/e2e-ui.sh` |
| What does a test suite cover | `docs/testing.md` |
@@ -98,6 +99,12 @@ bounds, what's mined per issue): `docs/handoffs/chicorytv-issue-queue.md` → "K
that file's standing kickoff for the two concurrent tracks (orientation ‖ selection). ersatztv#237
is a closed, archival historical tracker (superseded by `startup.parallel-orientation` in
`docs/decisions.md`) — not a live pointer.
- **`docs/defect-shapes-773.md`** — root-cause analysis of the recurring defect shapes across the
whole closed-issue corpus (#773): the measured class ranking, the four families they consolidate
into, the cheapest mechanical detector per class, the classes where **no** detector is plausible,
and an audit of which configured hooks/MCP servers/LSPs are actually invoked. Read it before
proposing a new guard or CI check — §4 is the detector menu, and it argues against enumerating
cases one incident at a time.
- **`docs/tracker-retrofit-triage-237.md`** — audit trail for the #524 triage of ersatztv#237's 111
comments (method, per-comment classification, totals). Evidence for the
`docs.tracker-comment-retrofit` decision; read it only when triaging another over-cap tracker.
+530
View File
@@ -0,0 +1,530 @@
# Recurring defect shapes across the closed-issue corpus (ersatztv#773)
Root-cause analysis over every closed issue in the tracker, the detectors that would have caught
each class, and an audit of the tooling already configured in this repo. This is an **analysis
artifact**, not a rulebook: the rules it argues for land as decision records and follow-up issues,
which are linked per class below.
Measured 2026-08-13 against `origin/main` at `f9f8f65ce`. **The issue corpus is a snapshot drawn at
19:45 that day** — a boundary that matters, because #767 closed nineteen minutes later and is
therefore absent from every count here (§3.7).
---
## 1. Method, and what these numbers can and cannot support
**Corpus.** All 349 closed issues (`state=closed&type=issues`, 7 pages, 349 distinct numbers,
#1–#757 — pull requests excluded). Of those, **95 carry a `## Closing record`** comment. Each
record was classified by one of five independent agents against a written taxonomy, with
`NEW:<name>` available so the taxonomy could not be self-fulfilling. A sixth agent re-rated a
stratified 15-issue sample blind, as an inter-rater control. A seventh sampled 25 of the 254
pre-convention closed issues from their raw bodies and comments, to test whether the shapes exist
outside the era that writes records.
**Three limits, stated up front because they bound every number below.**
1. **Record coverage is not uniform.** The `## Closing record` convention starts at #520
(2026-07-21). Coverage is 31/32 for #600–699 and 4/69 for #1–99. So frequencies over the 95
records are frequencies over *recent* practice.
2. **Corpus composition shifts underneath the measurement.** The share of records describing no
process failure at all falls from 12/19 (#60–#438) to 0/19 (#616–#671). Early issues are
feature work; recent issues are largely CI/process hardening. Guard-shaped defects therefore
dominate the recent records *partly by construction* — we have lately been building guards, so
our defects are in guards.
3. **A closing record is the closing author's self-report.** Raters were told to flag records whose
own claims outrun what they describe, and several did (#60, #651, #691, #720). But the corpus
systematically cannot contain a defect nobody noticed.
**What limit 2 does *not* undermine.** The obvious worry — that these classes are an artifact of
recent guard-building — was tested and **refuted**. The backward sample found high-confidence
instances of the same shapes well before the convention existed: twin-missed at #215 (REST playout
mutations skip the build-lock gating the Blazor path had) and #403 (an unhandled enum silent at 5 of
6 dispatch sites); wrapper drift at #198 and #287 (OpenAPI spec diverged from runtime JSON);
masking guards at #232, whose SPA `PENDING_GRACE_TICKS` "existed only to paper over the resulting
dishonest 200", and #234, where a blind `finally`-unlock was "papering over the stranded lock, so
the two had to land together"; overclaim at #1 ("the earlier 'confirmed' was based on one curl test
and several unverified assertions"). The convention changed the **density and legibility** of the
evidence, not the existence of the shapes.
Classification of pre-convention issues is harder but mostly possible — of the five the backward
rater flagged as unclassifiable, four (#94, #172, #265, #377) do carry a closing comment with a
clear root cause on re-inspection, and only **#157** (an unresolved flaky-test report with a single
comment) is genuinely evidence-starved. The `insufficient-evidence` verdict is therefore rarer than
first reported, which if anything strengthens the backward result.
---
## 2. What the corpus actually says
95 records; **69 describe a process failure**, 26 describe none. Percentages are of the 69.
| Rank | Class | n | % | Shape |
|---|---|---|---|---|
| 1 | **Vacuous verification** | 17 | 25% | A check was green having done no work |
| 2 | **Twin-missed** | 14 | 20% | The fix hit one instance and missed its structural twin |
| 3 | **Symptom-keyed guard** | 9 | 13% | The guard keyed on the symptom seen, not the defect's mechanism |
| 4 | *(new classes — see §3)* | 8 | 12% | Proposed by raters as fitting nothing above |
| 5 | **Environment divergence** | 7 | 10% | Behaviour differs by interpreter/provider/hardware, untested there |
| 6 | **Wrapper drift** | 5 | 7% | A hand-maintained mirror drifted from the contract it mirrors |
| 7 | **Overclaim / stale claim** | 3 | 4% | An assertion stronger than its evidence, or since gone false |
| 7 | **String-predicate churn** | 3 | 4% | A regex/grep predicate needing repeated rounds — but see §3.7: the *rounds* are cross-cutting, only the *substrate* is string-shaped |
| 9 | **Identity-not-capability** | 2 | 3% | Authorization scoped by *who*, not by *what is being changed* |
| 10 | **Masking guards** | 1 | 1% | Two guards on one condition; one hid the other's total failure |
**This reorders the issue's own ranking.** #773 put twin-missed first (9 instances) and vacuous
verification second (8), from a 50-issue sample. Over the full record corpus they swap: **vacuous
verification is the most common single shape.** Overclaim also drops sharply as a *primary* cause
(4%) — raters overwhelmingly assigned it as a *secondary*. It is better understood as a modifier
that rides on another failure than as a class with its own detector.
**Inter-rater control (measured, not asserted).** A sixth rater re-classified a stratified 15-issue
sample blind. **Raw agreement 12/15 (80%)** across an 11-category scheme. The three disagreements
are informative rather than noise:
- **#177** (twin-missed → no-failure) and **#521** (no-failure → twin-missed) disagree at the
"did a process failure occur" boundary and point in **opposite directions**, so that boundary is
noisy but not systematically biased. The 26 clean-record count should be read as ±3.
- **#649** was rated twin-missed by one and wrapper-drift by the other — which is precisely the blur
that Family C in §3 exists to resolve. Both raters named the other class as secondary.
The top-three separation is far larger than the disagreement. Treat ±1 rank as noise.
---
## 3. Consolidation: four families, one of which the issue's taxonomy missed
### Family A — reasoning about a representative instead of the population
**#773's central hypothesis, confirmed, and it extends further than proposed.** The issue asked
whether twin-missed and vacuous verification are the same underlying error. They are — and
symptom-keyed guards belong with them. All three are the same mistake at different targets:
- **twin-missed** — the *fix* was applied to a sample of the population,
- **vacuous-by-sampling** — the *verification* sampled an empty or unrepresentative subset,
- **symptom-keyed guard** — the *guard* matched only the member that was noticed.
**Size: 27 of 69 (39%)** — 9 twin-missed, 9 symptom-keyed, 9 vacuous-by-sampling. See §3.6 for the
full partition; the number is smaller than the naive merge because records that belong to Families
C and D are **not** double-counted here.
The merge is not merely rhetorical; it predicts a single detector, and the corpus already contains
that detector, reinvented several times without anyone noticing it was the same rule:
> "the by-id handler covered 4 of 10 media types… **Sweep by FIELD, not by the call site the issue
> names**" — #671
> "#644's guard keyed on the *symptom* (an inflated pageSize) rather than the *defect*… an at-cap
> request was structurally invisible to it" — #650
> "filtered on tools that *already* declared `pageNum`, so a tool wrapping a paged endpoint with no
> paging args escaped it entirely" — #616
> "the pre-existing test filters `Where(t => t.QueryParameters is {Count: > 0})`, so a tool that
> lost its query parameters escaped it entirely" — #757
Four sessions, four reinventions of one rule → **detector A** in §4.
**Scope limit, because it bounds the highest-value detector in this document.** The distinction is
what kind of thing the population is made of.
- **A population of *values*** — enum members, OpenAPI operations, registered tools — always has an
external authoritative source, and detector A applies directly. `Enum.GetValues<T>()` is that
source for an enum, which is precisely #503's fix for a catch-all `default` that silently absorbed
an unhandled value.
- **A population of *sites in code*** — the places that dispatch on a value, rather than the values
themselves — has no external list to assert against. #403 is the case: `PlaybackOrder` is an enum,
so its *values* are enumerable, but the defect was that 5 of 6 **dispatch sites** failed to handle
one, and no artifact anywhere enumerates those sites. That residue needs find-all-references
tooling, which is broken here and tracked in #777, not a set-equality assertion.
### Family B — the check never ran at all
**Size: 9 of 69 (13%)** — the 8 vacuous records that are not sampling errors, plus the one masking-guards
record. Here the check was *dead*, not mis-aimed. `scripts/tests/`
invoked by no CI job (#631); a validator whose call site could be deleted with the suite still green
(#621); a `${{ }}` in a shell **comment** silently dropping a step while the job reported success in
6s (#751); `if ! cmd; then status=$?` reading bash's logical negation, so a failing spec run exited 0
(#445); new pre-push logic never wired to receive stdin, "dead code that every unit test still passes
over" (#719). A sampling detector does nothing here. This needs **detector B**: proof the guard can
go red.
### Family C — two copies of one thing
**Size: 9 of 69 (13%)** — the 5 wrapper-drift records plus the 4 twin-missed records whose twin is
literally a second copy (#510, #649, #711, #756). Those 4 are counted **here and not in Family A**.
`.claude/` and its byte-identical `.codex/` mirror where only one was in the `PROTECTED` list (#711);
an advisory local hook and the server-enforced workflow where four rounds of hardening landed on the
copy with the lower stakes (#649); a hand-maintained MCP schema drifting from the generated OpenAPI
DTO by one field and silently clearing it (#754, again #757); a hand-rolled frontmatter parser
accepting YAML that PyYAML rejects (#674); a dropped-step guard added to one required workflow and
not its fail-open twin (#756).
The preferred fix is **deletion** — one implementation, called from both places, as #649 did by
extracting `scripts/pr-changed-files.sh` and recording `ci.shared-pr-file-enumeration`. (#649 also
shows why the duplication is dangerous: before the extraction the guard was "safe only by
redundancy," and a failing exit status "previously left the *entire* suite green.")
**But dedup is a preference, not a law, and #711
is the counterexample that proves it.** That record deliberately *kept* its enumerative list:
*"the list stays ENUMERATIVE rather than derived — a derived rule would have to be evaluated against
the very file list being classified, putting more moving parts inside a security predicate to save
one line per new tooling directory."* Deriving a security predicate from the input it judges is
worse than maintaining two entries. So: dedup by construction where the duplication is incidental;
keep the enumeration where deriving it would feed the judged input back into the judge.
### Family D — check-and-use race over mutable state *(the taxonomy missed this entirely)*
**Size: 5 of 69 (7%).** Three raters proposed overlapping new classes — *unprompted by the taxonomy*,
which had no bucket for this, though they shared a corpus and a brief and so were not strictly
independent. The stronger evidence is the **blind** re-rater, who saw none of their output and
proposed the same class again for #536 under its own name (`toctou-partial-atomicity` vs
`half-atomic-toctou`).
Four of the five (#536, #622, #706, #707) were tagged `NEW` by their raters. **#632 was rated
twin-missed as its primary**, with the race named as its secondary; it sits here because the
mechanism is the race, and it is counted here rather than in Family A.
| # | The race |
|---|---|
| 536 | Check-then-act split by an `await`; the write side was atomic, the read side a stale `Volatile.Read` |
| 622 | Merge consent bound to a head sha, then Gitea's async auto-merge evaluated against a later head |
| 632 | Head sha bound, but retargeting the PR base changes the diff while moving neither sha nor status |
| 706 | Status writes are read-then-write with no compare-and-set; an older run can finish last and win |
| 707 | Paged file enumeration diffs each page against the base's *live* tip, so a mid-paging advance drops rows |
**Five instances — larger than identity-not-capability, string-predicate churn and overclaim
individually**, and #773's taxonomy had no bucket for it. The unifying property:
a check and the action it authorizes are separated in time over state that can change in between,
with nothing pinning a version. #622's own record puts it exactly: *"The gate was never bypassed —
it was satisfied against a snapshot that stops being true."*
### Two further cross-era classes, from the pre-convention sample
- **Fail-open by default** — a surface that defaults to permissive when config is absent or a step
vanishes: API writes open when `Api:WriteKey` is unset (#280); a dropped step is fail-CLOSED in
`review-verdict.yml` but fail-OPEN in `docker-build.yml` (#751, #756, #768); a jq-1.6-inert
classifier (#647). Distinct from identity-not-capability: nothing is mis-scoped, the *default*
is wrong.
- **Destructive replace** — a full-replace path that silently destroys state a reconcile-by-id
would have kept: a schedule PUT resetting fill-group progression, which #252 fixed with a
"scoped positional/no-op reconcile" so that a no-op PUT-back stops being destructive; a
full-replace wrapper missing one field and clearing it (#754); a dedup fix that turned a
duplicate row into permanent data loss because the remove filter used a different key (#500).
### 3.6 The partition, stated exactly
The families overlap conceptually — a twin-missed record can be read as "another member of a
population" or as "a second copy" — so membership is given as an explicit partition, **mechanically
verified to cover all 69 records exactly once**. Read a family's n as a disjoint count, not a tally
of everything the family's description could fit:
Family letters A–E are the ones §4 names detectors for, and those letters correspond. **This table
assigns no letters below E.** §4's detectors F and G address fail-open and destructive-replace,
which are cross-era classes drawn from the pre-convention sample and are *not* members of this
69-record partition — so an F or G in §4 refers to nothing in this table.
| Family | n | % | Composition |
|---|---|---|---|
| **A** population reasoned about via a sample | 27 | 39% | 9 twin-missed + 9 symptom-keyed + 9 vacuous-by-sampling |
| **B** the check never ran | 9 | 13% | 8 vacuous-by-non-execution + 1 masking-guards |
| **C** two copies of one thing | 9 | 13% | 5 wrapper-drift + 4 twin-missed-by-duplication |
| **E** environment divergence | 7 | 10% | |
| **D** check-and-use race over mutable state | 5 | 7% | 4 rated `NEW` + #632 rated twin-missed with the race secondary |
| — overclaim / stale claim | 3 | 4% | no detector — see §4 |
| — string-predicate churn | 3 | 4% | no detector proposed; see §3.7 — the round-churn it is named for is cross-cutting |
| — identity-not-capability | 2 | 3% | |
| — unmerged singletons | 4 | 6% | #60, #503, #586, #688 — real, but one instance each |
| **Total** | **69** | — | the **n** column is exact; percentages are rounded to the nearest point and sum to 99% |
The partition is checked mechanically rather than by eye: the assertion is that the union of the
families equals the set of process-failure records, with no overlap. That check immediately caught
a transcription slip (#438 omitted from the control set) — a small live demonstration of detector A
applied to this analysis's own numbers.
Two caveats travel with these percentages. The Family A/C boundary is a judgment about whether a
twin is "a second copy" or "another member of a population", and #649 is exactly the record the two
raters split on. And per §1 limit 2, these are proportions **over a recent, guard-heavy corpus** —
the backward sample establishes that the shapes existed before, not that they occurred in these
ratios.
### 3.7 Round-churn is a property, not a class — and the fix text is the fourth sampling target
The ranked table lists **string-predicate churn** as a class, inherited from #773's own taxonomy
("a parser / string-matching predicate needing repeated rounds"). That name identifies a *substrate*
when the evidence identifies a *cross-cutting property*. Counting records that narrate three or more
review rounds:
| Family | n | records with ≥3 rounds |
|---|---|---|
| A — population via a sample | 10 | #460, #496, #616, #633, #644, #650, #671, #684, #726, #743 |
| B — the check never ran | 5 | #445, #620, #631, #685, #751 |
| C — two copies of one thing | 5 | #510, #649, #754, #756, #757 |
| E — environment divergence | 5 | #491, #643, #647, #648, #668 |
| D — check-and-use race | 3 | #622, #632, #706 |
| identity-not-capability | 2 | #697, #698 |
| **string-predicate churn** | 2 | #578, #629 |
| overclaim | 1 | #651 |
| **Total** | **33 of 69** | every family represented |
*Method: every one of the 69 records was read for round language — 48 mention "round" at all, 33 state
an explicit count of three or more. Two attributions are loose: #647 narrates four rounds that
happened on #643, and #629 counts a round it deliberately did not run.*
**The class named after the phenomenon holds 2 of the 33.** Round-churn should therefore be read the
way §2 reads overclaim — a modifier riding on another failure, not a class with its own detector.
#697 settles it independently: three BLOCKED rounds on credential scoping, no parser anywhere near it;
#698 ran six.
One honest qualification against over-correcting. String predicates *are* disproportionately
round-prone — 2 of that class's 3 records, against 10 of Family A's 27. The naming error is not that
the association is false; it is that the class was defined by the substrate where the property was
noticed, so 31 instances outside it had nowhere to be counted.
**Why the rounds happen, which is the part that generalises.** §3 Family A names three things that
get sampled instead of enumerated — the fix, the verification, the guard. There is a fourth: **the
review scope.** A re-review briefed to "check the reported finding" samples; the population is the
whole changed artifact, and the text written to *fix* the last round is unreviewed by construction at
the moment it is written. That is why each round's defect lands in the newest prose rather than in
the text under review.
**Two limits on this section, since it is the one making a claim about its own production.**
- **33 is a floor, not a total.** It counts records that *state* a round count; a record that ran
four rounds without narrating them is invisible here. Do not read the 36 remaining records as
single-round work.
- **Counting this by pattern rather than by enumeration under-reports it by roughly half.** A regex
over the corpus finds 18 of these 33 — the misses narrate their rounds in the same words as the
hits. If you re-derive this number, read all 69.
- **#767 — the case #773 describes as taking eight review rounds — is excluded by snapshot timing,
not by absence.** The corpus was drawn at 19:45 on 2026-08-13; #767 closed at 20:04, nineteen
minutes later, and now carries a full `## Closing record`. So the worst instance of the phenomenon
is missing from every count in this document, including the 33 above. It is named here rather than
quietly folded in, because re-drawing the corpus would move every denominator in §2 and §3.6 and
the snapshot boundary has to sit somewhere.
Reading its record changes nothing structurally and confirms Family A twice over — it reinvents
detector A independently, for the fifth and sixth times in this repo: *"Refuting two variants of a
channel is not clearing the channel"* (the working attacks through that channel were never tried),
and *"Enumerating **shapes** of a command loses. Nine disarms across two rounds; running the
command settles them together."* That last sentence is detector A in one line — stop sampling the
shapes you thought of, execute the population.
---
## 4. Detectors
Ranked by instances covered per unit of build cost. "Rule status" distinguishes a **docs problem**
(no rule exists) from a **hooks problem** (the rule exists and nothing enforces it) — they have
different fixes, and conflating them is why several of these recurred.
| # | Detector | Covers | Rule status |
|---|---|---|---|
| **A** | **A guard derives its expected set from the authoritative source and asserts set equality — never `filter`, never a sample.** The population comes from the enum / the OpenAPI doc / the workflow YAML / the provider list, and the assertion is equality against it. A filter over the population cannot see the member that is missing. | Family A: 27 records (39%), minus the code-structural residue noted in §3 | **rule-missing** as a general rule; instantiated ad hoc in #616, #644→#650, #671, #757 |
| **B** | **Every guard ships with a proof it can go red: delete or disarm *that guard alone* and the suite must fail.** Not "a test exists" — a mutation. | Family B: 9 records (13%) | **rule-present-unenforced** — stated in #685's record and in project memory, enforced nowhere |
| **C** | **Dedup by construction.** When two copies of one rule exist, delete one and have both callers invoke it — *unless* deriving the list would feed the judged input back into the judge, which is #711's reasoned exception. | Family C: 9 records (13%) | **rule-missing** |
| **D** | **Anything read-then-written against live Gitea/remote state pins a version or uses compare-and-set.** See the honesty note below — this is a fix pattern, and its *detector* is detector A applied to an enumerated inventory of such sites. | Family D: 5 records (7%) | **rule-missing** (partially addressed for the merge path by the per-sha required check from #622) |
| **E** | **Run the check under the interpreter/provider it will actually run under**, and preflight-log the version. | Environment divergence: 7 records | **rule-present** and working — `scripts/jq-preflight.sh` + `ci.jq-version-contract` closed the jq axis after #643/#647/#648. **Unclosed axes**: SQLite-vs-MySQL query semantics (#668), GPU generation (#505), CI-VM speed (#512) |
| **F** | **Test the DENY path with the production config value.** #756's lesson generalised: a fixture that omits a field tests only the default, so a fail-open in the production value stays invisible. Parametrise the whole matrix, including "config absent". | Fail-open-by-default: ~5 records | **rule-missing** |
| **G** | **Full-replace endpoints assert their complete field list in a test**, and prefer reconcile-by-id over delete-and-reinsert where child state exists. | Destructive replace: ~3 records | **rule-missing** |
**Detector A is the highest-value single change in this analysis** — one convention, 39% of the
recorded process failures over this corpus, and it is already proven four times in this repo under
four different names. Two honesty notes on that ranking, because it drives #774's priority:
- **The 39% is measured over a recent, guard-heavy corpus** (§1 limit 2). The backward sample shows
the shape predates that corpus; it does not show the *proportion* holds across eras. Read it as
"the largest family in the work we have been doing lately," which is still the right basis for
prioritising the next change, but not as a timeless property of the project.
- **Detector A also applies to review scope, which is free.** Per §3.7, a re-review briefed to
"check the reported finding" samples the artifact. Briefing it to sweep the whole changed artifact
enumerates it. This costs one sentence in a review brief and is the only detector here with no
build step at all.
- **Detector D is weaker than its neighbours in this table and is listed anyway.** A, B, F and G
each name a check that fails when violated. D names a *fix pattern* with no general lint — you
cannot mechanically spot "this code should have pinned a sha." What makes it actionable is that
the population is small and enumerable: the handful of scripts that touch live remote state. So
D's real detector is detector A applied to that inventory, which is why #778's scope is "enumerate
every such site and mark each pinned / CAS / knowingly-unsafe" rather than "write a linter."
### Classes where no mechanical detector is plausible — stated rather than papered over
The issue explicitly asked for this, and inventing a weak detector here would itself be the
symptom-keyed-guard mistake.
- **Overclaim / stale claim.** No check can tell that a sentence is stronger than the evidence
behind it. Partial mitigations exist and should not be oversold: `stale-after` frontmatter (#603)
dates a claim, and #578's retracted-term grep catches a *specific known* retraction propagating
into generated artifacts. Neither detects a fresh overclaim. This stays a review responsibility.
- **Per-task review blindness** (#60). A review scoped to one task's diff structurally cannot see a
defect that only exists once several tasks compose — five Important findings at #60 were invisible
to six per-task reviews and surfaced only in a whole-branch pass. The fix is a *process* step
(a whole-branch review before close), not a check.
- **Omitted brief constraint** (#586). A delegated brief silent on a hazard gets a plausible-but-wrong
default — "an omitted rule isn't an unenforced rule, it's a rule replaced by whatever default the
agent reaches for." A brief lint is conceivable but would be a keyword matcher, i.e. exactly the
string-predicate class. A hazards checklist in the brief template is the honest ceiling.
- **Guard-parity by verb** (#458), and re-deriving an inherited exemption against a new failure mode
(#484). Both need judgment about whether a prior rationale still applies.
### A meta-finding about this repo's own knowledge base
`docs/decisions/records/` holds **189 active records**, and the two most relevant to this analysis
(`testing.enumerating-guard-identity-not-position`, `ci.required-job-step-execution-markers`) are
extraordinarily detailed — each a full account of one incident. The corpus grows **one record per
instance**. That is Family A operating on our own process: we are enumerating cases rather than
removing the mechanism. Detector A, C and B are class-level rules precisely because the per-instance
record has already been tried 189 times.
---
## 5. Part 2 — audit of the tooling already configured
Judged against the measured classes above, per the issue's sequencing. Everything here was run, not
assumed.
### 5.1 The LSPs: two of three are broken, and the one that matters most is the most broken
#773 asked whether routine LSP use would catch Family A (find *all* references rather than the one
in front of you). It would help — **and it is not available.** Measured in this session:
| LSP | State | Evidence |
|---|---|---|
| `csharp-lsp` | **Broken — cannot initialize** | `findReferences` on `ChannelPlaylist.ToM3U()` → `System.InvalidOperationException: .NET SDK cannot be resolved, because libhostfxr.dylib cannot be found inside /opt/homebrew/Cellar/dotnet/10.0.302/bin/host/fxr`. That directory **does not exist** — Homebrew's dotnet layout is not what MSBuildLocator expects. `dotnet --version` works (10.0.302), so builds are fine; only the language server is dead. |
| `typescript-lsp` | **Broken at the repo root** | `Could not find a valid TypeScript installation… ensure that the "typescript" dependency is installed in the workspace`. `typescript` lives in `web/node_modules`, not at the workspace root. Configuration problem, not a missing dependency. |
| `pyright-lsp` | **Works** | `documentSymbol` on `scripts/decisions_lib.py` returned the full symbol tree; `findReferences` on `active_files` correctly returned 4 references across 3 files, including cross-file hits in `decisions_validate.py` and `migrate_decisions_split.py`. |
**Two conclusions, and the second is the sharper one.**
1. The LSP that would help most is the one that is dead. The residue detector A cannot reach —
populations of *sites in code* rather than of values — is overwhelmingly C#: #403 (5 of 6
dispatch sites) and #671 (a by-id handler covering 4 of 10 media types) are "find every site"
problems, and `csharp-lsp` cannot answer a single query. (#510 is *not* an example here despite
looking like one: its record names duplication as the root cause and its switch keys on a plain
enum, so it belongs to Family C and `Enum.GetValues<T>()` is its authoritative source.)
2. **A dispatched subagent could not reach the LSP tool at all.** The agent tasked with testing the
three LSPs reported `ToolSearch` returning "No matching deferred tools found" for every query,
while the same tool resolved immediately in this main session. So the standing note
*"workflow agents must use csharp-lsp"* is doubly rotten: the server is broken **and** the agents
it addresses cannot invoke it even when it works. This is a live instance of overclaim/stale
claim (§2 rank 7) sitting in our own guidance.
### 5.2 Python: the global instruction and this repo disagree, and the repo is silent
`~/.claude/CLAUDE.md` instructs `ruff check`, `ruff format --check`, and `pyright` after modifying
Python. Measured against reality:
| Claim | Measured |
|---|---|
| ruff/pyright run in CI or hooks | **No.** `grep -rn "ruff\|pyright" .gitea/workflows/ .husky/` returns **nothing**. |
| What *does* run on Python | `pr-checks.yml` → `decisions-guard` (`decisions_validate.py`, `build_decisions_catalog.py --check`) and `script-tests` (`pytest scripts/tests`). Neither Husky hook touches Python. |
| `ruff check scripts/` | **47 errors** — but 28 are `E702` (semicolons) and **7 `S105` "hardcoded password" are false positives** on test stubs (`env["ETV_GITEA_TOKEN"] = "stub"`). The real cleanup is small. |
| `ruff format --check scripts/` | **10 of 20 files would be reformatted.** |
| `pyright scripts/` | **2 errors**, both `reportMissingImports` for `etv_client` in `scripts/scripted-schedules/entrypoint.py` — a package resolvable only in that script's deploy environment. Effectively clean. |
| Repo-level ruff config | **None.** No `ruff.toml`/`pyproject.toml`. Ruff silently falls back to whichever `~/.config/ruff/ruff.toml` the operator's machine happens to have. |
**The last row is the finding that matters**, and it is Family E (environment divergence) in our own
toolchain: Python lint behaviour here is a function of an un-versioned file on one laptop. A second
machine lints differently, or not at all. One of the two must move — either the repo adopts a
committed ruff config and enforces it, or the global instruction stops claiming this repo enforces
something it does not.
### 5.3 Configured vs actually invoked
Measured over the session transcript corpus (811 files under
`~/.claude/projects/-Users-timothy-ersatztv/`, this session excluded).
| Tool | Configured | Actually invoked | Verdict |
|---|---|---|---|
| `gitea` MCP | project `.mcp.json` **and** user scope (duplicate, identical values) | Heavy, same-day | Keep; de-duplicate the config |
| `ssh-mcp` | project `.mcp.json` | 16 transcripts, same-day | Keep |
| `mempalace` MCP | user scope | 10 transcripts, **last hit 2026-07-25** (19 days) | Keep, but see below |
| **`LSP` (all three servers)** | 3 plugins enabled + `csharp-lsp` in `enabledMcpjsonServers` | **0 calls in 811 transcripts** | Broken *and* unused |
| `context7` | enabled, `CONTEXT7_API_KEY` set | 0 calls | Dead |
| `playwright` | enabled | 145 calls, last 2026-07-25 | Keep |
| `superpowers` | enabled | 29 transcripts, last 2026-08-04 | Keep |
| `feature-dev`, `ralph-loop`, `security-guidance` | enabled | 0 skill invocations | Dead as configured |
| `codex` plugin | enabled | 0 *skill* invocations — but the documented workflow is `codex exec` via Bash, which this search cannot see | Not dead; measured the wrong surface |
| `nuget`, `docker-mcp` | in `.mcp.json`, explicitly **disabled** | 0 (expected) | Correctly off |
The zero for `LSP` is verified rather than assumed: the same query shape returns 23,661 `Bash` calls
and 5,102 `Read` calls over the same corpus, so the search was demonstrably capable of finding hits.
**Re-derive it with that positive control**, not on its own — `rg -c` over multiple files prints
`path:count`, so the obvious way to total it sums the paths and returns zero for everything.
**Hooks are the good news.** All 13 scripts in `.claude/hooks/` are wired from either
`.claude/settings.json` or `.husky/*`, and no settings entry points at a missing path — there are
**no dead hook scripts**, contrary to the issue's suspicion. Husky hooks have genuine fired-output
evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message
missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`).
### 5.4 We cannot currently tell whether our own hooks fire
The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own
evidence. In this harness version only **`Stop`** hooks emit a structured record
(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that
matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**.
What the audit could count for those hooks was filename mentions in settings dumps and prior audits,
which is not proof of execution. Only `design-sync-reminder.sh`'s Stop path and the three husky
message strings quoted in §5.3 are confirmed fires; the `.claude/hooks/` scripts are confirmed
*wired*, which is a different claim.
So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That
is Family B (the check never ran) applied to the hook layer itself, and this project has already
paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and
#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it
rather than infer it.** A one-line append to a session-scoped log from each hook would convert every
row in that table from inference to measurement, and it is cheap.
### 5.5 Tools we do *not* have that would address a named class
Each is justified against specific issues, not general merit; anything that could not be tied to
recorded instances is left out rather than padded in.
| Tool | Class it addresses | Justification |
|---|---|---|
| **Stryker.NET** (mutation testing for C#) | Family B — detector B, mechanised | Detector B currently relies on an author remembering to write a mutation proof. Stryker generates them. #621 (a guard whose call site could be deleted with the suite green), #685 (two guards where deleting either left the suite green) and #719 (logic never wired to stdin) are all surviving-mutant detections by construction. Cost is real (mutation runs are slow), so scope it to the guard/validator projects rather than the whole solution. |
| ~~`shellcheck`~~ — **tested and rejected** | (would have been Family B, shell half) | The obvious candidate for #445 (`if ! cmd; then status=$?`, where bash sets `$?` to the *logical negation* so a failing run exited 0). **Measured: it does not catch it.** ShellCheck 0.11.0 on that exact construct reports nothing, and even `-o all` returns only an unrelated brace-style nit; run against our 13 hook scripts it yields 2 `SC2034` unused-variable warnings. Recorded here as a negative result so it is not proposed again on plausibility. |
| **A duplicate-code detector** (`jscpd` or equivalent) | Family C | #440 shipped two byte-identical Lucene escapers in different files; #711 and #649 are the same shape at the config/CI layer. A duplication report would have surfaced all three at authoring time — subject to #711's exception, so it should advise, not block. |
| **An enum-exhaustiveness analyzer rule** for C# `switch` *statements* | Family A, code-structural residue | #503's catch-all `default` silently absorbed an unhandled enum value and produced byte-identical output to a handled one. Switch *expressions* already warn; statements do not. This is a rule to enable in the analyzers we already run (Meziantou), not a new dependency — the cheapest item here. |
**Explicitly not proposed:** a "brief linter" for omitted delegation constraints (#586) and any
overclaim detector. §4 argues both are implausible, and inventing them here to look thorough would
be the symptom-keyed mistake. Nothing in the corpus suggests a tool we lack would beat detector A
for Family A — the four in-repo reinventions show the convention works when applied; the failure is
that it was never written down once.
**Evidence status of these rows, which differs and matters.** Only the shellcheck row was
*executed* — and running it refuted the reason it had been proposed, which is why it is struck
through. The three surviving rows are **reasoned from precedent**: each names issues whose recorded
mechanism the tool addresses, but none has been run against this repo to confirm it fires. That is a
weaker warrant and should be discharged before any of them is adopted — a tool justified by "it
plausibly catches this class" is the same species of claim as the guards this document criticises.
Treat these three as candidates to test, not findings.
**On MemPalace**, the 19-day gap deserves care rather than a verdict: `CLAUDE.md` makes it the
documented discovery path for decisions, so either sessions are skipping the documented path, or
they are correctly reading `docs/decisions/README.md` directly (which the same contract permits, and
which is authoritative). The measurement cannot distinguish those, and I am not going to guess.
---
## 6. Follow-ups
This issue is analysis; it spawns implementation rather than doing it.
| Issue | Detector / finding | Covers | Priority |
|---|---|---|---|
| #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, 27 records (39%) | high |
| #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B, 9 records (13%) | high |
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high |
| #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium |
| #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium |
| #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |
| #780 | Reconcile Python tooling — commit a ruff config and enforce it, or stop claiming we do | environment divergence in our own toolchain | medium |
| #781 | Retire enabled-but-never-invoked plugins/MCP servers; de-duplicate the `gitea` server | config that reads as coverage | low |
**Detector C (dedup by construction) has no issue of its own on purpose.** It is not a thing to build;
it is the shape the fixes in #774 and #778 should take when they find two copies of one rule — with
#711's exception carried along, since that record deliberately keeps its enumeration and is a
counterexample rather than a supporting case.
Not filed, deliberately: overclaim/stale-claim, per-task review blindness, and omitted brief
constraints. §4 argues no plausible mechanical detector exists for these, and filing an issue for
each would produce exactly the weak enumerating guard this analysis recommends against.