docs(773): root-cause the recurring defect shapes across the full closed-issue corpus
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 12s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 16s
review-verdict/h10 Exempt: docs-only change (no code, no protected path)
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 10s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 8s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 8s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
PR Gates / Script tests (pytest) (pull_request) Successful in 2m14s

Analysis over all 349 closed issues (95 carry a `## Closing record`), classified
by five independent raters against a written taxonomy, with an inter-rater control
(12/15 blind agreement) and a backward-generalization check over the 254
pre-convention issues.

Findings that change the picture the issue started from:

- The ranking reverses. Vacuous verification is the most common shape (17/69),
  not twin-missed (14/69), over the full corpus vs the issue's 50-issue sample.
- The issue's central hypothesis holds and extends: twin-missed, vacuous-by-
  sampling and symptom-keyed guards are one error (reasoning about a
  representative instead of the population) — ~32/69 (46%), one detector.
- The shapes predate the closing-record convention (#1, #215, #232, #403), so
  they are not an artifact of recent guard-building work.
- A class the taxonomy missed entirely: check-and-use races over mutable state
  (#536, #622, #632, #706, #707) — five instances, proposed independently by
  three raters and re-derived by a fourth, blind.
- Overclaim drops to 4% as a primary cause; it is a modifier, not a class.

Part 2 (tooling audit), all measured rather than assumed: csharp-lsp cannot
initialize and typescript-lsp cannot resolve typescript, the `LSP` tool has 0
calls across 811 transcripts, ruff/pyright are enforced nowhere despite the
global instruction, and PreToolUse/PostToolUse hook execution leaves no durable
trace — so we cannot currently tell whether our own guards fire.

Names the classes where no mechanical detector is plausible rather than
inventing weak ones. Spawns #774-#781; this issue is analysis, not implementation.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-13 20:01:25 +02:00
co-authored by Claude Opus 5
parent f9f8f65cee
commit c3ccfa95e0
2 changed files with 358 additions and 0 deletions
+7
View File
@@ -24,6 +24,7 @@ doc below, or that changes which sections a task signal points to.**
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible) |
| Live local run / Playwright-MCP verification | `docs/e2e-local.md` + `scripts/e2e-local.sh` |
| Adding/changing a UI-E2E browser flow | `docs/e2e-local.md` → "UI-E2E harness" + `scripts/e2e-ui.sh` |
| What does a test suite cover | `docs/testing.md` |
@@ -98,6 +99,12 @@ bounds, what's mined per issue): `docs/handoffs/chicorytv-issue-queue.md` → "K
that file's standing kickoff for the two concurrent tracks (orientation ‖ selection). ersatztv#237
is a closed, archival historical tracker (superseded by `startup.parallel-orientation` in
`docs/decisions.md`) — not a live pointer.
- **`docs/defect-shapes-773.md`** — root-cause analysis of the recurring defect shapes across the
whole closed-issue corpus (#773): the measured class ranking, the four families they consolidate
into, the cheapest mechanical detector per class, the classes where **no** detector is plausible,
and an audit of which configured hooks/MCP servers/LSPs are actually invoked. Read it before
proposing a new guard or CI check — §4 is the detector menu, and it argues against enumerating
cases one incident at a time.
- **`docs/tracker-retrofit-triage-237.md`** — audit trail for the #524 triage of ersatztv#237's 111
comments (method, per-comment classification, totals). Evidence for the
`docs.tracker-comment-retrofit` decision; read it only when triaging another over-cap tracker.
+351
View File
@@ -0,0 +1,351 @@
# Recurring defect shapes across the closed-issue corpus (ersatztv#773)
Root-cause analysis over every closed issue in the tracker, the detectors that would have caught
each class, and an audit of the tooling already configured in this repo. This is an **analysis
artifact**, not a rulebook: the rules it argues for land as decision records and follow-up issues,
which are linked per class below.
Measured 2026-08-13 against the tracker as of issue #773 and `origin/main` at `f9f8f65ce`.
---
## 1. Method, and what these numbers can and cannot support
**Corpus.** All 349 closed issues (`state=closed&type=issues`, 7 pages, 349 distinct numbers,
#1–#757 — pull requests excluded). Of those, **95 carry a `## Closing record`** comment. Each
record was classified by one of five independent agents against a written taxonomy, with
`NEW:<name>` available so the taxonomy could not be self-fulfilling. A sixth agent re-rated a
stratified 15-issue sample blind, as an inter-rater control. A seventh sampled 25 of the 254
pre-convention closed issues from their raw bodies and comments, to test whether the shapes exist
outside the era that writes records.
**Three limits, stated up front because they bound every number below.**
1. **Record coverage is not uniform.** The `## Closing record` convention starts at #520
(2026-07-21). Coverage is 31/32 for #600–699 and 4/69 for #1–99. So frequencies over the 95
records are frequencies over *recent* practice.
2. **Corpus composition shifts underneath the measurement.** The share of records describing no
process failure at all falls from 12/19 (#60–#438) to 0/19 (#616–#671). Early issues are
feature work; recent issues are largely CI/process hardening. Guard-shaped defects therefore
dominate the recent records *partly by construction* — we have lately been building guards, so
our defects are in guards.
3. **A closing record is the closing author's self-report.** Raters were told to flag records whose
own claims outrun what they describe, and several did (#60, #651, #691, #720). But the corpus
systematically cannot contain a defect nobody noticed.
**What limit 2 does *not* undermine.** The obvious worry — that these classes are an artifact of
recent guard-building — was tested and **refuted**. The backward sample found high-confidence
instances of the same shapes well before the convention existed: twin-missed at #215 (REST playout
mutations skip the build-lock gating the Blazor path had) and #403 (an unhandled enum silent at 5 of
6 dispatch sites); wrapper drift at #198 and #287 (OpenAPI spec diverged from runtime JSON);
masking guards at #232 and #234 ("one bug papering over another"); overclaim at #1 ("the earlier
'confirmed' was based on one curl test and several unverified assertions"). The convention changed
the **density and legibility** of the evidence, not the existence of the shapes. Several older
issues (#94, #157, #172, #265, #377) genuinely cannot be classified from what they record, and are
reported as `insufficient-evidence` rather than forced into a bucket.
---
## 2. What the corpus actually says
95 records; **69 describe a process failure**, 26 describe none. Percentages are of the 69.
| Rank | Class | n | % | Shape |
|---|---|---|---|---|
| 1 | **Vacuous verification** | 17 | 25% | A check was green having done no work |
| 2 | **Twin-missed** | 14 | 20% | The fix hit one instance and missed its structural twin |
| 3 | **Symptom-keyed guard** | 9 | 13% | The guard keyed on the symptom seen, not the defect's mechanism |
| 4 | *(new classes — see §3)* | 8 | 12% | Proposed by raters as fitting nothing above |
| 5 | **Environment divergence** | 7 | 10% | Behaviour differs by interpreter/provider/hardware, untested there |
| 6 | **Wrapper drift** | 5 | 7% | A hand-maintained mirror drifted from the contract it mirrors |
| 7 | **Overclaim / stale claim** | 3 | 4% | An assertion stronger than its evidence, or since gone false |
| 7 | **String-predicate churn** | 3 | 4% | A regex/grep predicate needing repeated rounds |
| 9 | **Identity-not-capability** | 2 | 3% | Authorization scoped by *who*, not by *what is being changed* |
| 10 | **Masking guards** | 1 | 1% | Two guards on one condition; one hid the other's total failure |
**This reorders the issue's own ranking.** #773 put twin-missed first (9 instances) and vacuous
verification second (8), from a 50-issue sample. Over the full record corpus they swap: **vacuous
verification is the most common single shape.** Overclaim also drops sharply as a *primary* cause
(4%) — raters overwhelmingly assigned it as a *secondary*. It is better understood as a modifier
that rides on another failure than as a class with its own detector.
**Inter-rater control (measured, not asserted).** A sixth rater re-classified a stratified 15-issue
sample blind. **Raw agreement 12/15 (80%)** across an 11-category scheme. The three disagreements
are informative rather than noise:
- **#177** (twin-missed → no-failure) and **#521** (no-failure → twin-missed) disagree at the
"did a process failure occur" boundary and point in **opposite directions**, so that boundary is
noisy but not systematically biased. The 26 clean-record count should be read as ±3.
- **#649** was rated twin-missed by one and wrapper-drift by the other — which is precisely the blur
that Family C in §3 exists to resolve. Both raters named the other class as secondary.
The top-three separation is far larger than the disagreement. Treat ±1 rank as noise.
---
## 3. Consolidation: four families, one of which the issue's taxonomy missed
### Family A — reasoning about a representative instead of the population
**#773's central hypothesis, confirmed, and it extends further than proposed.** The issue asked
whether twin-missed and vacuous verification are the same underlying error. They are — and
symptom-keyed guards belong with them. All three are the same mistake at different targets:
- **twin-missed** — the *fix* was applied to a sample of the population,
- **vacuous-by-sampling** — the *verification* sampled an empty or unrepresentative subset,
- **symptom-keyed guard** — the *guard* matched only the member that was noticed.
**Two numbers, because the honest one is smaller than the headline.** Merging the three classes as
rated gives **40 of 69 records (58%)**. But vacuous verification splits: only part of it is a
sampling error, and the rest belongs to Family B below. Assigning each of the 17 vacuous records to
one half or the other — **my split, not the raters'** — puts ~9 in the sampling half and ~8 in the
never-ran half. So Family A is **~32 of 69 (46%)**, and 58% is the upper bound that holds only if
you accept the issue's original hypothesis wholesale. Either way it is the largest family by a wide
margin.
The merge is not merely rhetorical; it predicts a single detector, and the corpus already contains
that detector, independently reinvented several times:
> "the by-id handler covered 4 of 10 media types… **Sweep by FIELD, not by the call site the issue
> names**" — #671
> "#644's guard keyed on the *symptom* (an inflated pageSize) rather than the *defect*… an at-cap
> request was structurally invisible to it" — #650
> "filtered on tools that *already* declared `pageNum`, so a tool wrapping a paged endpoint with no
> paging args escaped it entirely" — #616
> "the pre-existing test filters `Where(t => t.QueryParameters is {Count: > 0})`, so a tool that
> lost its query parameters escaped it entirely" — #757
Four sessions, four reinventions of one rule → **detector A** in §4.
### Family B — the check never ran at all
The other ~8 of the 17 vacuous-verification records are not sampling errors: the check was *dead*. `scripts/tests/`
invoked by no CI job (#631); a validator whose call site could be deleted with the suite still green
(#621); a `${{ }}` in a shell **comment** silently dropping a step while the job reported success in
6s (#751); `if ! cmd; then status=$?` reading bash's logical negation, so a failing spec run exited 0
(#445); new pre-push logic never wired to receive stdin, "dead code that every unit test still passes
over" (#719). A sampling detector does nothing here. This needs **detector B**: proof the guard can
go red.
### Family C — two copies of one thing
Wrapper drift plus a chunk of twin-missed: `.claude/` and its byte-identical `.codex/` mirror where
only one was in the `PROTECTED` list (#711); an advisory local hook and the server-enforced workflow
where four rounds of hardening landed on the copy with the lower stakes (#649); a hand-maintained MCP
schema drifting from the generated OpenAPI DTO by one field, silently clearing it (#754, again #757);
a hand-rolled frontmatter parser accepting YAML that PyYAML rejects (#674); two byte-identical Lucene
escapers in different files (#440). The honest detector here is not a parity check — it is
**deletion**: one implementation, called from both places.
### Family D — check-and-use race over mutable state *(the taxonomy missed this entirely)*
Three raters independently proposed overlapping new classes that consolidate into one — and the
blind re-rater, having seen none of their output, proposed the same class again for #536 under
its own name (`toctou-partial-atomicity` vs `half-atomic-toctou`). That is convergent validity, not
a prompt echo:
| # | The race |
|---|---|
| 536 | Check-then-act split by an `await`; the write side was atomic, the read side a stale `Volatile.Read` |
| 622 | Merge consent bound to a head sha, then Gitea's async auto-merge evaluated against a later head |
| 632 | Head sha bound, but retargeting the PR base changes the diff while moving neither sha nor status |
| 706 | Status writes are read-then-write with no compare-and-set; an older run can finish last and win |
| 707 | Paged file enumeration diffs each page against the base's *live* tip, so a mid-paging advance drops rows |
**Five instances — larger than wrapper drift, identity-not-capability, string-predicate churn and
masking guards individually**, and #773's taxonomy had no bucket for it. The unifying property:
a check and the action it authorizes are separated in time over state that can change in between,
with nothing pinning a version. #622's own record puts it exactly: *"The gate was never bypassed —
it was satisfied against a snapshot that stops being true."*
### Two further cross-era classes, from the pre-convention sample
- **Fail-open by default** — a surface that defaults to permissive when config is absent or a step
vanishes: API writes open when `Api:WriteKey` is unset (#280); a dropped step is fail-CLOSED in
`review-verdict.yml` but fail-OPEN in `docker-build.yml` (#751, #756, #768); a jq-1.6-inert
classifier (#647). Distinct from identity-not-capability: nothing is mis-scoped, the *default*
is wrong.
- **Destructive replace** — a full-replace path that silently destroys state a reconcile-by-id
would have kept: a schedule PUT cascade-deleting fill-group progression on every save, "even a
no-op" (#252); a full-replace wrapper missing one field and clearing it (#754); a dedup fix that
turned a duplicate row into permanent data loss because the remove filter used a different key
(#500).
---
## 4. Detectors
Ranked by instances covered per unit of build cost. "Rule status" distinguishes a **docs problem**
(no rule exists) from a **hooks problem** (the rule exists and nothing enforces it) — they have
different fixes, and conflating them is why several of these recurred.
| # | Detector | Covers | Rule status |
|---|---|---|---|
| **A** | **A guard derives its expected set from the authoritative source and asserts set equality — never `filter`, never a sample.** The population comes from the enum / the OpenAPI doc / the workflow YAML / the provider list, and the assertion is equality against it. A filter over the population cannot see the member that is missing. | Family A: ~32 records (46%); 40 (58%) if the whole vacuous class merges in | **rule-missing** as a general rule; instantiated ad hoc in #616, #644→#650, #671, #757 |
| **B** | **Every guard ships with a proof it can go red: delete or disarm *that guard alone* and the suite must fail.** Not "a test exists" — a mutation. | Family B + masking guards: ~9 records | **rule-present-unenforced** — stated in #685's record and in project memory, enforced nowhere |
| **C** | **Dedup by construction.** When two copies of one rule exist, delete one and have both callers invoke it. Prefer this over any parity check; a parity check is itself a Family-A guard that can be keyed wrong. | Family C: ~9 records | **rule-missing** |
| **D** | **Anything read-then-written against live Gitea/remote state pins a version or uses compare-and-set** — bind the base sha as well as the head, page against a pinned tip, make the status write conditional. | Family D: 5 records | **rule-missing** (partially addressed for the merge path by the per-sha required check from #622) |
| **E** | **Run the check under the interpreter/provider it will actually run under**, and preflight-log the version. | Environment divergence: 7 records | **rule-present** and working — `scripts/jq-preflight.sh` + `ci.jq-version-contract` closed the jq axis after #643/#647/#648. **Unclosed axes**: SQLite-vs-MySQL query semantics (#668), GPU generation (#505), CI-VM speed (#512) |
| **F** | **Test the DENY path with the production config value.** #756's lesson generalised: a fixture that omits a field tests only the default, so a fail-open in the production value stays invisible. Parametrise the whole matrix, including "config absent". | Fail-open-by-default: ~5 records | **rule-missing** |
| **G** | **Full-replace endpoints assert their complete field list in a test**, and prefer reconcile-by-id over delete-and-reinsert where child state exists. | Destructive replace: ~3 records | **rule-missing** |
**Detector A is the highest-value single change in this analysis** — one convention, ~46% of the
recorded process failures, and it is already proven four times in this repo under four different
names.
### Classes where no mechanical detector is plausible — stated rather than papered over
The issue explicitly asked for this, and inventing a weak detector here would itself be the
symptom-keyed-guard mistake.
- **Overclaim / stale claim.** No check can tell that a sentence is stronger than the evidence
behind it. Partial mitigations exist and should not be oversold: `stale-after` frontmatter (#603)
dates a claim, and #578's retracted-term grep catches a *specific known* retraction propagating
into generated artifacts. Neither detects a fresh overclaim. This stays a review responsibility.
- **Per-task review blindness** (#60). A review scoped to one task's diff structurally cannot see a
defect that only exists once several tasks compose — five Important findings at #60 were invisible
to six per-task reviews and surfaced only in a whole-branch pass. The fix is a *process* step
(a whole-branch review before close), not a check.
- **Omitted brief constraint** (#586). A delegated brief silent on a hazard gets a plausible-but-wrong
default — "an omitted rule isn't an unenforced rule, it's a rule replaced by whatever default the
agent reaches for." A brief lint is conceivable but would be a keyword matcher, i.e. exactly the
string-predicate class. A hazards checklist in the brief template is the honest ceiling.
- **Guard-parity by verb** (#458), and re-deriving an inherited exemption against a new failure mode
(#484). Both need judgment about whether a prior rationale still applies.
### A meta-finding about this repo's own knowledge base
`docs/decisions/records/` holds **189 active records**, and the two most relevant to this analysis
(`testing.enumerating-guard-identity-not-position`, `ci.required-job-step-execution-markers`) are
extraordinarily detailed — each a full account of one incident. The corpus grows **one record per
instance**. That is Family A operating on our own process: we are enumerating cases rather than
removing the mechanism. Detector A, C and B are class-level rules precisely because the per-instance
record has already been tried 189 times.
---
## 5. Part 2 — audit of the tooling already configured
Judged against the measured classes above, per the issue's sequencing. Everything here was run, not
assumed.
### 5.1 The LSPs: two of three are broken, and the one that matters most is the most broken
#773 asked whether routine LSP use would catch Family A (find *all* references rather than the one
in front of you). It would help — **and it is not available.** Measured in this session:
| LSP | State | Evidence |
|---|---|---|
| `csharp-lsp` | **Broken — cannot initialize** | `findReferences` on `ChannelPlaylist.ToM3U()` → `System.InvalidOperationException: .NET SDK cannot be resolved, because libhostfxr.dylib cannot be found inside /opt/homebrew/Cellar/dotnet/10.0.302/bin/host/fxr`. That directory **does not exist** — Homebrew's dotnet layout is not what MSBuildLocator expects. `dotnet --version` works (10.0.302), so builds are fine; only the language server is dead. |
| `typescript-lsp` | **Broken at the repo root** | `Could not find a valid TypeScript installation… ensure that the "typescript" dependency is installed in the workspace`. `typescript` lives in `web/node_modules`, not at the workspace root. Configuration problem, not a missing dependency. |
| `pyright-lsp` | **Works** | `documentSymbol` on `scripts/decisions_lib.py` returned the full symbol tree; `findReferences` on `active_files` correctly returned 4 references across 3 files, including cross-file hits in `decisions_validate.py` and `migrate_decisions_split.py`. |
**Two conclusions, and the second is the sharper one.**
1. The LSP that would help most is the one that is dead. C# is where Family A bites hardest — #671
(4 of 10 media types), #510 (three switch arms, not one), #403 (5 of 6 dispatch sites) are all
"find every site" problems in C#, and `csharp-lsp` cannot answer a single query.
2. **A dispatched subagent could not reach the LSP tool at all.** The agent tasked with testing the
three LSPs reported `ToolSearch` returning "No matching deferred tools found" for every query,
while the same tool resolved immediately in this main session. So the standing note
*"workflow agents must use csharp-lsp"* is doubly rotten: the server is broken **and** the agents
it addresses cannot invoke it even when it works. This is a live instance of overclaim/stale
claim (§2 rank 7) sitting in our own guidance.
### 5.2 Python: the global instruction and this repo disagree, and the repo is silent
`~/.claude/CLAUDE.md` instructs `ruff check`, `ruff format --check`, and `pyright` after modifying
Python. Measured against reality:
| Claim | Measured |
|---|---|
| ruff/pyright run in CI or hooks | **No.** `grep -rn "ruff\|pyright" .gitea/workflows/ .husky/` returns **nothing**. |
| What *does* run on Python | `pr-checks.yml` → `decisions-guard` (`decisions_validate.py`, `build_decisions_catalog.py --check`) and `script-tests` (`pytest scripts/tests`). Neither Husky hook touches Python. |
| `ruff check scripts/` | **47 errors** — but 28 are `E702` (semicolons) and **7 `S105` "hardcoded password" are false positives** on test stubs (`env["ETV_GITEA_TOKEN"] = "stub"`). The real cleanup is small. |
| `ruff format --check scripts/` | **10 of 20 files would be reformatted.** |
| `pyright scripts/` | **2 errors**, both `reportMissingImports` for `etv_client` in `scripts/scripted-schedules/entrypoint.py` — a package resolvable only in that script's deploy environment. Effectively clean. |
| Repo-level ruff config | **None.** No `ruff.toml`/`pyproject.toml`. Ruff silently falls back to whichever `~/.config/ruff/ruff.toml` the operator's machine happens to have. |
**The last row is the finding that matters**, and it is Family E (environment divergence) in our own
toolchain: Python lint behaviour here is a function of an un-versioned file on one laptop. A second
machine lints differently, or not at all. One of the two must move — either the repo adopts a
committed ruff config and enforces it, or the global instruction stops claiming this repo enforces
something it does not.
### 5.3 Configured vs actually invoked
Measured over the session transcript corpus (811 files under
`~/.claude/projects/-Users-timothy-ersatztv/`, this session excluded).
| Tool | Configured | Actually invoked | Verdict |
|---|---|---|---|
| `gitea` MCP | project `.mcp.json` **and** user scope (duplicate, identical values) | Heavy, same-day | Keep; de-duplicate the config |
| `ssh-mcp` | project `.mcp.json` | 16 transcripts, same-day | Keep |
| `mempalace` MCP | user scope | 10 transcripts, **last hit 2026-07-25** (19 days) | Keep, but see below |
| **`LSP` (all three servers)** | 3 plugins enabled + `csharp-lsp` in `enabledMcpjsonServers` | **0 calls in 811 transcripts** | Broken *and* unused |
| `context7` | enabled, `CONTEXT7_API_KEY` set | 0 calls | Dead |
| `playwright` | enabled | 145 calls, last 2026-07-25 | Keep |
| `superpowers` | enabled | 29 transcripts, last 2026-08-04 | Keep |
| `feature-dev`, `ralph-loop`, `security-guidance` | enabled | 0 skill invocations | Dead as configured |
| `codex` plugin | enabled | 0 *skill* invocations — but the documented workflow is `codex exec` via Bash, which this search cannot see | Not dead; measured the wrong surface |
| `nuget`, `docker-mcp` | in `.mcp.json`, explicitly **disabled** | 0 (expected) | Correctly off |
The zero for `LSP` is verified rather than assumed: the same query shape returns 23,661 `Bash` calls
and 5,102 `Read` calls over the same corpus, so the search was demonstrably capable of finding hits.
*(The first attempt at this count returned 0 for `Bash` as well — a `rg -c` output-format slip that
summed the file path instead of the count. A zero without a positive control is exactly the vacuous
verification this document is about.)*
**Hooks are the good news.** All 13 scripts in `.claude/hooks/` are wired from either
`.claude/settings.json` or `.husky/*`, and no settings entry points at a missing path — there are
**no dead hook scripts**, contrary to the issue's suspicion. Husky hooks have genuine fired-output
evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message
missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`).
### 5.4 We cannot currently tell whether our own hooks fire
The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own
evidence. In this harness version only **`Stop`** hooks emit a structured record
(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that
matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**.
The counts for those rows are filename mentions in settings dumps and prior audits, not proof of
execution. Only `design-sync-reminder.sh`'s Stop path and the three husky message strings are
confirmed fires.
So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That
is Family B (the check never ran) applied to the hook layer itself, and this project has already
paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and
#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it
rather than infer it.** A one-line append to a session-scoped log from each hook would convert every
row in that table from inference to measurement, and it is cheap.
**On MemPalace**, the 19-day gap deserves care rather than a verdict: `CLAUDE.md` makes it the
documented discovery path for decisions, so either sessions are skipping the documented path, or
they are correctly reading `docs/decisions/README.md` directly (which the same contract permits, and
which is authoritative). The measurement cannot distinguish those, and I am not going to guess.
---
## 6. Follow-ups
This issue is analysis; it spawns implementation rather than doing it.
| Issue | Detector / finding | Covers | Priority |
|---|---|---|---|
| #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, ~32 records (46%) | high |
| #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B + masking guards, ~9 records | high |
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high |
| #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium |
| #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium |
| #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |
| #780 | Reconcile Python tooling — commit a ruff config and enforce it, or stop claiming we do | environment divergence in our own toolchain | medium |
| #781 | Retire enabled-but-never-invoked plugins/MCP servers; de-duplicate the `gitea` server | config that reads as coverage | low |
**Detector C (dedup by construction) has no issue of its own on purpose.** It is not a thing to build;
it is the shape the fixes in #774 and #778 should take when they find two copies of one rule. Filing
it separately would invite a parity check, which is the weaker construction the corpus already
rejected twice (#649, #711).
Not filed, deliberately: overclaim/stale-claim, per-task review blindness, and omitted brief
constraints. §4 argues no plausible mechanical detector exists for these, and filing an issue for
each would produce exactly the weak enumerating guard this analysis recommends against.