Files
ersatztv/docs/decisions/records/testing/verification-code-needs-its-own-proof.md
T
timothyandClaude Opus 5 de8051dae6 fix(796): ship the proof the record was claiming an exemption from
Cold review BLOCKED the first commit on two findings, both correct, and both
refuted by execution rather than argument.

1. The central premise was false. The record said `scripts/mcp_smoke.py` "cannot
   participate" in the mutation harness because driving it needs the gitignored
   `.mcp.json` and a cold-built language server. It takes its config path and
   server name as POSITIONAL ARGUMENTS (`mcp_smoke.py:78`), so a test can hand it
   a synthetic config pointing at a stub responder. The record failed its own
   headline rule — that claim was read, not executed.

2. The split was drawn on the wrong axis. "CI cannot execute it" also describes
   every `.claude/hooks/*.sh` and `.husky/pre-push`, all of which carry MUTATION
   rows because a pytest proof drives them in a sandbox. The discriminator is
   whether a proof test can drive the construct HERMETICALLY.

So instead of restating the exemption, this ships the proof:

- `scripts/tests/test_mcp_smoke.py` — a stub JSON-RPC responder and five cases
  pinning the defects the checker has already had (impostor identity, a body
  carrying only an id, a missing expected tool, a pre-answered request), plus an
  anti-vacuity positive control. Hermetic: no `.mcp.json`, no language server,
  7.9s.
- A declared clause in `mutation_manifest.py` targeting `mcp_smoke.py`'s
  unguessable request id, using the `guard=test / target=script` shape that
  already exists for `mutation_harness_lib.py` — no population derivation changes.

Witnessed red, not asserted: replacing `id_init = secrets.randbelow(...)` with
`id_init = 1` makes the pre-answer ACCEPTED at `initialize`, so the run dies one
stage later at `tools/list` (rc 9 -> 10) and the proof reddens with its declared
diagnostic. Only that one test reddened; the mutation is clause-scoped.

`mcp_smoke.py` still gets no inventory row — a row for it is rejected as a
phantom, measured — so the row goes to the test file, which joins the population
automatically. Also corrected: the "acquires a real row" migration the mechanism
would refuse (HIGH 3); "shellcheck caught 0 of 5", re-measured as neither of the
TWO shell defects, the other three being Python and outside its reach (MEDIUM 4);
a threshold misattributed to `release.verdict-vocabulary-shared`, which carries a
categorical ban rather than a count (MEDIUM 5); an unverifiable "no shared
reviewers" claim (MEDIUM 6); a dangling cross-reference to a private memory
filename (LOW 7); and the scope-limit item that actually excludes defects 2/4/5
(LOW 8).

Docs updated in the same PR per CLAUDE.md: `docs/README.md`'s task-signal map now
routes to four guard rules rather than three, and `docs/guard-inventory.md`'s
scope limit item 6 records that being outside the population is not an exemption.

Refs: #796
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 19:55:31 +02:00

10 KiB

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
testing.verification-code-needs-its-own-proof 2026-08-28 — verification code is code under test: the obligation follows the VERDICT, and the exempt branch is what no proof test can drive hermetically (#796) active 2026-08-28 none none VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files sitting in the guard population or named like guards; every one of the five defects measured below was in code nobody had counted as a guard. THE DISCRIMINATOR IS WHETHER A PROOF TEST CAN DRIVE THE CONSTRUCT HERMETICALLY, NOT WHETHER CI RUNS IT, and the difference is load-bearing rather than pedantic: every `.claude/hooks/*.sh` and `.husky/pre-push` is operator-run and CI never fires them AS HOOKS, yet all carry `MUTATION` rows, because a pytest proof drives them in a sandbox. A does-CI-run-it test would sort those already-proven guards into the exempt branch. "IT NEEDS A REAL SERVER" IS AN EXEMPTION CLAIM AND IT IS USUALLY FALSE: `scripts/mcp_smoke.py` takes its config path and server name as POSITIONAL ARGUMENTS, so a test hands it a synthetic config in a tmpdir pointing at a stub responder and drives every protocol branch it has — no gitignored `.mcp.json`, no language server — and it is proven that way in `scripts/tests/test_mcp_smoke.py`. A checker is never excused because its CALLER cannot run in CI. WHERE THE PROOF LIVES: the checker itself needs NO `docs/guard-inventory.md` row, because that population is derived from workflow and hook call sites and a transitively-reached script is rejected as a phantom; the proof goes in an ordinary `scripts/tests/test_*.py`, which joins the population automatically and carries the row, while the manifest entry names THAT test as `guard` and the checker as `target`. That shape already exists for `mutation_harness_lib.py` and needs no change to any population. ONLY WHAT NO PROOF TEST CAN DRIVE HERMETICALLY falls back to a WITNESSED NEGATIVE CONTROL — the failure path executed once, the false-green form reproduced, both recorded in the PR, and its decay stated because nothing re-runs it. `scripts/check-local-lsp.sh` is the genuine case: it asserts developer-machine installs (a global `csharp-ls`, a `host/fxr` under `DOTNET_ROOT`, a root `node_modules`) that no runner has. STOP-AND-SUBTRACT, adopted here as this record's own threshold: when a review round's finding was CREATED by the previous round's fix and that happens TWICE CONSECUTIVELY, delete the layer generating them rather than guarding it. READING THE CONSTRUCT IS NOT EXECUTING IT: ShellCheck 0.11.0 flagged NEITHER of the two shell defects below, and each is indistinguishable from correct by reading. the verification harness is itself unproven code · a false green in the checker rather than the checked · `[ -x "$cmd" ]` is true for a DIRECTORY · perl alarm/exec exits 0 without `or die` · macOS has no `timeout` and no `gtimeout` · smoke test accepts an impostor server · a server that pre-answers a guessable id · each fix created the next round's defect · stop and subtract rather than add a layer · hermetically drivable is the test, not does-CI-run-it · a checker is not exempt because its CALLER cannot run in CI · the checker needs no inventory row — name the proof test as `guard` and the checker as `target` · a transitively-reached script is rejected as a PHANTOM · witnessed negative control for what cannot be driven · paths: `scripts/mcp_smoke.py`, `scripts/tests/test_mcp_smoke.py`, `scripts/check-local-lsp.sh`, `scripts/tests/mutation_manifest.py`, `docs/guard-inventory.md` · issues: #796, #793, #790, #785, #777, #776 THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e 'alarm shift; exec @ARGV or die "exec failed: $!\n"' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-28: `perl -e 'alarm shift; exec @ARGV' 5 /nope` exits 0; the `or die` form exits 2 and says why). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and an absent `gtimeout` exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps travel with it: `[ -x "$cmd" ]` does not establish that a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. The single in-repo call site carrying all three is `scripts/check-local-lsp.sh`.

The measurement. #793 added one small operator-run check — a smoke test that starts the csharp-lsp MCP server and lists its tools. Five independent cold-review rounds found five false greens in the verification code itself, each introduced by the fix for the previous one and each reproduced with a control before being fixed:

# The check Why it reported success without checking
1 [ -x "$cmd" ] a directory satisfies -x; [ -x /bin ] is true
2 smoke test accepts any id-matching reply an impostor server, and {"id":1}, both passed
3 perl -e 'alarm shift; exec @ARGV' wrapper perl exits 0 when exec fails → prints PASS having run nothing
4 message/line caps added to bound #3's replacement an over-long line had its suffix re-parsed as a fresh message
5 pending-id registered before send a server could pre-answer an id it could predict

Why this is a separate rule and not an application of #775. testing.guard-ships-with-mutation- proof is scoped by a population — the guard files docs/guard-inventory.md derives and grades. Defect 1 and 3 are a perl wrapper inside an if, which is not a file at all; 2, 4 and 5 live in scripts/mcp_smoke.py, which is a file but is reached only through another script and so falls under scope limit item 6. That scoping is what keeps the inventory from becoming a chore that gets rubber-stamped; what was missing is the claim that the obligation follows the verdict, not the file.

The discriminator, stated precisely because the obvious one is wrong. It is not "does CI run this": CI never fires .claude/hooks/*.sh or .husky/pre-push as hooks either, and all of them carry MUTATION rows, because a pytest proof drives them in a sandbox. The question is whether a proof test can drive the construct hermetically, and "it needs a live server" usually fails that test rather than passing it. mcp_smoke.py takes its config path and server name as positional arguments, so scripts/tests/test_mcp_smoke.py hands it a synthetic config pointing at a stub responder and drives every protocol branch — identity mismatch, a body carrying only an id, a missing tool, a pre-answered request — in under eight seconds. check-local-lsp.sh genuinely cannot be driven that way: it asserts a global csharp-ls on PATH, a host/fxr under DOTNET_ROOT, and a root node_modules, which is developer-machine state no runner has. That, and not "it is operator-run", is what puts it in the exempt branch — and the part of the construct carrying the defect history sits on the provable side of the seam.

Where the proof lives, and why the checker needs no row. docs/guard-inventory.md's population is derived from workflow and hook call sites, so a row for mcp_smoke.py is rejected outright — measured, not assumed:

AssertionError: guard-inventory.md lists ['scripts/mcp_smoke.py'], which no longer exist or are
no longer invoked by any workflow or hook. A row for a guard that does not run reads as coverage
and is not.

The row is therefore never the route. The proof goes in scripts/tests/test_*.py, which joins the population automatically and carries the row, and the manifest entry names that test as guard with the checker as target — the shape test_mutation_harness.py already uses for mutation_harness_lib.py, itself outside the population under scope limit item 7. No population derivation changes; the declared clause is re-applied every run like any other.

The residual, for the exempt branch only. A witnessed negative control recorded in a PR is weaker than a mutation re-run every suite: nothing re-executes it, and a later edit can silently undo what it proved. The alternative there is not a stronger check but a green one that runs nowhere — this repo's most expensive recurring failure (#631, #719, #751).

Stop-and-subtract. Rounds 3→4 are the clearest case: round 3 replaced the broken wrapper, round 4 found that the caps round 3 added re-parsed an over-long line's suffix, and the fix was deleting those caps — three defects with one cause. Round 5 likewise replaced lock-based ordering with unguessable random ids: strictly less code, and it removed the race rather than policing it. Two consecutive rounds of fix-creates-the-next-defect is adopted here as the threshold; the six-round text-comparison case behind release.verdict-vocabulary-shared is precedent for the shape, not its source, and it concluded in a categorical ban rather than a count.

Read-the-construct fails; execute-it works. Defect 3 is indistinguishable from correct by reading. ShellCheck 0.11.0 over reconstructed forms of the two shell defects returns only style nits (SC2292, SC2250) and an unused-variable warning — nothing about either; the remaining three are Python and outside its reach, so the honest figure is two of two, not zero of five. #776 reached the same conclusion the same week from a different subsystem, with no shared code: exec … 2>/dev/null silences the shell, and a trap's exit "$?" invents a status. The idiom behind defect 3 came from written project memory recommending the form without or die — correct advice, written down, was still a single point of failure, because the recipe had never had its own failure path executed.