Files
ersatztv/docs/decisions/records/testing/verification-code-needs-its-own-proof.md
T
timothyandClaude Opus 5 a993b90ee1 docs(796): verification code is code under test, and the proof obligation splits by where the checker runs
#793 added one small operator-run check and five cold-review rounds found five
false greens in the verification code itself, each introduced by the fix for the
previous one. All five sat outside `docs/guard-inventory.md`'s population and
always would have: a `perl` wrapper inside an `if` is not a file, and the script
holding it is invoked by no workflow or hook.

Records `testing.verification-code-needs-its-own-proof`: the obligation follows
the VERDICT rather than the file, and how it is discharged depends on where the
construct runs. A checker CI can execute keeps the existing contract (an
inventory row, and at MUTATION a declared clause re-applied every run). A checker
CI cannot execute — operator-run, needing gitignored config or a live server —
gets neither, and ships a witnessed negative control recorded in its PR instead.

Extending the manifest to the second class was considered and rejected because it
would reproduce the defect it was meant to close: `scripts/mcp_smoke.py` needs
`.mcp.json` (gitignored) and a cold-built language server, so a row for it would
be a declared mutation that never executes — the exact false green
`testing.mutation-claims-are-executed` exists to abolish. The residual (a recorded
control decays; nothing re-runs it) is stated in the record rather than papered over.

Also folds in the stop-and-subtract trigger (two consecutive rounds of
fix-creates-the-next-defect means delete the layer, not guard it) and the macOS
timeout recipe, whose wrong form was the recommended one.

Every runtime claim in the record was executed rather than read, per its own rule:
`perl -e 'alarm shift; exec @ARGV' 5 /nope` exits 0 (the false green, reproduced)
while the `or die` form exits 2; `[ -x /bin ]` is true; neither `timeout` nor
`gtimeout` is present; `.mcp.json` is ignored at `.gitignore:12`.

Refs: #796
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 19:55:31 +02:00

9.1 KiB
Raw Blame History

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
testing.verification-code-needs-its-own-proof 2026-08-28 — verification code is code under test: a harness, wrapper or checker ships a witnessed failure path, and where CI cannot run it that proof is a recorded negative control rather than a manifest row (#796) active 2026-08-28 none none VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files that happen to sit in the guard population or are named like guards. A verification construct is not trusted because it is small, because it is new, or because it is the thing doing the checking; every one of the five defects measured below was in code nobody had counted as a guard. HOW THE OBLIGATION IS DISCHARGED DEPENDS ON WHERE THE CONSTRUCT RUNS, and that split is this record's decision rather than a restatement of #775. (1) A checker CI CAN execute keeps the existing contract unchanged: a row in `docs/guard-inventory.md`, and at `MUTATION` a declared clause in `scripts/tests/mutation_manifest.py` re-applied every run. (2) A checker CI CANNOT execute — operator-run, requiring gitignored config or a live server; `scripts/check-local-lsp.sh` and `scripts/mcp_smoke.py` are today's examples, both already outside the guard population and classified `N/A` in `docs/remote-state-inventory.md` — gets NEITHER a guard-inventory row NOR a manifest entry. It ships a WITNESSED NEGATIVE CONTROL: the failure path is EXECUTED once, the false-green form is reproduced, and both are recorded in the PR that introduces or changes the construct. THE REASON FOR REFUSING THE EXTENSION IS THE POINT — a manifest row for a checker the harness cannot run would be a declared mutation that never executes, which is exactly the false green `testing.mutation-claims-are-executed` exists to abolish, so extending the manifest would reproduce the defect the extension was meant to close. STOP-AND-SUBTRACT: when a review round's finding was CREATED by the previous round's fix and that happens TWICE CONSECUTIVELY, stop adding guards to the construct and DELETE the layer generating them — this is the general form of the withdraw-after-N-rounds threshold the string-predicate rule already carries. READING THE CONSTRUCT IS NOT EXECUTING IT: for shell and process-level constructs the failure path must actually be run, because shellcheck caught 0 of the 5 defects here and every one of them is indistinguishable from correct by reading. the verification harness is itself unproven code · a false green in the checker rather than the checked · `[ -x "$cmd" ]` is true for a DIRECTORY · perl alarm/exec exits 0 without `or die` · macOS has no `timeout` and no `gtimeout` · smoke test accepts an impostor server · each fix created the next round's defect · stop and subtract rather than add a layer · an operator-run checker gets NO guard-inventory row and NO manifest entry · a declared mutation that never executes is a false green · witnessed negative control recorded in the PR · shellcheck caught 0 of 5 · paths: `scripts/check-local-lsp.sh`, `scripts/mcp_smoke.py`, `scripts/tests/mutation_manifest.py`, `docs/guard-inventory.md`, `docs/remote-state-inventory.md` · issues: #796, #793, #790, #785, #777, #776 THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e 'alarm shift; exec @ARGV or die "exec failed: $!\n"' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-14: `perl -e 'alarm shift; exec @ARGV' 5 /nope` exits 0; with `or die` it dies loudly). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and a `gtimeout` that is absent exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps that travel with it: `[ -x "$cmd" ]` does not establish a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. The single in-repo call site carrying all three is `scripts/check-local-lsp.sh`.

The measurement. #793 added one small operator-run check — a smoke test that starts the csharp-lsp MCP server and lists its tools. Five independent cold-review rounds found five false greens in the verification code itself, and each was introduced by the fix for the previous one:

# The check Why it reported success without checking
1 [ -x "$cmd" ] a directory satisfies -x; [ -x /bin ] is true
2 smoke test accepts any id-matching reply an impostor server, and {"id":1}, both passed
3 perl -e 'alarm shift; exec @ARGV' wrapper perl exits 0 when exec fails → prints PASS having run nothing
4 message/line caps added to bound #3's replacement an over-long line had its suffix re-parsed as a fresh message
5 pending-id registered before send a server could pre-answer an id it could predict

None was hypothetical: each was reproduced with a control before being fixed.

Why this is a separate rule and not an application of #775. testing.guard-ships-with-mutation- proof is scoped by a population — the guard files docs/guard-inventory.md derives and grades. All five defects above sat outside that population and always would have: a perl wrapper inside an if is not a file, and the script containing it is invoked by no workflow or hook. The scoping is not a bug in the inventory; it is what keeps the inventory from becoming a chore that gets rubber-stamped. What was missing is the claim that the obligation follows the verdict rather than the file, which is what this record adds.

Why the manifest is not extended, stated as the decision it is. The obvious move — bring operator-run checkers into mutation_manifest.py — was considered and rejected. The harness applies every declared clause to an isolated copy of the repository on each run and requires the row's named test to go red. scripts/mcp_smoke.py cannot participate: driving it needs .mcp.json, which is gitignored, and a cold-built language server that takes 30–60s to come up. A row for it would therefore be a declared mutation that never executes — a MUTATION grade that is only a claim, which is the precise shape testing.mutation-claims-are-executed was written to abolish. Extending the manifest would have reproduced, inside the fix, the defect the fix was for. That pattern has now happened often enough here to be worth naming rather than rediscovering.

The residual, stated rather than papered over: a witnessed negative control recorded in a PR is weaker than a mutation re-run every suite. It decays — nothing re-executes it, and a later edit can silently undo what it proved. It is chosen because the alternative is not a stronger check but a green one that runs nowhere, and a check that runs nowhere has been this repo's most expensive recurring failure (#631, #719, #751). Where a checker later becomes CI-executable, it moves to branch (1) and acquires a real row.

Stop-and-subtract, and the evidence for the threshold. Rounds 3→4 above are the clearest case: round 3 replaced the broken wrapper, round 4 found that the caps round 3 added re-parsed an over-long line's suffix, and the fix was deleting those caps — three defects with one cause. Round 5 likewise replaced lock-based ordering with unguessable random ids: strictly less code, and it removed the race rather than policing it. Two consecutive rounds of fix-creates-the-next-defect is the signal that the layer, not the clause, is wrong. Compare fix-the-boundary-not-the-site; the string-predicate rule already carries a withdraw threshold, and this is its general form.

Read-the-construct fails; execute-it works. Defect 3 is indistinguishable from correct by reading, and shellcheck flagged none of the five. #776 reached the same conclusion the same week from a different subsystem — exec … 2>/dev/null silences the shell, and a trap's exit "$?" invents a status — with no shared code or reviewers between them. Two independent measurements of one claim: for shell and process-level constructs, static reading and linting do not substitute for running the failure path.

Applied to itself. The perl … exec idiom that produced defect 3 came from written project memory that recommended the form without or die. Correct advice, written down, was still a single point of failure — the recipe had never had its own failure path executed. That is why the corrected idiom lives in this record's mechanics rather than only at the call site: the call site is right today, and was the only in-repo copy at the time of writing, but a buried-in-one-script recipe is not discoverable by the next session that needs it.