#793 added one small operator-run check and five cold-review rounds found five false greens in the verification code itself, each introduced by the fix for the previous one. All five sat outside `docs/guard-inventory.md`'s population and always would have: a `perl` wrapper inside an `if` is not a file, and the script holding it is invoked by no workflow or hook. Records `testing.verification-code-needs-its-own-proof`: the obligation follows the VERDICT rather than the file, and how it is discharged depends on where the construct runs. A checker CI can execute keeps the existing contract (an inventory row, and at MUTATION a declared clause re-applied every run). A checker CI cannot execute — operator-run, needing gitignored config or a live server — gets neither, and ships a witnessed negative control recorded in its PR instead. Extending the manifest to the second class was considered and rejected because it would reproduce the defect it was meant to close: `scripts/mcp_smoke.py` needs `.mcp.json` (gitignored) and a cold-built language server, so a row for it would be a declared mutation that never executes — the exact false green `testing.mutation-claims-are-executed` exists to abolish. The residual (a recorded control decays; nothing re-runs it) is stated in the record rather than papered over. Also folds in the stop-and-subtract trigger (two consecutive rounds of fix-creates-the-next-defect means delete the layer, not guard it) and the macOS timeout recipe, whose wrong form was the recommended one. Every runtime claim in the record was executed rather than read, per its own rule: `perl -e 'alarm shift; exec @ARGV' 5 /nope` exits 0 (the false green, reproduced) while the `or die` form exits 2; `[ -x /bin ]` is true; neither `timeout` nor `gtimeout` is present; `.mcp.json` is ignored at `.gitignore:12`. Refs: #796 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
9.1 KiB
key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
| key | title | status | since | supersedes | superseded-by | rule | signals | mechanics |
|---|---|---|---|---|---|---|---|---|
| testing.verification-code-needs-its-own-proof | 2026-08-28 — verification code is code under test: a harness, wrapper or checker ships a witnessed failure path, and where CI cannot run it that proof is a recorded negative control rather than a manifest row (#796) | active | 2026-08-28 | none | none | VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files that happen to sit in the guard population or are named like guards. A verification construct is not trusted because it is small, because it is new, or because it is the thing doing the checking; every one of the five defects measured below was in code nobody had counted as a guard. HOW THE OBLIGATION IS DISCHARGED DEPENDS ON WHERE THE CONSTRUCT RUNS, and that split is this record's decision rather than a restatement of #775. (1) A checker CI CAN execute keeps the existing contract unchanged: a row in `docs/guard-inventory.md`, and at `MUTATION` a declared clause in `scripts/tests/mutation_manifest.py` re-applied every run. (2) A checker CI CANNOT execute — operator-run, requiring gitignored config or a live server; `scripts/check-local-lsp.sh` and `scripts/mcp_smoke.py` are today's examples, both already outside the guard population and classified `N/A` in `docs/remote-state-inventory.md` — gets NEITHER a guard-inventory row NOR a manifest entry. It ships a WITNESSED NEGATIVE CONTROL: the failure path is EXECUTED once, the false-green form is reproduced, and both are recorded in the PR that introduces or changes the construct. THE REASON FOR REFUSING THE EXTENSION IS THE POINT — a manifest row for a checker the harness cannot run would be a declared mutation that never executes, which is exactly the false green `testing.mutation-claims-are-executed` exists to abolish, so extending the manifest would reproduce the defect the extension was meant to close. STOP-AND-SUBTRACT: when a review round's finding was CREATED by the previous round's fix and that happens TWICE CONSECUTIVELY, stop adding guards to the construct and DELETE the layer generating them — this is the general form of the withdraw-after-N-rounds threshold the string-predicate rule already carries. READING THE CONSTRUCT IS NOT EXECUTING IT: for shell and process-level constructs the failure path must actually be run, because shellcheck caught 0 of the 5 defects here and every one of them is indistinguishable from correct by reading. | the verification harness is itself unproven code · a false green in the checker rather than the checked · `[ -x "$cmd" ]` is true for a DIRECTORY · perl alarm/exec exits 0 without `or die` · macOS has no `timeout` and no `gtimeout` · smoke test accepts an impostor server · each fix created the next round's defect · stop and subtract rather than add a layer · an operator-run checker gets NO guard-inventory row and NO manifest entry · a declared mutation that never executes is a false green · witnessed negative control recorded in the PR · shellcheck caught 0 of 5 · paths: `scripts/check-local-lsp.sh`, `scripts/mcp_smoke.py`, `scripts/tests/mutation_manifest.py`, `docs/guard-inventory.md`, `docs/remote-state-inventory.md` · issues: #796, #793, #790, #785, #777, #776 | THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e 'alarm shift; exec @ARGV or die "exec failed: $!\n"' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-14: `perl -e 'alarm shift; exec @ARGV' 5 /nope` exits 0; with `or die` it dies loudly). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and a `gtimeout` that is absent exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps that travel with it: `[ -x "$cmd" ]` does not establish a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. The single in-repo call site carrying all three is `scripts/check-local-lsp.sh`. |
The measurement. #793 added one small operator-run check — a smoke test that starts the
csharp-lsp MCP server and lists its tools. Five independent cold-review rounds found five false
greens in the verification code itself, and each was introduced by the fix for the previous one:
| # | The check | Why it reported success without checking |
|---|---|---|
| 1 | [ -x "$cmd" ] |
a directory satisfies -x; [ -x /bin ] is true |
| 2 | smoke test accepts any id-matching reply | an impostor server, and {"id":1}, both passed |
| 3 | perl -e 'alarm shift; exec @ARGV' wrapper |
perl exits 0 when exec fails → prints PASS having run nothing |
| 4 | message/line caps added to bound #3's replacement | an over-long line had its suffix re-parsed as a fresh message |
| 5 | pending-id registered before send | a server could pre-answer an id it could predict |
None was hypothetical: each was reproduced with a control before being fixed.
Why this is a separate rule and not an application of #775. testing.guard-ships-with-mutation- proof is scoped by a population — the guard files docs/guard-inventory.md derives and grades. All
five defects above sat outside that population and always would have: a perl wrapper inside an
if is not a file, and the script containing it is invoked by no workflow or hook. The scoping is
not a bug in the inventory; it is what keeps the inventory from becoming a chore that gets
rubber-stamped. What was missing is the claim that the obligation follows the verdict rather than
the file, which is what this record adds.
Why the manifest is not extended, stated as the decision it is. The obvious move — bring
operator-run checkers into mutation_manifest.py — was considered and rejected. The harness applies
every declared clause to an isolated copy of the repository on each run and requires the row's named
test to go red. scripts/mcp_smoke.py cannot participate: driving it needs .mcp.json, which is
gitignored, and a cold-built language server that takes 30–60s to come up. A row for it would
therefore be a declared mutation that never executes — a MUTATION grade that is only a claim,
which is the precise shape testing.mutation-claims-are-executed was written to abolish. Extending
the manifest would have reproduced, inside the fix, the defect the fix was for. That pattern has now
happened often enough here to be worth naming rather than rediscovering.
The residual, stated rather than papered over: a witnessed negative control recorded in a PR is weaker than a mutation re-run every suite. It decays — nothing re-executes it, and a later edit can silently undo what it proved. It is chosen because the alternative is not a stronger check but a green one that runs nowhere, and a check that runs nowhere has been this repo's most expensive recurring failure (#631, #719, #751). Where a checker later becomes CI-executable, it moves to branch (1) and acquires a real row.
Stop-and-subtract, and the evidence for the threshold. Rounds 3→4 above are the clearest case:
round 3 replaced the broken wrapper, round 4 found that the caps round 3 added re-parsed an
over-long line's suffix, and the fix was deleting those caps — three defects with one cause.
Round 5 likewise replaced lock-based ordering with unguessable random ids: strictly less code, and
it removed the race rather than policing it. Two consecutive rounds of fix-creates-the-next-defect
is the signal that the layer, not the clause, is wrong. Compare fix-the-boundary-not-the-site;
the string-predicate rule already carries a withdraw threshold, and this is its general form.
Read-the-construct fails; execute-it works. Defect 3 is indistinguishable from correct by
reading, and shellcheck flagged none of the five. #776 reached the same conclusion the same week
from a different subsystem — exec … 2>/dev/null silences the shell, and a trap's exit "$?"
invents a status — with no shared code or reviewers between them. Two independent measurements of
one claim: for shell and process-level constructs, static reading and linting do not substitute for
running the failure path.
Applied to itself. The perl … exec idiom that produced defect 3 came from written project
memory that recommended the form without or die. Correct advice, written down, was still a single
point of failure — the recipe had never had its own failure path executed. That is why the corrected
idiom lives in this record's mechanics rather than only at the call site: the call site is right
today, and was the only in-repo copy at the time of writing, but a buried-in-one-script recipe is
not discoverable by the next session that needs it.