docs(796): sweep every remaining quantity and novelty claim, and delete the ones that rot

Self-run sweep of the whole change for hand-maintained counts, universals and
novelty claims — the class round 3 identified as this change's defect generator.
It found four more, including one introduced by round 3's own fix.

- **"the SECOND row of that shape"** (guard-inventory item 6) was FALSE. Round 3
  correctly said my "vocabulary stretch" framing overstated novelty; my correction
  overstated precision in the other direction. Measured by joining the inventory's
  self-citing rows against the manifest: EIGHT rows are a self-citing GUARD whose
  declared `target` is a different file, and `test_review_verdict_vocabulary.py` ->
  `scripts/check-review-verdict.sh` is a closer analogue than the one I named. The
  ordinal is deleted rather than corrected — two examples, no count.

- **"the single in-repo site carrying the perl recipe AND the check-the-exit-code
  discipline"** asserted a uniqueness that cannot be checked; plenty of scripts
  check exit codes. Narrowed to the half that is verifiable and verified: the perl
  recipe appears in exactly one place.

- **"Every case below asserts a REFUSAL"** was false of the block it introduces —
  the positive control sits below that comment and asserts a pass.

- **"the ONLY thing refusing a server that answers before it is asked"** was
  unqualified in both the test message and the manifest `why`, while round 2 had
  measured that the `tools/list` stage is held by two mechanisms. Both now say AT
  THE `initialize` STAGE, and say why the pending-registration cannot help there:
  that id is already in flight when the pre-answer arrives.

Claims swept and confirmed TRUE rather than changed: "every one of the five defects
was in code nobody had counted as a guard"; "FOUR of them carry MUTATION rows";
"NEITHER of the two shell defects"; "NONE of those controls survived as a test"
(nothing but this file and the manifest references `mcp_smoke.py`); "replacing
`id_tools` alone leaves every case here green".

Verification: 1104 passed, 2 skipped; ruff format+check clean over all 47 tracked
Python files; decisions-validate OK.

Refs: #796
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-28 21:13:12 +02:00
co-authored by Claude Opus 5
parent 42bb3f146e
commit 13001ee2dd
4 changed files with 17 additions and 13 deletions
@@ -7,7 +7,7 @@ supersedes: none
superseded-by: none
rule: 'VERIFICATION CODE IS CODE UNDER TEST. The obligation in `testing.guard-ships-with-mutation-proof` attaches to whatever emits a PASS/FAIL somebody acts on — a smoke test, a timeout wrapper, an operator-run checker, the harness that applies the mutations — and NOT only to files sitting in the guard population or named like guards; every one of the five defects measured below was in code nobody had counted as a guard. THE DISCRIMINATOR IS WHETHER A PROOF TEST CAN DRIVE THE CONSTRUCT HERMETICALLY, NOT WHETHER CI RUNS IT, and the difference is load-bearing rather than pedantic: `.claude/hooks/*.sh` and `.husky/pre-push` are operator-run and CI never fires them AS HOOKS, yet FOUR of them carry `MUTATION` rows — `posttooluse-worktree-marker.sh`, `pretooluse-bom-guard.sh`, `pretooluse-worktree-guard.sh` and `.husky/pre-push` — because a pytest proof drives them in a sandbox. A does-CI-run-it test would sort those four already-proven guards into the exempt branch. (Most of the remaining hooks are graded `NONE`; that is unproven debt, not evidence for the other discriminator.) "IT NEEDS A REAL SERVER" IS AN EXEMPTION CLAIM AND IT IS USUALLY FALSE: `scripts/mcp_smoke.py` takes its config path and server name as POSITIONAL ARGUMENTS, so a test hands it a synthetic config in a tmpdir pointing at a stub responder and drives it with no gitignored `.mcp.json` and no language server, which is how it is proven in `scripts/tests/test_mcp_smoke.py`. That coverage is a SUBSET and no count of it is kept here: the file''s own docstring enumerates the cases it drives and names what it does not, so there is one list rather than a number to keep in step with it. A checker is never excused because its CALLER cannot run in CI. WHERE THE PROOF LIVES: the checker itself needs NO `docs/guard-inventory.md` row, because that population is derived from workflow and hook call sites and a transitively-reached script is rejected as a phantom; the proof goes in an ordinary `scripts/tests/test_*.py`, which joins the population automatically and carries the row, while the manifest entry names THAT test as `guard` and the checker as `target`. That shape already exists for `mutation_harness_lib.py` and needs no change to any population. ONLY WHAT NO PROOF TEST CAN DRIVE HERMETICALLY falls back to a WITNESSED NEGATIVE CONTROL — the failure path executed once, the false-green form reproduced, both recorded in the PR, and its decay stated because nothing re-runs it. `scripts/check-local-lsp.sh` is the genuine case: it asserts developer-machine installs (a global `csharp-ls`, a `host/fxr` under `DOTNET_ROOT`, a root `node_modules`) that no runner has. STOP-AND-SUBTRACT, adopted here as this record''s own threshold: when a review round''s finding was CREATED by the previous round''s fix and that happens TWICE CONSECUTIVELY, delete the layer generating them rather than guarding it. READING THE CONSTRUCT IS NOT EXECUTING IT: ShellCheck 0.11.0 flagged NEITHER of the two shell defects below, and each is indistinguishable from correct by reading.'
signals: 'the verification harness is itself unproven code · a false green in the checker rather than the checked · `[ -x "$cmd" ]` is true for a DIRECTORY · perl alarm/exec exits 0 without `or die` · macOS has no `timeout` and no `gtimeout` · smoke test accepts an impostor server · a server that pre-answers a guessable id · each fix created the next round''s defect · stop and subtract rather than add a layer · hermetically drivable is the test, not does-CI-run-it · a checker is not exempt because its CALLER cannot run in CI · the checker needs no inventory row — name the proof test as `guard` and the checker as `target` · a transitively-reached script is rejected as a PHANTOM · witnessed negative control for what cannot be driven · paths: `scripts/mcp_smoke.py`, `scripts/tests/test_mcp_smoke.py`, `scripts/check-local-lsp.sh`, `scripts/tests/mutation_manifest.py`, `docs/guard-inventory.md` · issues: #796, #793, #790, #785, #777, #776'
mechanics: 'THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e ''alarm shift; exec @ARGV or die "exec failed: $!\n"'' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-28: `perl -e ''alarm shift; exec @ARGV'' 5 /nope` exits 0; the `or die` form exits 2 and says why). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and an absent `gtimeout` exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps travel with it: `[ -x "$cmd" ]` does not establish that a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. `scripts/check-local-lsp.sh` is the single in-repo site carrying the perl recipe and the check-the-exit-code discipline; the `-x` trap appears there only in a comment recording what it deliberately does not do.'
mechanics: 'THE macOS TIMEOUT RECIPE, recorded here because the WRONG form was the recommended one and a session that followed it verbatim produced a false green (#793): `perl -e ''alarm shift; exec @ARGV or die "exec failed: $!\n"'' <secs> <cmd...>`. The `or die` is load-bearing — without it perl exits 0 when `exec` fails, so a missing or misspelt binary reports SUCCESS having run nothing (measured 2026-08-28: `perl -e ''alarm shift; exec @ARGV'' 5 /nope` exits 0; the `or die` form exits 2 and says why). macOS has neither `timeout` (GNU coreutils) nor `gtimeout` on this fleet, and an absent `gtimeout` exits 127 with empty output, which reads as a clean pass to any caller grepping stdout for a failure string. Two traps travel with it: `[ -x "$cmd" ]` does not establish that a command is runnable, because a directory satisfies `-x` (`[ -x /bin ]` is true); and whatever wrapper is used, its EXIT CODE must be checked rather than its stdout. The perl recipe appears in exactly one place in-repo, `scripts/check-local-lsp.sh`; the `-x` trap appears there only in a comment recording what it deliberately does not do.'
---
**The measurement.** #793 added one small operator-run check — a smoke test that starts the
+6 -5
View File
@@ -75,11 +75,12 @@ inventory shipped with were sitting in the gap:
derived from the call sites, so one is rejected as a phantom — but the manifest can still name
the checker, as `target`, with its proof test as `guard`. `scripts/mcp_smoke.py` is the worked
example: no row of its own, proven by `scripts/tests/test_mcp_smoke.py`, which carries the row.
That row is `GUARD` citing itself, which is the SECOND row of that shape rather than a new one:
`test_mutation_harness.py` already self-cites while its declared `target` is another script
(`mutation_harness_lib.py`, outside this population under item 7). Worth noting only because the
self-citation carve-out in `test_every_proof_ref_points_at_a_row_marked_PROOF` is worded for a
checker guarding a repo invariant, and both of these guard a script instead.
That row is `GUARD` citing itself while its declared `target` is a different file — an ESTABLISHED
shape here, not a new one, and deliberately not counted: `test_review_verdict_vocabulary.py`
targets `scripts/check-review-verdict.sh` and `test_mutation_harness.py` targets
`mutation_harness_lib.py`. Worth noting only because the self-citation carve-out in
`test_every_proof_ref_points_at_a_row_marked_PROOF` is worded for a checker guarding a repo
invariant, whereas these guard a script.
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
population and hold no rows. They are not guards (they assert nothing on their own), but the
+4 -3
View File
@@ -322,9 +322,10 @@ MUTATIONS: tuple[Mutation, ...] = (
"`scripts/mcp_smoke.py` is a checker reached only transitively, through "
"`scripts/check-local-lsp.sh`, so it holds no inventory row of its own and the clause has to "
"be disarmed in it directly (`testing.verification-code-needs-its-own-proof`). The clause is "
"the unguessable request id, which is the ONLY thing refusing a server that answers before "
"it is asked — a lock cannot reject a frame emitted before the request, which is why #793 "
"round 5 replaced the lock rather than tightening it. Disarmed, the stub's pre-answer is "
"the unguessable request id, which AT THE `initialize` STAGE is the only thing refusing a "
"server that answers before it is asked: the pending-registration cannot help there, because "
"that id is already in flight when the pre-answer arrives, which is why #793 round 5 replaced "
"the lock rather than tightening it. Disarmed, the stub's pre-answer is "
"accepted at `initialize` and the run dies one stage later at `tools/list`, so the proof "
"asserts the STAGE (rc 9 and the initialize diagnostic) rather than mere failure: the mutant "
"still fails, and a test checking only that would stay green over a real vulnerability. THE "
+6 -4
View File
@@ -122,8 +122,9 @@ def _run(tmp_path: Path, mode: str, *extra: str, timeout: int = 5):
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY AS A DEPENDENCY, NOT A CONVENTION. Every case below asserts a REFUSAL, and a harness
# that could not make the checker SUCCEED would report all of them green while proving nothing. A
# ANTI-VACUITY AS A DEPENDENCY, NOT A CONVENTION. Every case below except the positive control itself
# asserts a REFUSAL, and a harness that could not make the checker SUCCEED would report all of those
# green while proving nothing. A
# sibling test placed first does not establish that: `-k`, a node-id selection, or the mutation
# harness running one test by id all skip it. A fixture the rejection tests DEPEND on cannot be
# skipped that way.
@@ -192,8 +193,9 @@ def test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUE
)
assert rc == 9 and "no 'initialize' response" in out, (
f"a pre-answered id was ACCEPTED: the checker got past `initialize` against a server that "
f"replied before it was asked (rc={rc}). Unguessable request ids are the only thing closing "
f"that race — a lock cannot reject a frame emitted before the request.\n{out}"
f"replied before it was asked (rc={rc}). At THIS stage an unguessable id is the only thing "
f"closing that race — the pending-registration cannot help, because the id is already in "
f"flight when the pre-answer arrives.\n{out}"
)