From 1fc24e8cf6eead5423255277a992d2e1a001c20c Mon Sep 17 00:00:00 2001 From: Timothy Date: Fri, 4 Sep 2026 23:18:31 +0200 Subject: [PATCH 01/18] fix(881): a mutation-outcome claim is executed wherever it is written, bound to the sentence that makes it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `testing.mutation-claims-are-executed` was the right rule scoped to its first site: `mutation_manifest.py` declared itself "one per `MUTATION`-graded row of `docs/guard-inventory.md`", so the same claim written in a code comment, a test docstring or a decision record was outside it by construction. That is where all four of ersatztv#812's consecutive review-round defects lived. Extend the rule in place rather than adding a sibling record: a sibling would recreate the exact shape (a rule per site class, with the next site class outside both) that #773, #784 and #743 each are. The subject is unchanged; only the population widens. Mechanism: `CLAIMS` in `scripts/tests/mutation_manifest.py`, keyed on the PROSE. Each entry carries the tracked `site` and the verbatim `quote`, checked every run, so a reworded sentence reports as a retarget instead of drifting from the entry that justifies it — this is proposal 2 (a quotation of another file is a claim about that file) adopted where the referent is declared. Each entry also declares RED or GREEN and is executed in the existing sandbox. GREEN is new: 47 of the 128 candidate lines the corpus grep returns at efadbec29 assert that a mutation is NOT noticed, and no `MUTATION` row can express that, so the rule was unsatisfiable for them. The green direction is read by two separate clauses (exited 0, and something actually passed) so neither can mask the other, and each carries its own disarm proof. Proposal 3 (never anchor prose to a state your own commit moves) is rejected as a DETECTOR and kept as a phrasing rule: measured 2026-09-04, the only plausible pattern set for it matched 16 lines across the scanned corpus and every one was legitimate rationale prose. The seed set falsified a shipped claim on its first run: `post-review-verdict.sh` asserted that disarming its array-TYPE read-back test left the suite green. It does not — jq refuses to iterate a `null` `.statuses` and the script dies with the parse message, reddening `test_a_readback_whose_statuses_array_is_NULL_is_refused`. Comment corrected, entry graded RED. fixes #881 Decisions-Edit: yes Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015QqCpYFsKgnAnx6jVwrKiV --- docs/decisions/README.md | 2 +- .../testing/mutation-claims-are-executed.md | 57 +++++- docs/guard-inventory.md | 10 +- scripts/post-review-verdict.sh | 8 +- scripts/tests/mutation_harness_lib.py | 172 +++++++++++++--- scripts/tests/mutation_manifest.py | 104 +++++++++- scripts/tests/test_mutation_harness.py | 184 +++++++++++++++++- scripts/tests/tracked_files.py | 11 ++ 8 files changed, 509 insertions(+), 39 deletions(-) diff --git a/docs/decisions/README.md b/docs/decisions/README.md index b024b729a..fe7a4e82d 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -222,7 +222,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera | `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) | | `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin