diff --git a/docs/decisions/records/testing/mutation-claims-are-executed.md b/docs/decisions/records/testing/mutation-claims-are-executed.md index e75e3a687..b9964afff 100644 --- a/docs/decisions/records/testing/mutation-claims-are-executed.md +++ b/docs/decisions/records/testing/mutation-claims-are-executed.md @@ -7,7 +7,7 @@ supersedes: none superseded-by: none rule: 'A `MUTATION` row in `docs/guard-inventory.md` is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`, and `scripts/tests/test_mutation_harness.py` applies that mutation to an isolated copy of the repository on every run and requires the row''s OWN named test to go red. The manifest and the MUTATION rows are compared for SET EQUALITY in both directions, so a row cannot claim the grade without a mutation and a mutation cannot outlive the grade it justifies. EXIT STATUS IS NOT THE VERDICT: each entry also declares the DIAGNOSTIC its red must carry, matched against pytest''s exception output alone, because pytest reports a crashing test exactly as it reports a detecting one and a red for an unrelated reason is evidence about nothing. WHERE THE GUARD IS ITSELF A TEST, `target` may differ from `guard` and the exact-once check applies to the declared TARGET. Two shapes are admissible and the choice is not free. Where the guard''s assertion IS the check — a completeness comparison against a Markdown inventory — the mutation goes into the guarded ARTIFACT, per `testing.guard-ships-with-mutation-proof`''s checker-guard exception, because mutating such a checker''s own POPULATION demonstrates a false POSITIVE while proving nothing about the detection the row claims. Where the guard is a test module wrapping a separately mutable DETECTOR or helper, the clause may be in that detector, since disarming it is a real clause disarm and the module''s own assertion is what notices. THE MUTATION IS DECLARED, NEVER INFERRED: a harness that guessed which clause of a 90-line hook is the guard would manufacture the confident-but-empty coverage this exists to prevent, which is why `testing.guard-ships-with-mutation-proof` rejected a generic runner. Where a proof test already names its clause in source, the manifest reuses THAT string, so a retarget in either place is caught by the other. COARSENESS IS RECORDED, NOT HIDDEN: each entry is graded `CLAUSE` or `DETECTOR`, and a `DETECTOR` entry — one whose detector accumulates faults from independent arms, so disarming any single arm leaves its proof test green — must CARRY the finer mutation that survived, which is re-run every time and required to keep surviving. Guards that are not graded `MUTATION` each carry a STATED reason in that same manifest, keyed on the guard and compared for SET EQUALITY against the inventory''s `GUARD` rows in both directions — so a new guard cannot arrive without someone writing what a proof would need, and a reason cannot outlive the row it is about. Keying the reason on the row''s GRADE instead is tautological (a new guard inherits one and nobody looks at it) and a pinned COUNT moves only on net change; both were tried and are rejected. The sandbox is a real git repository built from `git ls-files` with working-tree content, never a filesystem walk. THE POPULATION IS EVERY DECLARED MUTATION-OUTCOME CLAIM, NOT ONLY THE INVENTORY''S ROWS (ersatztv#881): a sentence asserting that a specific mutation reddens, or does not redden, a named test is the same kind of claim wherever it is written — a code comment, a test docstring, a decision record — and is either a `CLAIMS` entry in `scripts/tests/mutation_manifest.py` that executes every run, or it is not written. A claim entry is keyed on the PROSE rather than on a guard: it carries the tracked `site` and the verbatim `quote`, both checked every run, so a reworded sentence reports as a RETARGET instead of drifting away from the entry that justifies it. THE QUOTE SPANS THE MUTATION AND THE OUTCOME IT ASSERTS: the outcome half is the checkable part, so a quote stopping at the mutation binds the uncontroversial words and leaves the assertion, and the test it names, outside the binding entirely. It declares `RED` or `GREEN`, and GREEN is the half no `MUTATION` row can express — a claim that a mutation is NOT noticed is read by THREE separate clauses, because each alone certifies a run that proves nothing: the run exited 0, something actually PASSED, and the named proof was shown to REACH the mutated clause. THE TWO DIRECTIONS ARE NOT SYMMETRIC, WHICH IS WHY THE THIRD CLAUSE EXISTS ON ONLY ONE OF THEM. A RED claim checks its own relevance for free — a proof that never executes the mutated file stays green and is refused in the same words as a clause that has stopped being load-bearing — while a GREEN claim gets precisely the green it asserts from a proof that reads a different file entirely, so exit status and a passing test TOGETHER still certify a run that never executed the clause. A GREEN entry therefore also declares a SECOND mutation of the SAME clause, required to REDDEN the same proof and carrying its own declared diagnostic, executed through the same red-reading path the inventory rows use; only once that passes is the green evidence about this clause at all. It is worth what the `DETECTOR` survivor was worth: re-run, and required to KEEP surviving. THE REFERENT IS DECLARED, NEVER INFERRED, exactly as the clause is: a checker hunting for sentences that merely SOUND like mutation claims would be a string predicate over prose, which this repo refuses to make load-bearing, so what is enforced is a claim that named its own referent, and the direction that cannot be mechanised — prose that declares nothing — is stated rather than papered over. TWO NEIGHBOURING PROPOSALS ARE SETTLED HERE. A QUOTATION of another file is a claim about that file: adopted, and the `quote` field IS the check, wherever the referent is declared; a general quotation-checker is rejected for the same reason inference is, since a quotation whose referent is unnamed has nothing to resolve. ANCHORING PROSE TO A STATE THE COMMIT ITSELF MOVES (''the last edit to that path'', ''the only X that''): rejected as a detector and kept as a phrasing rule. The rejection is measured, and the scan is PINNED IN THE BODY rather than summarised as a figure here, for the same reason the population scan is — a count quoted without the predicate that produced it is the defect this record is about. What it shows is that the shape is indistinguishable BY PATTERN from ordinary rationale (''X is the only thing that can Y''), which this rule carves out, so a detector would fire mostly on prose it must not flag, and an advisory check that does that stops being read. Anchor to a date or a sha under `docs.no-session-narrative`''s dated-measurement clause; inside this rule''s own population the anchor is the manifest entry, which is re-executed and therefore cannot go stale.' signals: 'mutation harness · declared clause not inferred · MUTATION row is executed · set equality manifest vs inventory · CLAUSE vs DETECTOR granularity · surviving finer mutation is re-run · pytest exit code 1 is the only red that counts · every other exit status is rejected · positive control before any mutation · sandbox from git ls-files · paths: `scripts/tests/test_mutation_harness.py`, `scripts/tests/mutation_manifest.py`, `scripts/tests/mutation_harness_lib.py`, `docs/guard-inventory.md` · prose claim bound to its declared site and verbatim quote · GREEN direction executes a claim that a mutation is NOT noticed · a GREEN entry declares a reach mutation proving its proof touches the clause · quotation of another file is a claim about that file · state anchors rejected as a detector, kept as a phrasing rule · issues: #790, #775, #774, #778, #806, #812, #881' -mechanics: 'The harness builds ONE sandbox per session — tracked files only, from `git ls-files -s`, in an environment with every `GIT_*` variable stripped so an exported `GIT_DIR`/`GIT_COMMON_DIR`/`GIT_CONFIG_*` cannot point its `init`/`add`/`commit` at the real repository — and resets it between mutations with `git reset --hard` TO A BASELINE COMMIT RECORDED OUTSIDE THE REPOSITORY, plus `git clean -qffdx`. Both halves matter: bare `git reset --hard` resets to whatever HEAD is, so a proof test that COMMITS inside the sandbox moves HEAD onto a commit carrying the mutant and every later reset restores it faithfully; and a baseline held as a REF inside the sandbox is one more thing a proof can move. `core.hooksPath`, `commit.gpgsign` and `core.worktree` are pinned on every git invocation for the same reason. The `-ff` removes a nested repository a proof may have left. It runs the full set of named proof tests UNMUTATED first, inside the fixture rather than as a separate test so ordering is a dependency and not a convention. Only pytest exit code 1 counts as red; every other status is rejected. That is what closes the two ways a proof ref goes stale, and they are measured rather than assumed: with an explicit `file.py::function` node id a missing file and a missing function both exit 4, while 5 needs a collection that succeeded and selected nothing. The sandbox skips the `ErsatzTV-macOS` gitlink and recreates the `.claude/skills/jellyfin` symlink as a symlink. `CLAIMS` reuses the same sandbox, the same reset and the same clause application (`_mutate_and_run`, split out so the two directions cannot come to disagree about what a red is worth), and its proof refs join the positive control for the reason the row proofs do, which holds in BOTH directions: over a proof that was ALREADY red, a RED claim is satisfied by redness its own mutation did not cause, and a GREEN claim is REFUSED for a reason that has nothing to do with its mutation — reported as a wrong sentence when what is wrong is the proof. What that control cannot do is see ONE ref that collects nothing — its assertions are over the aggregate of every ref, so the other refs supply the word `passed` — nor ONE ref that collects plenty and touches nothing the claim mutates, which is the larger of the two, since such a proof is green before and after and a GREEN claim would read as verified. Both discriminations are per-claim, in the GREEN direction''s own vacuity and relevance gates. THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE, and that is the whole of what the order buys: it is the only one of the three that costs a second full run of the proof, under the reach mutation with a sandbox reset between, so an entry the status or the vacuity gate has already refused never pays for it. What keeps the three gates worth three is that each carries its own disarm proof, not the sequence they run in. The sandbox is reset between a claim''s two proof runs, for the reason it is reset between mutations. Each claim''s `site` is tested for membership of the git INDEX, not `is_file`, so a claim pointing at an untracked file fails here instead of passing on one machine. Cost, re-measured 2026-09-05 WITH the relevance gate, on an M-series Mac with `-p no:randomly`, whole file, three baseline/branch pairs run back to back: the `CLAIMS` half adds 31.7-43.6%, which is the span of the three pairs and not a rounding of it — 43.6% (46.8s -> 67.2s), 34.1% (50.1s -> 67.2s) and 31.7% (48.9s -> 64.4s), under varying concurrent load. The figure this replaces, 12.7-16.6% measured 2026-09-04, was taken before the GREEN direction had a relevance gate and is retired rather than adjusted: the gate runs a GREEN entry''s proof a SECOND time under the reach mutation, with a sandbox reset between, and its three disarm proofs each drive further inner pytest runs. A measurement is a property of the population that produced it, so growing the population invalidates it — re-measure, never scale. Quote the PERCENTAGE rather than the seconds - the absolute figures moved 30% with machine load in one session, and the earlier ~14s here covered fewer entries. The ~4min `script-tests` suite it was measured against is PRE-#870 and no longer holds: that change made `count_pr_mutations` read to its 20-page cap, and `test_pr_changed_files.py` went from 202s to 474s locally (286 -> 294 tests, same machine, `-p no:randomly`, 528383cf3 vs the #870 branch), so the job is roughly double. Re-measure on a runner before quoting a job figure.' +mechanics: 'The harness builds ONE sandbox per session — tracked files only, from `git ls-files -s`, in an environment with every `GIT_*` variable stripped so an exported `GIT_DIR`/`GIT_COMMON_DIR`/`GIT_CONFIG_*` cannot point its `init`/`add`/`commit` at the real repository — and resets it between mutations with `git reset --hard` TO A BASELINE COMMIT RECORDED OUTSIDE THE REPOSITORY, plus `git clean -qffdx`. Both halves matter: bare `git reset --hard` resets to whatever HEAD is, so a proof test that COMMITS inside the sandbox moves HEAD onto a commit carrying the mutant and every later reset restores it faithfully; and a baseline held as a REF inside the sandbox is one more thing a proof can move. `core.hooksPath`, `commit.gpgsign` and `core.worktree` are pinned on every git invocation for the same reason. The `-ff` removes a nested repository a proof may have left. It runs the full set of named proof tests UNMUTATED first, inside the fixture rather than as a separate test so ordering is a dependency and not a convention. Only pytest exit code 1 counts as red; every other status is rejected. That is what closes the two ways a proof ref goes stale, and they are measured rather than assumed: with an explicit `file.py::function` node id a missing file and a missing function both exit 4, while 5 needs a collection that succeeded and selected nothing. The sandbox skips the `ErsatzTV-macOS` gitlink and recreates the `.claude/skills/jellyfin` symlink as a symlink. `CLAIMS` reuses the same sandbox, the same reset and the same clause application (`_mutate_and_run`, split out so the two directions cannot come to disagree about what a red is worth), and its proof refs join the positive control for the reason the row proofs do, which holds in BOTH directions: over a proof that was ALREADY red, a RED claim is satisfied by redness its own mutation did not cause, and a GREEN claim is REFUSED for a reason that has nothing to do with its mutation — reported as a wrong sentence when what is wrong is the proof. What that control cannot do is see ONE ref that collects nothing — its assertions are over the aggregate of every ref, so the other refs supply the word `passed` — nor ONE ref that collects plenty and touches nothing the claim mutates, which is the larger of the two, since such a proof is green before and after and a GREEN claim would read as verified. Both discriminations are per-claim, in the GREEN direction''s own vacuity and relevance gates. THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE: it is the only one of the three that costs a second full run of the proof, under the reach mutation with a sandbox reset between, so an entry the status or the vacuity gate has already refused never pays for it. What keeps the three gates worth three is that each carries its own disarm proof, rather than the position it sits in. The sandbox is reset between a claim''s two proof runs, for the reason it is reset between mutations. Each claim''s `site` is tested for membership of the git INDEX, not `is_file`, so a claim pointing at an untracked file fails here instead of passing on one machine. Cost, re-measured 2026-09-05 WITH the relevance gate, on an M-series Mac with `-p no:randomly`, whole file, three baseline/branch pairs run back to back: the `CLAIMS` half adds 31.7-43.6%, which is the span of the three pairs and not a rounding of it — 43.6% (46.8s -> 67.2s), 34.1% (50.1s -> 67.2s) and 31.7% (48.9s -> 64.4s), under varying concurrent load. The figure this replaces, 12.7-16.6% measured 2026-09-04, was taken before the GREEN direction had a relevance gate and is retired rather than adjusted: the gate runs a GREEN entry''s proof a SECOND time under the reach mutation, with a sandbox reset between, and its three disarm proofs each drive further inner pytest runs. A measurement is a property of the population that produced it, so growing the population invalidates it — re-measure, never scale. Quote the PERCENTAGE rather than the seconds - the absolute figures moved 30% with machine load in one session, and the earlier ~14s here covered fewer entries. The ~4min `script-tests` suite it was measured against is PRE-#870 and no longer holds: that change made `count_pr_mutations` read to its 20-page cap, and `test_pr_changed_files.py` went from 202s to 474s locally (286 -> 294 tests, same machine, `-p no:randomly`, 528383cf3 vs the #870 branch), so the job is roughly double. Re-measure on a runner before quoting a job figure.' --- **What #775 left open, in its own words: "nothing checks that a row claiming `MUTATION` is telling diff --git a/scripts/tests/mutation_harness_lib.py b/scripts/tests/mutation_harness_lib.py index c40358256..8501e596f 100644 --- a/scripts/tests/mutation_harness_lib.py +++ b/scripts/tests/mutation_harness_lib.py @@ -430,13 +430,13 @@ def verify_claim(sandbox: Path, claim: Claim) -> Verdict: disarmed on its own and seen to matter; a single clause covering all three would be a guard nothing can witness failing. - THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE, and that is the whole of what the - order buys: it is the only one of the three that costs a second full run of the proof — under the - reach mutation, with a sandbox reset in between — so an entry the status or the vacuity gate has - already refused never pays for it. What keeps the three worth three is that each carries its own - disarm proof, not the sequence they are written in. The sandbox is reset between the two runs - because the first proof run can dirty the tree, and a second result that is a function of the - first's is what `reset_sandbox` exists to prevent. + THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE: it is the only one of the three + that costs a second full run of the proof — under the reach mutation, with a sandbox reset in + between — so an entry the status or the vacuity gate has already refused never pays for it. What + keeps the three worth three is that each carries its own disarm proof, rather than the position + it sits in. The sandbox is reset between the two runs because the first proof run can dirty the + tree, and a second result that is a function of the first's is what `reset_sandbox` exists to + prevent. """ if claim.outcome == Claim.RED: return verify_mutation(