Files
ersatztv/docs/decisions/records/testing/mutation-claims-are-executed.md
T
timothy 29d7a06e65
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 27s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m53s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 5m57s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 5m56s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / Build & push image (amd64) (push) Successful in 4m20s
fix(806): guard populations over FILES derive from the git index, not a filesystem walk (#818)
`testing.guard-derives-population-from-source` (#774) was silent on the commonest
population in our own guards — files in a directory — and every one answered with a
filesystem walk. A walk is not authoritative: it reports build output, generated
shims and editor droppings, and differs per machine. #778 measured the cost by
getting the same population wrong three times in one PR.

CONVERTED (a completeness claim over tracked files): `test_guard_inventory.py`,
`test_hook_fire_log.py`, `test_ci_image_pin_population.py` (which also gained
`*.yaml`), `test_remote_state_inventory.py` (folded onto the shared derivation), and
`test_pr_changed_files.py` (not on the issue's list — found by sweeping the whole
repo).

ASSESSED AND RECORDED, not silently skipped: `_repo_copy` takes its file list from
the index for hermeticity though it makes no completeness claim;
`test_ci_dropped_step_guard.py` has no filesystem population at all; the decisions
corpus is recorded as unexamined rather than cleared; and the SPA page-size guard is
deferred to #819 with its obstacle documented. This is not "replace every glob".

`scripts/tests/tracked_files.py` is the single derivation.
`test_guard_populations_derive_from_git.py` proves it in two measured complements:
exhaustive removal catches a hardcoded `.exists()` admit and memoisation; the call
log catches an append-only source that yields nothing on this machine — #778's
shape — which removal cannot see because it has nothing to remove.

Twelve rounds of independent cold review, alternating model families in isolated
worktrees. The production derivations were confirmed sound every round; every
blocking finding after the first was in the proofs or in prose claims about them.
Counts over growing populations were removed rather than corrected, after three
drifted.

fixes #806

Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
2026-08-22 16:07:58 +00:00

11 KiB

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
testing.mutation-claims-are-executed 2026-08-22 — a MUTATION grade is EXECUTED every run, from a declared clause, or it is not that grade (#790) active 2026-08-22 none none A `MUTATION` row in `docs/guard-inventory.md` is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`, and `scripts/tests/test_mutation_harness.py` applies that mutation to an isolated copy of the repository on every run and requires the row's OWN named test to go red. The manifest and the MUTATION rows are compared for SET EQUALITY in both directions, so a row cannot claim the grade without a mutation and a mutation cannot outlive the grade it justifies. EXIT STATUS IS NOT THE VERDICT: each entry also declares the DIAGNOSTIC its red must carry, matched against pytest's exception output alone, because pytest reports a crashing test exactly as it reports a detecting one and a red for an unrelated reason is evidence about nothing. WHERE THE GUARD IS ITSELF A TEST, `target` may differ from `guard` and the exact-once check applies to the declared TARGET. Two shapes are admissible and the choice is not free. Where the guard's assertion IS the check — a completeness comparison against a Markdown inventory — the mutation goes into the guarded ARTIFACT, per `testing.guard-ships-with-mutation-proof`'s checker-guard exception, because mutating such a checker's own POPULATION demonstrates a false POSITIVE while proving nothing about the detection the row claims. Where the guard is a test module wrapping a separately mutable DETECTOR or helper, the clause may be in that detector, since disarming it is a real clause disarm and the module's own assertion is what notices. THE MUTATION IS DECLARED, NEVER INFERRED: a harness that guessed which clause of a 90-line hook is the guard would manufacture the confident-but-empty coverage this exists to prevent, which is why `testing.guard-ships-with-mutation-proof` rejected a generic runner. Where a proof test already names its clause in source, the manifest reuses THAT string, so a retarget in either place is caught by the other. COARSENESS IS RECORDED, NOT HIDDEN: each entry is graded `CLAUSE` or `DETECTOR`, and a `DETECTOR` entry — one whose detector accumulates faults from independent arms, so disarming any single arm leaves its proof test green — must CARRY the finer mutation that survived, which is re-run every time and required to keep surviving. Guards that are not graded `MUTATION` each carry a STATED reason in that same manifest, keyed on the guard and compared for SET EQUALITY against the inventory's `GUARD` rows in both directions — so a new guard cannot arrive without someone writing what a proof would need, and a reason cannot outlive the row it is about. Keying the reason on the row's GRADE instead is tautological (a new guard inherits one and nobody looks at it) and a pinned COUNT moves only on net change; both were tried and are rejected. The sandbox is a real git repository built from `git ls-files` with working-tree content, never a filesystem walk. mutation harness · declared clause not inferred · MUTATION row is executed · set equality manifest vs inventory · CLAUSE vs DETECTOR granularity · surviving finer mutation is re-run · pytest exit code 1 is the only red that counts · every other exit status is rejected · positive control before any mutation · sandbox from git ls-files · paths: `scripts/tests/test_mutation_harness.py`, `scripts/tests/mutation_manifest.py`, `scripts/tests/mutation_harness_lib.py`, `docs/guard-inventory.md` · issues: #790, #775, #774, #778, #806 The harness builds ONE sandbox per session — tracked files only, from `git ls-files -s`, in an environment with every `GIT_*` variable stripped so an exported `GIT_DIR`/`GIT_COMMON_DIR`/`GIT_CONFIG_*` cannot point its `init`/`add`/`commit` at the real repository — and resets it between mutations with `git reset --hard` TO A BASELINE COMMIT RECORDED OUTSIDE THE REPOSITORY, plus `git clean -qffdx`. Both halves matter: bare `git reset --hard` resets to whatever HEAD is, so a proof test that COMMITS inside the sandbox moves HEAD onto a commit carrying the mutant and every later reset restores it faithfully; and a baseline held as a REF inside the sandbox is one more thing a proof can move. `core.hooksPath`, `commit.gpgsign` and `core.worktree` are pinned on every git invocation for the same reason. The `-ff` removes a nested repository a proof may have left. It runs the full set of named proof tests UNMUTATED first, inside the fixture rather than as a separate test so ordering is a dependency and not a convention. Only pytest exit code 1 counts as red; every other status is rejected. That is what closes the two ways a proof ref goes stale, and they are measured rather than assumed: with an explicit `file.py::function` node id a missing file and a missing function both exit 4, while 5 needs a collection that succeeded and selected nothing. The sandbox skips the `ErsatzTV-macOS` gitlink and recreates the `.claude/skills/jellyfin` symlink as a symlink. Cost: ~14s, against a ~4min `script-tests` suite.

What #775 left open, in its own words: "nothing checks that a row claiming MUTATION is telling the truth." It said so deliberately — the bookkeeping half (population, set equality, proof refs resolving to a real def) was mechanised, and the judgement half was left with review. Both cold reviews of #774/#775 came back to the same place: the witnessing had happened once, by hand, and hand-run evidence decays the moment someone edits the guard. Two false MUTATION grades were caught by reading in the first draft, and a third round would likely have found more.

Why a generic runner was rejected, and what changed. #775 rejected a mutation runner for shell hooks on the grounds that it "would have to know which clause of a 90-line hook is the guard". That objection is correct and is not answered by better inference — it is answered by not inferring. The clause is declared, one entry per row, and for most rows the declaration already existed: the proof tests name their own clauses in source (= "efbbbf" ]; then, UNSET_CLAUSE, if [ "$RC" -eq 0 ]; then). Reusing those strings rather than inventing parallel ones is what makes the two halves catch each other's drift.

What a green harness proves, and what it does not. It proves the proof ref names a test that still exists and still collects; that the declared clause still occurs, exactly once, in the entry's declared target, which is not always the guard's own file; and — via the positive control — that every named proof test is GREEN on the unmutated sandbox, so "the mutation was noticed" cannot be confused with "the test was already red". The proofs that carry a disarm of their own run it during that control; the two that are plain production set-equality checks have none, and the harness supplies theirs. It does not prove the declared clause is the only thing the guard hangs on. For three entries (measured 2026-08-22: the BOM guard, build_decisions_catalog.py and prove-fix.sh) the redness arrives through the proof test's own "this clause has moved, RETARGET it" assertion rather than through changed behaviour. That is the intended reading rather than a hole: those tests perform their behavioural disarm themselves on every green run; what they could not do was notice their own clause reference going stale. Every other entry reddens behaviourally, and each one's declared diagnostic says which.

The DETECTOR grade is the honest half, and it exists because measuring found a case. #775's record concedes that neutering pin_population_faults wholesale is "coarser than disarming one clause at a time — coarse enough that a single surviving clause would not be noticed". Running the finer mutations settled which guards actually admit clause-level proof: all but one do, including pin_population_faults, where disarming the against-the-registry comparison alone reddens the proof because a job that loses its container: block leaves the two derived sets equal. One does not. instrumentation_faults accumulates from four independent arms and a stripped hook trips three of them at once, so disarming any single arm leaves the proof test green — measured, by running if not _SOURCES_SINK.search(text): → if False: and watching it pass. That surviving mutation is carried as data and re-run on every suite: if it ever starts reddening, the guard has become clause-provable and the entry must be regraded. A grade that cannot decay quietly is the point.

Three shapes that were tried and rejected, because each looks like verification and is not. Emptying a checker's own population reddens its proof with an IndexError — a crash, not a detection, and any breakage of the derivation satisfies it equally. Shrinking the guard-inventory checker's hook glob reddens its proof by making every real row report as PHANTOM: a false positive, which says nothing about the missing-row detection the row claims. And planting a phantom row in docs/remote-state-inventory.md reddens a proof test that builds a phantom of its own and asserts the difference is exactly one entry — so the declared diagnostic appears through fixture contamination, in the opposite direction from the one the row claims. All three are replaced by artifact-level mutations that reach the guard's production assertion — a deleted inventory row, a renamed one — and expect is what makes the difference visible instead of arguable.

Stating the other guards, and the two cheaper shapes that do not work. Keying the reason on the row's grade is tautological: a new guard graded NONE inherits a sentence automatically and nobody ever looks at that particular guard. Pinning the count of undeclared guards is no better, because it moves only on net change — one guard arriving as another is promoted leaves it unchanged. What ships is a per-guard mapping compared for set equality against the inventory's GUARD rows. That is a hand-maintained table, which is normally the duplication family to avoid (#774's wrong summary counts, #788's verdict vocabulary) — but bidirectional set equality is what makes docs/guard-inventory.md itself safe, and it is what turns "state the ones you cannot declare" into something a new guard cannot slip past. Set equality ALONE is not sufficient and the gap is demonstrable: every comparison here reduces rows through a set or a dict, so a DUPLICATED row is invisible to all of them and a table contradicting itself reports full coverage. Duplicate rejection is a separate, explicit check, and copying the set-equality design without it recreates the hole.

Two words that are not interchangeable. A NONE row means the row nominates no proof; it does not mean the guard is untested. scripts/ci-prove-ban-detects.sh is graded NONE and is driven end to end by test_ci_release_path_scan_job.py, and eight hooks are driven through their deciding path by test_hook_fire_log.py — whose matrix asserts instrumentation TRANSPARENCY, that the wrapped and unwrapped runs agree, never that the decision is right or that a particular clause produced it. A statement that reads "no test drives this" over those guards sends the next session's proof work in the wrong direction, so each entry says which of the two is actually missing.