Extend mutation-claim enforcement to prose (#881) #914

Merged
timothy merged 18 commits from 881-mutation-claim-prose-scope into main 2026-09-05 08:40:34 +02:00
12 changed files with 843 additions and 48 deletions
+1 -1
View File
@@ -28,7 +28,7 @@ doc below, or that changes which sections a task signal points to.**
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite) — plus `…/verification-code-needs-its-own-proof.md`, which extends the same obligation BEYOND guards to the harness, wrapper or checker doing the checking, and says where its proof lives when the checker holds no row |
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite — and so does a PROSE claim that some mutation reddens, or does not redden, a named test, wherever it is written: it is a `CLAIMS` entry in the same manifest, bound to its site and verbatim quote, or it is not written) — plus `…/verification-code-needs-its-own-proof.md`, which extends the same obligation BEYOND guards to the harness, wrapper or checker doing the checking, and says where its proof lives when the checker holds no row |
| Adding or bounding a consequential numeric config field (an FFmpeg profile tunable, a pipeline knob) | `docs/api-conventions.md` §3d — reject out of range with a 422 naming the bound and its consequence, never accept-then-rewrite; validate against the constants the renderer reads, keep the render-time clamp for pre-existing rows, and let an UNCHANGED legacy value through on update. Then `api.ffmpeg-profile-numeric-bounds` |
| Testing a surface gated by config / an env var / a credential | `docs/decisions/records/testing/deny-path-at-production-config-value.md` — cover the setting absent, at its production value, and each opt-out, and assert the DENY branch |
| Touching a full-replace write path or a hand-built request object | `docs/decisions/records/testing/full-replace-asserts-field-list.md` — derive the field list from the DTO and assert set equality; reconcile by id where child state exists. In the SPA the same rule is enforced by the type system: `docs/spa-conventions.md` §4b — build the body as `Complete<T>`, annotating BOTH the wrapper parameter and every construction site |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+12 -1
View File
@@ -261,7 +261,7 @@ recorded as unexamined rather than as cleared.
| `scripts/tests/test_merge_consent_head_change.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_merge_consent_required_check.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_migration_equivalence.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names, or when a DECLARED PROSE CLAIM in `CLAIMS` stops producing the outcome its sentence asserts (either direction) | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
| `scripts/tests/test_post_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_prepush_unsets_git_env.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_pr_changed_files.py` | the `script-tests` job | PROOF | NONE | — |
@@ -369,6 +369,17 @@ set equality in both directions, so the grade and the mutation cannot drift apar
graded `MUTATION` without declaring its clause fails the suite. Full contract and its limits:
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
**That file also executes a SECOND population, which is not this table and must not be read off it.**
`CLAIMS` in the same manifest binds declared PROSE claims — a sentence in a code comment, a test
docstring or a decision record asserting that some mutation reddens, or does not redden, a named
test — to the file and the sentence that make them (ersatztv#881). Those entries carry a `site` and
a verbatim `quote` instead of a guard, so a rewritten sentence reports as a retarget; and they may
declare a GREEN outcome, which no row here can — one that also declares a REACH mutation of the same
clause required to redden the same proof, because a green from a proof that never touches the mutated
file is indistinguishable from the green the sentence claims. A guard's row says nothing about
whether a claim about that guard has been declared, and the two populations are derived differently
on purpose.
One thing that column still does not say: it does not say the declared clause is the ONLY thing the
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
+6 -2
View File
@@ -186,8 +186,12 @@ def added_lines(diff: str):
p = raw[4:]
# `/dev/null` on the new side means the file was DELETED. Stated honestly: this arm
# is DEFENSIVE, not load-bearing — a deletion contributes no `+` lines, so nothing is
# yielded for it either way, and removing this arm reddens no test. It is kept because
# `path` should never name a file the added lines do not belong to. A
# yielded for it either way. Measured rather than asserted, and re-measured on every
# run: `test_check_doc_narrative.py` stays green with this arm removed, a declared
# claim in `scripts/tests/mutation_manifest.py` (ersatztv#881). The named file is the
# scope that is actually executed — a claim over every test in the repository would
# be wider than anything re-taking it. The arm is kept because `path` should never
# name a file the added lines do not belong to. A
# `--diff-filter=d` on the git call was removed rather than kept beside it: a second
# mechanism nobody can witness failing is how a duplicate guard hides its twin.
# Git terminates the `+++` filename with a TAB when the path contains a space, and a
+6 -2
View File
@@ -313,8 +313,12 @@ status_readback=$(api_get "repos/$owner/$repo/commits/$sha/status?limit=100") \
# Three outcomes are kept apart below, each with its own message, because they call for different
# actions: a body whose `.statuses` is not an array at
# all, a well-formed body that does not carry our context, and a body carrying MORE THAN ONE row for
# it. Collapsing them also makes the array-TYPE test unwitnessable — the absent-entry check catches
# every case on its own, so disarming the type test leaves the suite green.
# it. Keeping them apart is also what makes the array-TYPE test witnessable, and that is EXECUTED
# rather than asserted: disarming the type test feeds a `null` `.statuses` to `map(select(...))`,
# jq refuses to iterate it, and the script dies with the PARSE message instead of the shape one — so
# `test_a_readback_whose_statuses_array_is_NULL_is_refused` reddens. It carries no date because it
# is not a witnessing: it is a declared claim in `CLAIMS` (`scripts/tests/mutation_manifest.py`),
# re-taken on every run (ersatztv#881).
readback_matches=$(printf '%s' "$status_readback" | jq -c --arg c "$STATUS_CONTEXT" \
'if (.statuses | type) == "array" then (.statuses | map(select(.context == $c))) else null end') \
|| die "the '$STATUS_CONTEXT' status was posted on $short, but the read-back body could not be parsed — no comment was written."
+207 -23
View File
@@ -111,6 +111,69 @@ class Mutation:
return f"scripts/tests/{self.proof}"
@dataclass(frozen=True)
class Claim:
"""One PROSE claim about a mutation's outcome, bound to the sentence that makes it.
A `Mutation` above is keyed on a GUARD, because its population is the `MUTATION`-graded rows of
`docs/guard-inventory.md`. A claim is keyed on the PROSE instead: `site` names the tracked file
whose text makes the claim and `quote` is that text, verbatim. Both halves are checked every run
— the quote must occur exactly once in the site — so a reworded sentence reports as a retarget
rather than drifting away from the entry that justifies it. That binding is the thing #812's four
defects were missing; each was a sentence with nothing tying it to what it asserted.
`outcome` is `RED` or `GREEN`, and the GREEN direction is why this is not just another `Mutation`.
A `MUTATION` row always claims a red, so the harness only ever had to recognise one. Prose does
not, and the corpus carries the other direction too: `.gitea/workflows/review-verdict.yml`,
`scripts/check-doc-narrative.py` and `scripts/tests/hook_fire_isolation.py` each name a mutation —
an arm disarmed on its own, an arm removed, a dead branch deleted — and assert that nothing
notices it. How each of those sentences words its outcome is deliberately not repeated here: a
restatement is a second copy of an outcome, free to drift from the site while the entry that
executes it stays green, which is the drift a `Claim` exists to close. What they share is the
shape — a claim that a mutation is NOT noticed — and for that shape the rule "declare it or do
not write it" is unsatisfiable unless the harness can execute a negative. A GREEN entry therefore
carries NO `expect`: there is no failing run to read a diagnostic out of, and the three things
that could make a green vacuous — a run that errored, a run in which nothing passed, and a proof
that never reaches the mutated clause at all — are checked in `verify_claim` instead.
THE THIRD OF THOSE IS WHAT MAKES A GREEN READABLE, and it needs its own declaration. A red is
self-checking: a proof that ignores the mutated file stays green and `verify_mutation` refuses it,
naming the clause as not load-bearing. A green has no such property — an unrelated proof produces
exactly the green the sentence claims, so exit status and "something passed" together still
certify a run that never executed the clause. `reach_replacement` closes that: a SECOND mutation
of the SAME clause, declared to REDDEN the same proof with the diagnostic in `reach_expect`, and
executed through `verify_mutation` so the red is read exactly as a `MUTATION` row's is. Passing it
means the proof reaches this clause and its value changes what the proof observes; only then does
the declared green say anything. RED entries leave both fields empty — there is nothing there for
them to add.
What a GREEN claim is worth is exactly what the retired `DETECTOR` survivor was worth: it is
re-run and required to KEEP surviving, so the day the mutation starts being noticed the entry
goes red and the sentence has to be rewritten. `testing.mutation-claims-are-executed` records the
direction that matters — a claim that can no longer decay is a tautology reading like a proof.
"""
RED = "RED"
GREEN = "GREEN"
site: str
quote: str
target: str
clause: str
replacement: str
proof: str
outcome: str
expect: str
why: str
reach_replacement: str = ""
reach_expect: str = ""
@property
def node_id(self) -> str:
"""A claim may name a whole proof FILE, where the prose says "leaves the suite green"."""
return f"scripts/tests/{self.proof}"
@dataclass(frozen=True)
class Verdict:
ok: bool
@@ -271,35 +334,55 @@ _PYTEST_RED_MEANINGS = {
}
def _mutate_and_run(
sandbox: Path, target_path: str, clause: str, replacement: str, node_id: str
) -> tuple[Verdict | None, subprocess.CompletedProcess | None]:
"""Apply one declared clause mutation and run one proof ref under it.
Split out of `verify_mutation` so the RED and GREEN directions share ONE application path rather
than two copies of it. Everything that can go wrong BEFORE the proof runs — an absent target, a
clause that has moved or acquired a second site, a replacement identical to the clause — is the
same question in both directions and is answered here; each direction then reads the result its
own way. A `Verdict` comes back only when the mutation could not be applied at all.
"""
target = sandbox / target_path
if not target.is_file():
return Verdict(False, f"the mutation target {target_path} does not exist in the sandbox"), None
original = target.read_text(encoding="utf-8")
occurrences = original.count(clause)
if occurrences != 1:
return (
Verdict(
False,
f"the declared clause occurs {occurrences} times in {target_path}, not once. "
"RETARGET it rather than loosening the match — a clause that has moved, or that now "
"matches a second site, means the recorded proof no longer points at what it claims to.",
),
None,
)
mutated = original.replace(clause, replacement, 1)
if mutated == original:
return Verdict(False, "the replacement is identical to the clause, so nothing was mutated"), None
target.write_text(mutated, encoding="utf-8")
try:
return None, run_pytest(sandbox, [node_id])
finally:
target.write_text(original, encoding="utf-8")
def verify_mutation(sandbox: Path, mutation: Mutation) -> Verdict:
"""Apply one declared mutation in `sandbox` and require its named test to go red.
The sandbox is left as it was found; callers still `reset_sandbox` between mutations because a
driven guard can dirty the tree in ways this function does not know about.
"""
target = sandbox / mutation.target
if not target.is_file():
return Verdict(False, f"the mutation target {mutation.target} does not exist in the sandbox")
original = target.read_text(encoding="utf-8")
occurrences = original.count(mutation.clause)
if occurrences != 1:
return Verdict(
False,
f"the declared clause occurs {occurrences} times in {mutation.target}, not once. "
"RETARGET it rather than loosening the match — a clause that has moved, or that now "
"matches a second site, means the recorded proof no longer points at what it claims to.",
)
mutated = original.replace(mutation.clause, mutation.replacement, 1)
if mutated == original:
return Verdict(False, "the replacement is identical to the clause, so nothing was mutated")
target.write_text(mutated, encoding="utf-8")
try:
result = run_pytest(sandbox, [mutation.node_id])
finally:
target.write_text(original, encoding="utf-8")
refusal, result = _mutate_and_run(sandbox, mutation.target, mutation.clause, mutation.replacement, mutation.node_id)
if refusal is not None:
return refusal
assert result is not None # `_mutate_and_run` returns exactly one of the two
output = result.stdout + result.stderr
if result.returncode != 1:
@@ -327,3 +410,104 @@ def verify_mutation(sandbox: Path, mutation: Mutation) -> Verdict:
f"{output[-3000:]}",
)
return Verdict(True, "the named test went red under the declared mutation, with the declared diagnostic")
def verify_claim(sandbox: Path, claim: Claim) -> Verdict:
"""Execute one declared prose claim: apply its mutation and require its DECLARED direction.
RED delegates to `verify_mutation`, deliberately and not for brevity: a claim that a mutation IS
noticed is the same assertion a `MUTATION` row makes, and giving it a second implementation is
how the two would come to disagree about what a red is worth (exit status, diagnostic, an
already-red proof).
GREEN is the direction that has no equivalent above, and it is checked by THREE separate clauses
because each alone is satisfiable by a run that proves nothing. Exit status alone accepts a run
in which every test SKIPPED — pytest exits 0 for that — "something passed" alone accepts a file
that also failed, and both together still accept a proof that never touches the mutated file:
an unrelated test run produces the same green as the claimed one. That last one is answered by
the entry's declared `reach_replacement`, a second mutation of the SAME clause required to REDDEN
the same proof. They are kept apart rather than combined into one condition so each can be
disarmed on its own and seen to matter; a single clause covering all three would be a guard
nothing can witness failing.
THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE: it is the only one of the three
that costs a second full run of the proof — under the reach mutation, with a sandbox reset in
between — so an entry the status or the vacuity gate has already refused never pays for it. What
keeps the three worth three is that each carries its own disarm proof, rather than the position
it sits in. The sandbox is reset between the two runs because the first proof run can dirty the
tree, and a second result that is a function of the first's is what `reset_sandbox` exists to
prevent.
"""
if claim.outcome == Claim.RED:
return verify_mutation(
sandbox,
Mutation(
guard=claim.site,
target=claim.target,
clause=claim.clause,
replacement=claim.replacement,
proof=claim.proof,
granularity=Mutation.CLAUSE,
expect=claim.expect,
why=claim.why,
),
)
if claim.outcome != Claim.GREEN:
return Verdict(
False,
f"unknown outcome {claim.outcome!r} on the claim at {claim.site} — declare RED (the "
"mutation is noticed) or GREEN (it is not). An outcome this function does not recognise "
"must not read as a verified claim.",
)
refusal, result = _mutate_and_run(sandbox, claim.target, claim.clause, claim.replacement, claim.node_id)
if refusal is not None:
return refusal
assert result is not None # `_mutate_and_run` returns exactly one of the two
output = result.stdout + result.stderr
if result.returncode != 0:
return Verdict(
False,
f"the prose at {claim.site} says this mutation leaves {claim.proof} GREEN, but the run "
f"exited {result.returncode}. If the mutation is now noticed that is good news about the "
"code and bad news about the sentence: rewrite the sentence, and regrade this entry RED "
f"with the diagnostic it produces.\n--- inner pytest output ---\n{output[-3000:]}",
)
if "passed" not in output:
return Verdict(
False,
f"the run of {claim.proof} exited 0 but NOTHING PASSED — every test was skipped, "
"deselected or absent. A green nothing ran is not evidence that the mutation went "
f"unnoticed.\n--- inner pytest output ---\n{output[-3000:]}",
)
reset_sandbox(sandbox)
reach = verify_mutation(
sandbox,
Mutation(
guard=claim.site,
target=claim.target,
clause=claim.clause,
replacement=claim.reach_replacement,
proof=claim.proof,
granularity=Mutation.CLAUSE,
expect=claim.reach_expect,
why=claim.why,
),
)
if not reach.ok:
return Verdict(
False,
f"{claim.proof} stayed green under the declared mutation, but the DECLARED REACH "
"mutation of the same clause did not redden it, so that green is not evidence about "
"this clause: a proof that never executes it produces exactly the same result. Either "
"the proof no longer reaches the clause — in which case the sentence at "
f"{claim.site} is about something this proof cannot see — or the reach mutation itself "
f"needs re-declaring.\n--- the reach verdict ---\n{reach.reason}",
)
return Verdict(
True,
"the named proof stayed green under the declared mutation, with tests actually run, and "
"reddened under the declared reach mutation of the same clause",
)
+142 -6
View File
@@ -1,4 +1,7 @@
"""The DECLARED clause mutations, one per `MUTATION`-graded row of `docs/guard-inventory.md`.
"""The DECLARED clause mutations: one per `MUTATION`-graded row of `docs/guard-inventory.md`
in `MUTATIONS`, plus one per declared PROSE claim about a mutation's outcome in `CLAIMS`
(ersatztv#881). The two populations are derived differently and are kept apart for that reason;
the `CLAIMS` section below states its own.
Data only. The machinery that applies these is `mutation_harness_lib.py`; the checks that keep this
file honest are `test_mutation_harness.py`.
@@ -22,10 +25,12 @@ and why; no count is kept here, because a count of the entries below is a second
from __future__ import annotations
from scripts.tests.mutation_harness_lib import Mutation
from scripts.tests.mutation_harness_lib import Claim, Mutation
CLAUSE = Mutation.CLAUSE
DETECTOR = Mutation.DETECTOR
RED = Claim.RED
GREEN = Claim.GREEN
MUTATIONS: tuple[Mutation, ...] = (
Mutation(
@@ -317,10 +322,20 @@ MUTATIONS: tuple[Mutation, ...] = (
"`mutation_harness_lib.py` so a clause of it can be disarmed in an isolated copy at all. The "
"clause is the DIAGNOSTIC gate — the check that a failing proof failed with the diagnostic "
"its row declares. Disarmed, a red for any unrelated reason is certified as a guard doing "
"its job, which is the shape that made two rows in this very file measure nothing. The other "
"gate, the one requiring pytest exit code 1, carries its own proof in "
"`test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_TEST`; the "
"inventory holds one ref per row, so this entry names the stronger of the two.",
"its job, which is the shape that made two rows in this very file measure nothing. THE "
"INVENTORY HOLDS ONE REF PER ROW, so this entry names the gate verified rows hang on most "
"directly, and the gates that read a proof RUN carry disarm proofs of their own: "
"`test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_TEST` for the "
"exit-status gate and, on the GREEN direction `CLAIMS` uses, the status, vacuity and "
"relevance gates. "
"So does the PRE-FLIGHT refusal that no later gate can stand in for — a replacement "
"identical to its clause mutates nothing, so the proof runs against the original tree and "
"an already-red one reddens exactly like a detection: "
"`test_MUTATION_disarming_the_IDENTICAL_REPLACEMENT_gate_CERTIFIES_A_ROW_THAT_MUTATED_NOTHING`. "
"Named rather than counted, because a count moves silently when one is added — and naming "
"is not a claim the list is CLOSED: the two remaining pre-flight refusals, an absent target "
"and a clause that does not occur exactly once, decide before any proof runs and this entry "
"says nothing about what they are worth.",
),
Mutation(
guard="scripts/ci-toolchain-image-resolves.sh",
@@ -544,3 +559,124 @@ UNDECLARED: dict[str, str] = {
"have to go into the guarded artifact — the release-path scan job in the workflow. Which "
"weakening of that job is THE defect it exists to catch has not been settled.",
}
# ------------------------------------------------------------------------------------------------
# THE PROSE CLAIMS — same rule, wider population (ersatztv#881)
# ------------------------------------------------------------------------------------------------
#
# `MUTATIONS` above covers the `MUTATION`-graded rows of `docs/guard-inventory.md` and nothing else,
# which is where ersatztv#812's four defects got in: a claim about a mutation's outcome written in a
# TEST DOCSTRING, a DECISION RECORD or a CODE COMMENT was outside the executed-claim rule by
# construction, so four consecutive review rounds each found one that was simply false. The rule was
# already the right rule; its population was its first site.
#
# So a claim entry is keyed on the PROSE, not on a guard. `site` names the tracked file that makes
# the claim and `quote` is the sentence, verbatim: the harness requires the quote to occur exactly
# once there, so the entry and the sentence cannot drift apart silently. That binding is the half
# #812's defects were missing — every one of them was an assertion with nothing tying it to its
# referent.
#
# THE QUOTE SPANS THE MUTATION AND THE OUTCOME. The mutation half of a sentence is the part that
# is uncontroversial; the checkable assertion is the outcome — `reddens <this test>`, `leaves
# the suite green`. A quote that stops before it binds the wrong half: the words that make the
# sentence a claim could then be reworded, or the test renamed in the prose, while the entry's
# binding still resolves.
#
# THE REFERENT IS DECLARED, NEVER INFERRED — the same doctrine the clause mutations above are built
# on, applied one level up. A checker that went looking for sentences that SOUND like mutation claims
# would be a string predicate over prose, which `docs/defect-shapes-773.md` §4 and
# `docs.no-session-narrative` both refuse to make load-bearing. What is checked here is a claim that
# named its own referent; what is NOT checked is a claim that named none, and that limit is stated in
# `testing.mutation-claims-are-executed` rather than papered over.
#
# `outcome` is RED or GREEN. GREEN is new and is the point: a sentence asserting that a mutation is
# NOT noticed is one no `MUTATION` row can express, so before this the rule "declare it or do not
# write it" had no way to be satisfied for that shape at all. A
# GREEN entry carries no `expect` — there is no red to read a diagnostic out of — and is worth
# exactly what the retired `DETECTOR` survivor was worth: it is required to KEEP surviving, and the
# day the mutation starts being noticed the entry reddens and the sentence has to be rewritten.
#
# A GREEN ENTRY ALSO DECLARES ITS `reach_replacement`, AND IS WORTHLESS WITHOUT IT. The two
# directions are not symmetric: a RED claim checks itself, because a proof that never executes the
# mutated file stays green and the harness refuses it in the same words it refuses a clause that has
# stopped being load-bearing. A GREEN claim has no such property — an unrelated proof produces
# exactly the green the sentence asserts, so a green certifies nothing until the proof is shown to
# reach the clause. `reach_replacement` is a SECOND mutation of the SAME clause, declared to redden
# the same proof with `reach_expect`, and it is executed through the same `verify_mutation` the rows
# above use, so its red is read exactly as theirs is rather than on exit status.
#
# No count is kept here either, for the reason stated at the top of this file.
CLAIMS: tuple[Claim, ...] = (
Claim(
site="scripts/post-review-verdict.sh",
# The quote spans the OUTCOME half and the named test, not only the mutation half: the
# checkable assertion here is `<mutation> reddens <this test>`, and a quote stopping at
# the comma would leave the words that make it a claim free to be reworded — or the test
# renamed in the prose — while the entry's binding still resolves.
quote=(
"rather than asserted: disarming the type test feeds a `null` `.statuses` to "
"`map(select(...))`,\n"
"# jq refuses to iterate it, and the script dies with the PARSE message instead of "
"the shape one — so\n"
"# `test_a_readback_whose_statuses_array_is_NULL_is_refused` reddens."
),
target="scripts/post-review-verdict.sh",
clause='if (.statuses | type) == "array" then',
replacement="if true then",
proof="test_post_review_verdict.py::test_a_readback_whose_statuses_array_is_NULL_is_refused",
outcome=RED,
expect="Cannot iterate over null (null)",
why="The sentence explains why three read-back outcomes are kept apart with separate "
"messages, and it asserts that the array-TYPE test is what keeps them apart. Disarmed, a "
"`null` `.statuses` reaches `map(select(...))`, jq refuses to iterate it and the `|| die` "
"fires with the PARSE message, so the named test fails on the SHAPE message it asserts. The "
"declared diagnostic is jq's own, not that assertion's expected text: the expected text also "
"appears in the comparison pytest prints, so matching it would certify any red inside the "
"same assertion, including one produced by an unrelated change to the script.",
),
Claim(
site="scripts/check-doc-narrative.py",
quote="`test_check_doc_narrative.py` stays green with this arm removed",
target="scripts/check-doc-narrative.py",
clause=' path = None if p == "/dev/null" else (p[2:] if p.startswith("b/") else p)',
replacement=' path = p[2:] if p.startswith("b/") else p',
proof="test_check_doc_narrative.py",
outcome=GREEN,
expect="",
# THE RELEVANCE GATE, and without it this entry certifies nothing: a proof that never
# executes this line produces exactly the green the sentence predicts. The reach mutation
# drops the `b/` stripping the same expression performs, so every `+++ b/<path>` header
# resolves to a path `is_scanned_path` rejects and the run scans NOTHING — which is what the
# declared diagnostic reads, from the ACTUAL half of the comparison rather than the expected
# one. Passing it means the proof reaches this clause and what the clause computes changes
# what the proof observes, which is the precondition the green half needs and cannot supply.
reach_replacement=" path = p",
reach_expect="scanned 0 file(s); 0 advisory warning(s)",
why="The `/dev/null` arm is declared DEFENSIVE in its own comment — a deleted file "
"contributes no `+` lines, so nothing is yielded for it either way. The comment says so and "
"then names an outcome, which is a measurement, and until this entry nothing re-took it. The "
"mutation removes exactly the arm the comment is about, leaving the `b/` stripping it is "
"wrapped around, so a red here would mean the arm has become load-bearing and the comment is "
"now wrong. The quote names the proof FILE rather than every test in the repository, because "
"the file is what the harness runs: an outcome half broader than the executed scope is the "
"same drift in miniature.",
),
Claim(
site="docs/decisions/records/docs/record-wing-parse-guard.md",
quote="deleting the single line that WIRES it into `main()` reddens a dedicated test",
target="scripts/decisions_validate.py",
clause=" wing_faults=record_wing_faults() + yaml_faults,",
replacement=" wing_faults=yaml_faults,",
proof="test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan",
outcome=RED,
expect="a wing fault must fail the validator",
why="A RED claim, and one made in a DECISION RECORD — the site class two of #812's four "
"defects lived in, and the one furthest from anything that executes. The record says the "
"wiring line's removal previously left the whole suite green while a real record vanished, "
"so the dedicated test is the only thing standing between that defect and a silent return. "
"The mutation drops `record_wing_faults()` from the call and keeps `yaml_faults`, which is "
"the deletion the sentence describes rather than a caricature of it.",
),
)
+6 -3
View File
@@ -110,9 +110,12 @@ def test_a_DELETED_doc_is_not_reported_as_added_content(repo: Path) -> None:
"""A deletion contributes no added lines.
Stated exactly: this pins the BEHAVIOUR, and it is NOT a proof of the `+++ /dev/null` arm, which
is defensive — removing that arm reddens nothing, because a deletion yields no `+` lines either
way. The script comment says the same. A docstring claiming a proof it does not have is worse
than no docstring: it is the thing a later reader trusts instead of re-checking."""
is defensive — a deletion yields no `+` lines either way. What removing that arm does to this
file is deliberately NOT restated here: it is a declared claim in
`scripts/tests/mutation_manifest.py`, re-executed on every run (ersatztv#881), and a second copy
of an outcome is free to drift from the entry that executes it. A docstring claiming a proof it
does not have is worse than no docstring: it is the thing a later reader trusts instead of
re-checking."""
(repo / "docs" / "del.md").write_text(f"x\n{NARRATIVE}\n", encoding="utf-8")
commit(repo)
(repo / "docs" / "del.md").unlink()
@@ -135,6 +135,11 @@ POPULATION_EXEMPT = {
# DIRECTORY — because `<file>/**` and the pathspec `<file>` select different sets while spelling
# the same string. Reviewed 2026-08-30.
"test_ci_image_paths_pin_agreement.py": "index used for a per-member directory test, not a population",
# Its population is `CLAIMS` in `scripts/tests/mutation_manifest.py` — declared entries, not
# files. It consults the index only to answer a per-member question about each entry's `site`:
# is this named path one git TRACKS, as opposed to a scratch file that would be present on the
# author's machine and absent in the sandbox and in CI. Reviewed 2026-09-04.
"test_mutation_harness.py": "index used for a per-member tracked test on declared claim sites, not a population",
}
_HELPER = "scripts.tests.tracked_files"
@@ -624,9 +629,10 @@ def test_every_index_derived_module_is_registered():
WHAT NO ASSERTION CAN DECIDE, dated so it is re-examined rather than assumed: whether an
exemption is still WARRANTED. A stale key and an unexplained one are both caught below, but an
exempt module that later grows a real derived population stays uncovered and silent. Reviewed
2026-08-30 — two entries. One uses the helper only to assemble a tmp fixture copy; the other
consults the index per-member, to decide whether a named path is a directory, and derives its
population from parsed workflow documents instead. Neither asserts a file-population claim.
2026-09-04 — three entries. One uses the helper only to assemble a tmp fixture copy; the other
two consult the index per-member, to decide whether a named path is a directory and whether a
named path is tracked at all, and derive their populations from parsed workflow documents and
from a declared manifest instead. None asserts a file-population claim.
"""
exempt = {Path(__file__).name} | set(POPULATION_EXEMPT)
importers = _modules_importing_the_helper()
+311 -3
View File
@@ -48,14 +48,17 @@ import pytest
from scripts.tests.mutation_harness_lib import (
_BASELINES,
Claim,
Mutation,
_git,
build_sandbox,
reset_sandbox,
run_pytest,
verify_claim,
verify_mutation,
)
from scripts.tests.mutation_manifest import MUTATIONS, UNDECLARED
from scripts.tests.mutation_manifest import CLAIMS, MUTATIONS, UNDECLARED
from scripts.tests.tracked_files import tracked_file_set
REPO_ROOT = Path(__file__).resolve().parents[2]
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
@@ -201,6 +204,117 @@ def test_every_entry_declares_a_SPECIFIC_diagnostic_it_must_redden_with():
assert not clashes, f"{m.guard}'s expectation is contained in {clashes}'s — neither is specific"
# ------------------------------------------------------------------------------------------------
# THE PROSE CLAIMS — the same rule over a population that is not the inventory (ersatztv#881)
# ------------------------------------------------------------------------------------------------
def test_the_CLAIM_population_is_NOT_EMPTY():
"""ANTI-VACUITY for everything below, which is parametrized over `CLAIMS`.
`@parametrize` over an empty tuple collects one placeholder reported as SKIPPED, not a failure,
so an emptied `CLAIMS` would leave the whole prose half of this file passing over nothing while
reading exactly like coverage. That is the same reason
`test_the_DETECTOR_survivor_set_is_empty_ON_PURPOSE` exists — with the opposite expectation,
because an empty survivor set is a recorded decision and an empty claim set is a mechanism that
has quietly stopped being used.
"""
assert CLAIMS, (
"CLAIMS is empty, so every check below passes over nothing. An opt-in binding nobody uses is "
"indistinguishable from no binding at all — declare the claims, or retire the mechanism "
"deliberately and say so in testing.mutation-claims-are-executed."
)
def test_every_CLAIM_is_BOUND_to_the_PROSE_it_justifies():
"""The binding, both halves, because a claim entry unbound from its sentence proves nothing.
A claim is an assertion ABOUT a sentence in another file. If the sentence is reworded, moved or
deleted, the entry keeps executing a mutation nobody is claiming anything about and reports it as
verified — which is precisely the decay `MUTATION` rows were suffering before #790, one level up.
MEMBERSHIP is answered by the git INDEX, never by the disk
(`testing.guard-derives-population-from-source`): a claim pointing at an untracked scratch file
would otherwise pass here on the author's machine and be absent in the sandbox and in CI.
EXISTENCE on disk is a separate assertion below, because a tracked path with no file is a tree
mid-edit and should be reported, not read as an absent site.
"""
tracked = tracked_file_set()
untracked = sorted({c.site for c in CLAIMS if c.site not in tracked})
assert not untracked, (
f"these claim sites are not tracked by git: {untracked}. The prose has to live in the "
"repository for the binding to mean anything."
)
for claim in CLAIMS:
# Existence ASSERTED, not filtered, per `tracked_files.py`'s own contract: a path in the
# index with no file on disk means the tree is mid-edit, and a bare `FileNotFoundError`
# from the read below would name the path without saying why this file cares.
site = REPO_ROOT / claim.site
assert site.is_file(), (
f"git tracks {claim.site} but there is no file there, so the sentence this entry is "
"bound to cannot be read. The tree is mid-edit; this is reported rather than skipped."
)
text = site.read_text(encoding="utf-8")
occurrences = text.count(claim.quote)
assert occurrences == 1, (
f"{claim.site}: the declared quote occurs {occurrences} times, not once — "
f"{claim.quote!r}. RETARGET the entry rather than loosening the match. If the sentence "
"was deleted, delete this entry; if it was reworded, re-read it and re-declare, because "
"a reworded claim is a NEW claim and this harness has never executed it."
)
assert claim.quote.strip() == claim.quote and len(claim.quote) >= 20, (
f"{claim.site}: the quote is too short or padded to identify a sentence: {claim.quote!r}"
)
def test_every_CLAIM_declares_a_DIRECTION_and_the_FIELDS_THAT_DIRECTION_NEEDS():
"""RED and GREEN are read differently, and the shape has to say which before anything runs.
A RED claim is the assertion a `MUTATION` row makes and needs the same specific diagnostic: exit
status alone would let a crash certify it. A GREEN claim has no failing run to read a diagnostic
out of, so an `expect` on one could only be dead text — and dead text beside a live field is how
a reader comes to believe something is checked. `verify_claim` owns the unknown-outcome case, so
it is deliberately not re-asserted here; two copies of one rule is how they come to disagree.
THE REACH FIELDS ARE REQUIRED ON A GREEN AND FORBIDDEN ON A RED, and that asymmetry is the whole
difference between the two directions. A red is self-checking: a proof that never executes the
mutated file stays green and the harness refuses the entry. A green is not — an unrelated proof
produces the same green the sentence claims — so a GREEN entry must declare a second mutation of
the same clause that DOES redden its proof, with its own specific diagnostic, or the entry is
unreadable. A RED entry declaring them would be dead text of exactly the kind the `expect` rule
above rejects.
"""
for claim in CLAIMS:
if claim.outcome == Claim.RED:
assert len(claim.expect.strip()) >= 20, (
f"{claim.site}: a RED claim needs a specific diagnostic, not {claim.expect!r} — "
"otherwise any red at all satisfies it, including a crash."
)
assert claim.reach_replacement == "" and claim.reach_expect == "", (
f"{claim.site}: a RED claim must declare no reach mutation. Its own mutation already "
"proves the proof reaches the clause — a green there is what the harness refuses — "
"so these fields would be read by nothing."
)
elif claim.outcome == Claim.GREEN:
assert claim.expect == "", (
f"{claim.site}: a GREEN claim must carry no `expect` ({claim.expect!r}). There is no "
"failing run to match it against, so it would read as a check and be none."
)
assert claim.reach_replacement and claim.reach_replacement != claim.replacement, (
f"{claim.site}: a GREEN claim must declare a `reach_replacement` distinct from the "
"mutation it claims is unnoticed. Without one, a proof that never executes this "
"clause produces exactly the green the sentence asserts and the entry certifies "
"nothing."
)
assert len(claim.reach_expect.strip()) >= 20, (
f"{claim.site}: the reach mutation's diagnostic is {claim.reach_expect!r}. It is read "
"by the same gate a RED claim's is, and for the same reason: a red for an unrelated "
"reason would certify reach the proof does not have."
)
assert claim.why.strip(), f"{claim.site}: every claim must say what its mutation does"
assert claim.target and claim.clause and claim.replacement, f"{claim.site}: incomplete mutation"
# ------------------------------------------------------------------------------------------------
# THE EXECUTION — one isolated repository, reused, reset between mutations
# ------------------------------------------------------------------------------------------------
@@ -221,12 +335,26 @@ def sandbox(tmp_path_factory):
# simply absent there and the run below reports "file or directory not found" — accurate, and
# unreadable as a diagnosis. Named here instead, because this is what a developer adding a guard
# hits first.
missing = sorted({m.node_id.split("::")[0] for m in MUTATIONS if not (sb / m.node_id.split("::")[0]).is_file()})
node_ids = [m.node_id for m in MUTATIONS] + [c.node_id for c in CLAIMS]
missing = sorted({n.split("::")[0] for n in node_ids if not (sb / n.split("::")[0]).is_file()})
assert not missing, (
f"these proof files are not in the sandbox: {missing}. The sandbox is derived from "
"`git ls-files`, so an unstaged new file is not in it — `git add` it and re-run."
)
result = run_pytest(sb, [m.node_id for m in MUTATIONS])
# The claim proofs join the control for the reason the row proofs do, and it holds in BOTH
# directions: over a proof that was ALREADY red, a RED claim is satisfied by redness its mutation
# did not cause, and a GREEN claim is refused for a reason that has nothing to do with its
# mutation — reported as a wrong sentence when what is wrong is the proof.
# WHAT THIS CONTROL CANNOT DO, so it is not read as more. It cannot see a single ref that
# collects nothing — both assertions below are over the AGGREGATE of every ref, so the word
# `passed` is supplied by the others. Nor can it see a ref that collects plenty and reaches
# NOTHING the claim mutates, which is the larger of the two: such a proof is green here and green
# under the mutation, and a GREEN claim would read as verified on a run that never executed its
# clause. Both discriminations are per-claim and live in `verify_claim` — its vacuity gate and
# its relevance gate — each disarmed and witnessed in
# `test_MUTATION_disarming_the_GREEN_VACUITY_gate_accepts_a_run_in_which_NOTHING_PASSED` and
# `test_MUTATION_disarming_the_GREEN_RELEVANCE_gate_accepts_a_proof_that_NEVER_REACHES_THE_CLAUSE`.
result = run_pytest(sb, node_ids)
assert result.returncode == 0, (
"the proof tests named by the inventory are NOT green on an unmutated copy of this "
"repository, so nothing below can distinguish 'the mutation was noticed' from 'the test was "
@@ -280,6 +408,16 @@ def test_MUTATION_the_declared_clause_reddens_the_named_proof(sandbox, mutation)
assert verdict.ok, f"{mutation.guard}: {verdict.reason}"
@pytest.mark.parametrize("claim", CLAIMS, ids=lambda c: f"{c.site}:{c.outcome}")
def test_CLAIM_the_declared_mutation_produces_the_DECLARED_OUTCOME(sandbox, claim):
"""The prose, executed. A red here is not necessarily a defect in the code — for a GREEN claim it
usually means the mutation is noticed NOW, which is good news about the code and a sentence that
has to be rewritten. Either way the claim has stopped being true, which is the whole point."""
reset_sandbox(sandbox)
verdict = verify_claim(sandbox, claim)
assert verdict.ok, f"{claim.site} — {claim.quote!r}: {verdict.reason}"
_SURVIVORS = tuple(m for m in MUTATIONS if m.granularity == Mutation.DETECTOR)
@@ -351,6 +489,17 @@ def test_every_SURVIVING_clause_mutation_still_does(sandbox, mutation):
EXIT_STATUS_CLAUSE = " if result.returncode != 1:"
DIAGNOSTIC_CLAUSE = " if mutation.expect not in diagnostic:"
# The GREEN direction's three gates (#881). They are separate clauses precisely so each can be
# disarmed alone: a run that ERRORED, a run in which NOTHING PASSED and a proof that never reaches
# the mutated clause are different ways for a green to be worthless, and a single condition covering
# them is a guard nothing can witness failing.
GREEN_STATUS_CLAUSE = " if result.returncode != 0:"
GREEN_VACUITY_CLAUSE = ' if "passed" not in output:'
GREEN_RELEVANCE_CLAUSE = " if not reach.ok:"
# The PRE-FLIGHT refusal that decides before any proof runs and still needs a disarm proof: a
# replacement equal to its clause writes the target back unchanged, so what the proof ref then runs
# against is the ORIGINAL tree.
IDENTICAL_REPLACEMENT_CLAUSE = " if mutated == original:"
LIB = REPO_ROOT / "scripts" / "tests" / "mutation_harness_lib.py"
@@ -400,6 +549,11 @@ def _lib_with(tmp_path: Path, clause: str, replacement: str, label: str):
spec.loader.exec_module(mutant)
finally:
sys.modules.pop(spec.name, None)
# The copy's `_BASELINES` starts EMPTY — it is a fresh module object — so a sandbox registered
# with the real library is invisible to it and its `reset_sandbox` would refuse a tree it has no
# baseline for. Share the registry rather than rebuilding the sandbox for the mutant: the subject
# of every proof below is one clause of this copy, not where its sandbox came from.
mutant._BASELINES.update(_BASELINES)
return mutant
@@ -470,6 +624,30 @@ def test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_
)
def test_MUTATION_disarming_the_IDENTICAL_REPLACEMENT_gate_CERTIFIES_A_ROW_THAT_MUTATED_NOTHING(tmp_path):
"""The pre-flight refusal, isolated the same way, and it is the one no later gate can stand in for.
A replacement equal to its clause leaves the file byte-identical, so the proof ref runs against an
UNMUTATED tree. Over a proof that is already red for its own reasons, that is indistinguishable
from a detection — exit 1, carrying the declared diagnostic — so the exit-status and diagnostic
gates both accept and this one is what refuses. Which is why the fixture fails on its own: a green
fixture would be rejected by the status gate instead and the two would mask each other (#685).
"""
boom = "a failure this fixture produces with no mutation applied at all"
sb, inert = _inert_sandbox(tmp_path, f" raise RuntimeError({boom!r})\n")
no_op = replace(inert, replacement=inert.clause, expect=boom)
verdict = verify_mutation(sb, no_op)
assert not verdict.ok, "a replacement identical to its clause was accepted as a mutation"
assert "nothing was mutated" in verdict.reason, verdict.reason
mutant = _lib_with(tmp_path, IDENTICAL_REPLACEMENT_CLAUSE, " if False:", "identical replacement gate")
assert mutant.verify_mutation(sb, no_op).ok, (
"the identical-replacement gate was replaced with a constant and a row that mutated NOTHING "
"was still rejected, so the verdict does not hang on the clause that compares the two texts"
)
def test_a_clause_that_no_longer_OCCURS_ONCE_is_reported_rather_than_applied(tmp_path):
"""The failure paths, driven directly, because this is where a harness quietly stops harnessing.
@@ -500,3 +678,133 @@ def test_the_sandbox_is_left_UNCHANGED_by_a_verdict(tmp_path):
before = subject.read_text()
verify_mutation(sb, inert)
assert subject.read_text() == before, "verify_mutation left its mutation in the sandbox"
# ------------------------------------------------------------------------------------------------
# THE GREEN DIRECTION'S OWN PROOFS — three gates, each disarmed ALONE (ersatztv#881)
# ------------------------------------------------------------------------------------------------
# The reach mutation these fixtures declare: it replaces the marker with a test that FAILS, so the
# proof reddens with a diagnostic nothing else in the fixture produces. That is what a real reach
# declaration does — a second mutation of the same clause whose red is read through the diagnostic
# gate, not through exit status.
_REACH_DIAGNOSTIC = "a deliberate failure the reach mutation injects"
_REACH_REPLACEMENT = f'def test_reach_red():\n assert False, "{_REACH_DIAGNOSTIC}"'
def _inert_claim_sandbox(tmp_path: Path, body: str, reach_replacement: str = _REACH_REPLACEMENT) -> tuple[Path, Claim]:
"""A synthetic sandbox holding one test file, and a GREEN claim over it.
The claim's mutation is an inert comment edit, so whatever the file's tests do under it is what
they do without it — which is what isolates the VERDICT's reading of the result from the
mutation's effect. `body` decides the outcome each proof below needs, and `reach_replacement`
decides what the relevance gate sees; the default reddens, so a caller testing an EARLIER gate
is not stopped by this one.
It is a REAL git repository, unlike `_inert_sandbox` above, because `verify_claim` resets between
its two runs and `reset_sandbox` refuses a tree it has no recorded baseline for. Registering the
baseline is what `build_sandbox` does for the repo-sized one; here it is three git calls.
"""
sb = tmp_path / "sb"
tests = sb / "scripts" / "tests"
tests.mkdir(parents=True)
(tests / "test_inert.py").write_text(f"# INERT MARKER\n{body}")
_git(sb, "init", "-q", ".")
_git(sb, "add", "-A", "-f", ".")
_git(sb, "commit", "-qm", "inert claim sandbox")
_BASELINES[str(sb.resolve())] = _git(sb, "rev-parse", "HEAD").stdout.decode().strip()
return sb, Claim(
site="scripts/tests/test_inert.py",
quote="a sentence this fixture does not have to contain, since the binding is checked elsewhere",
target="scripts/tests/test_inert.py",
clause="# INERT MARKER",
replacement="# INERT MARKER, CHANGED",
proof="test_inert.py",
outcome=Claim.GREEN,
expect="",
why="an inert mutation over a file whose outcome the fixture fixes",
reach_replacement=reach_replacement,
reach_expect=_REACH_DIAGNOSTIC,
)
def test_an_UNKNOWN_outcome_is_REFUSED_rather_than_read_as_either(tmp_path):
"""`verify_claim` branches on the outcome, so a value it does not recognise must not fall through
to a verdict. A typo'd direction reading as verified is the shape this whole file exists to
remove, one level up from the claims themselves."""
sb, claim = _inert_claim_sandbox(tmp_path, "\n\ndef test_ok():\n assert True\n")
verdict = verify_claim(sb, replace(claim, outcome="green"))
assert not verdict.ok, "an unrecognised outcome was accepted"
assert "unknown outcome" in verdict.reason, verdict.reason
def test_MUTATION_disarming_the_GREEN_EXIT_STATUS_gate_accepts_a_proof_that_WENT_RED(tmp_path):
"""Disarm the status gate alone and a GREEN claim whose proof FAILED must start reporting verified.
The fixture's file both fails and passes a test, deliberately: a file that only failed would also
be rejected by the vacuity gate below, and the two would mask each other — a mutant that reddens
because a SECOND clause is doing the work proves nothing about the one under test (#685).
"""
body = "\n\ndef test_ok():\n assert True\n\n\ndef test_red():\n assert False\n"
sb, claim = _inert_claim_sandbox(tmp_path, body)
verdict = verify_claim(sb, claim)
assert not verdict.ok, "a proof that went RED was accepted as a verified GREEN claim"
assert "exited 1" in verdict.reason, verdict.reason
mutant = _lib_with(tmp_path, GREEN_STATUS_CLAUSE, " if False:", "green status gate")
assert mutant.verify_claim(sb, claim).ok, (
"the green-direction status gate was replaced with a constant and a proof that FAILED was "
"still rejected, so the verdict does not hang on the status it reads"
)
def test_MUTATION_disarming_the_GREEN_VACUITY_gate_accepts_a_run_in_which_NOTHING_PASSED(tmp_path):
"""The second gate, isolated the same way. Pytest exits 0 when every test SKIPS, so exit status
alone certifies a green in which nothing was given the chance to notice the mutation — which is
exactly the vacuous-completeness failure the inventory itself exists to prevent."""
body = (
"\n\nimport pytest\n\n\n@pytest.mark.skip(reason='nothing runs here')\ndef test_skipped():\n assert True\n"
)
sb, claim = _inert_claim_sandbox(tmp_path, body)
verdict = verify_claim(sb, claim)
assert not verdict.ok, "a run in which every test skipped was accepted as a verified GREEN claim"
assert "NOTHING PASSED" in verdict.reason, verdict.reason
mutant = _lib_with(tmp_path, GREEN_VACUITY_CLAUSE, " if False:", "green vacuity gate")
assert mutant.verify_claim(sb, claim).ok, (
"the green-direction vacuity gate was replaced with a constant and a run in which no test "
"passed was still rejected, so the verdict does not hang on the clause that reads it"
)
def test_MUTATION_disarming_the_GREEN_RELEVANCE_gate_accepts_a_proof_that_NEVER_REACHES_THE_CLAUSE(tmp_path):
"""The third gate, and the one the other two cannot stand in for.
This is the hole in the shape: a proof that never executes the mutated clause exits 0 with tests
genuinely passing, so the status gate and the vacuity gate both accept, and the run is
indistinguishable from the green the sentence claims. Only the declared reach mutation separates
them — and here it does not redden either, which is exactly what a proof that reads a different
file looks like.
The fixture makes the reach mutation inert too, rather than pointing the proof at a second file:
"the proof does not depend on this clause" is the property under test, and an inert mutation of
the clause is that property in its smallest form. `verify_mutation` refuses it in the words it
uses for a clause that has stopped being load-bearing, which is the same fact from the other end.
"""
sb, claim = _inert_claim_sandbox(
tmp_path,
"\n\ndef test_ok():\n assert True\n",
reach_replacement="# INERT MARKER, ALSO CHANGED",
)
verdict = verify_claim(sb, claim)
assert not verdict.ok, "a green produced by a proof that never reaches the clause was accepted"
assert "DECLARED REACH" in verdict.reason, verdict.reason
mutant = _lib_with(tmp_path, GREEN_RELEVANCE_CLAUSE, " if False:", "green relevance gate")
assert mutant.verify_claim(sb, claim).ok, (
"the relevance gate was replaced with a constant and a green from a proof that does not "
"depend on the clause was still rejected, so the verdict does not hang on the reach mutation"
)
+11
View File
@@ -82,6 +82,17 @@ def _git_ls_files() -> list[str]:
return paths
def tracked_file_set() -> set[str]:
"""Every tracked path as a set, for guards that test MEMBERSHIP of one NAMED path.
The other helpers here select a population by directory; this one answers "does git track
exactly this path", which is what a guard checking a hand-written reference needs. Same
authoritative source, so a reference to a file that was deleted or renamed reports as absent
instead of being silently satisfied by whatever is on disk.
"""
return set(_git_ls_files())
def tracked_children(directory: str, patterns: tuple[str, ...]) -> set[str]:
"""Tracked files that are DIRECT children of `directory` and match one of `patterns`.