Extend mutation-claim enforcement to prose (#881) #914
+1
-1
@@ -28,7 +28,7 @@ doc below, or that changes which sections a task signal points to.**
|
||||
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
|
||||
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
|
||||
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
|
||||
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite) — plus `…/verification-code-needs-its-own-proof.md`, which extends the same obligation BEYOND guards to the harness, wrapper or checker doing the checking, and says where its proof lives when the checker holds no row |
|
||||
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible), then the rules every guard must satisfy: `docs/decisions/records/testing/guard-derives-population-from-source.md`, `…/guard-ships-with-mutation-proof.md` and `…/mutation-claims-are-executed.md` (a `MUTATION` grade carries a DECLARED clause mutation that is re-run every suite — and so does a PROSE claim that some mutation reddens, or does not redden, a named test, wherever it is written: it is a `CLAIMS` entry in the same manifest, bound to its site and verbatim quote, or it is not written) — plus `…/verification-code-needs-its-own-proof.md`, which extends the same obligation BEYOND guards to the harness, wrapper or checker doing the checking, and says where its proof lives when the checker holds no row |
|
||||
| Adding or bounding a consequential numeric config field (an FFmpeg profile tunable, a pipeline knob) | `docs/api-conventions.md` §3d — reject out of range with a 422 naming the bound and its consequence, never accept-then-rewrite; validate against the constants the renderer reads, keep the render-time clamp for pre-existing rows, and let an UNCHANGED legacy value through on update. Then `api.ffmpeg-profile-numeric-bounds` |
|
||||
| Testing a surface gated by config / an env var / a credential | `docs/decisions/records/testing/deny-path-at-production-config-value.md` — cover the setting absent, at its production value, and each opt-out, and assert the DENY branch |
|
||||
| Touching a full-replace write path or a hand-built request object | `docs/decisions/records/testing/full-replace-asserts-field-list.md` — derive the field list from the DTO and assert set equality; reconcile by id where child state exists. In the SPA the same rule is enforced by the type system: `docs/spa-conventions.md` §4b — build the body as `Complete<T>`, annotating BOTH the wrapper parameter and every construction site |
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+12
-1
@@ -261,7 +261,7 @@ recorded as unexamined rather than as cleared.
|
||||
| `scripts/tests/test_merge_consent_head_change.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_merge_consent_required_check.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_migration_equivalence.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
|
||||
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names, or when a DECLARED PROSE CLAIM in `CLAIMS` stops producing the outcome its sentence asserts (either direction) | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
|
||||
| `scripts/tests/test_post_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_prepush_unsets_git_env.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_pr_changed_files.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
@@ -369,6 +369,17 @@ set equality in both directions, so the grade and the mutation cannot drift apar
|
||||
graded `MUTATION` without declaring its clause fails the suite. Full contract and its limits:
|
||||
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
|
||||
|
||||
**That file also executes a SECOND population, which is not this table and must not be read off it.**
|
||||
`CLAIMS` in the same manifest binds declared PROSE claims — a sentence in a code comment, a test
|
||||
docstring or a decision record asserting that some mutation reddens, or does not redden, a named
|
||||
test — to the file and the sentence that make them (ersatztv#881). Those entries carry a `site` and
|
||||
a verbatim `quote` instead of a guard, so a rewritten sentence reports as a retarget; and they may
|
||||
declare a GREEN outcome, which no row here can — one that also declares a REACH mutation of the same
|
||||
clause required to redden the same proof, because a green from a proof that never touches the mutated
|
||||
file is indistinguishable from the green the sentence claims. A guard's row says nothing about
|
||||
whether a claim about that guard has been declared, and the two populations are derived differently
|
||||
on purpose.
|
||||
|
||||
One thing that column still does not say: it does not say the declared clause is the ONLY thing the
|
||||
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
|
||||
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
|
||||
|
||||
@@ -186,8 +186,12 @@ def added_lines(diff: str):
|
||||
p = raw[4:]
|
||||
# `/dev/null` on the new side means the file was DELETED. Stated honestly: this arm
|
||||
# is DEFENSIVE, not load-bearing — a deletion contributes no `+` lines, so nothing is
|
||||
# yielded for it either way, and removing this arm reddens no test. It is kept because
|
||||
# `path` should never name a file the added lines do not belong to. A
|
||||
# yielded for it either way. Measured rather than asserted, and re-measured on every
|
||||
# run: `test_check_doc_narrative.py` stays green with this arm removed, a declared
|
||||
# claim in `scripts/tests/mutation_manifest.py` (ersatztv#881). The named file is the
|
||||
# scope that is actually executed — a claim over every test in the repository would
|
||||
# be wider than anything re-taking it. The arm is kept because `path` should never
|
||||
# name a file the added lines do not belong to. A
|
||||
# `--diff-filter=d` on the git call was removed rather than kept beside it: a second
|
||||
# mechanism nobody can witness failing is how a duplicate guard hides its twin.
|
||||
# Git terminates the `+++` filename with a TAB when the path contains a space, and a
|
||||
|
||||
@@ -313,8 +313,12 @@ status_readback=$(api_get "repos/$owner/$repo/commits/$sha/status?limit=100") \
|
||||
# Three outcomes are kept apart below, each with its own message, because they call for different
|
||||
# actions: a body whose `.statuses` is not an array at
|
||||
# all, a well-formed body that does not carry our context, and a body carrying MORE THAN ONE row for
|
||||
# it. Collapsing them also makes the array-TYPE test unwitnessable — the absent-entry check catches
|
||||
# every case on its own, so disarming the type test leaves the suite green.
|
||||
# it. Keeping them apart is also what makes the array-TYPE test witnessable, and that is EXECUTED
|
||||
# rather than asserted: disarming the type test feeds a `null` `.statuses` to `map(select(...))`,
|
||||
# jq refuses to iterate it, and the script dies with the PARSE message instead of the shape one — so
|
||||
# `test_a_readback_whose_statuses_array_is_NULL_is_refused` reddens. It carries no date because it
|
||||
# is not a witnessing: it is a declared claim in `CLAIMS` (`scripts/tests/mutation_manifest.py`),
|
||||
# re-taken on every run (ersatztv#881).
|
||||
readback_matches=$(printf '%s' "$status_readback" | jq -c --arg c "$STATUS_CONTEXT" \
|
||||
'if (.statuses | type) == "array" then (.statuses | map(select(.context == $c))) else null end') \
|
||||
|| die "the '$STATUS_CONTEXT' status was posted on $short, but the read-back body could not be parsed — no comment was written."
|
||||
|
||||
@@ -111,6 +111,69 @@ class Mutation:
|
||||
return f"scripts/tests/{self.proof}"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Claim:
|
||||
"""One PROSE claim about a mutation's outcome, bound to the sentence that makes it.
|
||||
|
||||
A `Mutation` above is keyed on a GUARD, because its population is the `MUTATION`-graded rows of
|
||||
`docs/guard-inventory.md`. A claim is keyed on the PROSE instead: `site` names the tracked file
|
||||
whose text makes the claim and `quote` is that text, verbatim. Both halves are checked every run
|
||||
— the quote must occur exactly once in the site — so a reworded sentence reports as a retarget
|
||||
rather than drifting away from the entry that justifies it. That binding is the thing #812's four
|
||||
defects were missing; each was a sentence with nothing tying it to what it asserted.
|
||||
|
||||
`outcome` is `RED` or `GREEN`, and the GREEN direction is why this is not just another `Mutation`.
|
||||
A `MUTATION` row always claims a red, so the harness only ever had to recognise one. Prose does
|
||||
not, and the corpus carries the other direction too: `.gitea/workflows/review-verdict.yml`,
|
||||
`scripts/check-doc-narrative.py` and `scripts/tests/hook_fire_isolation.py` each name a mutation —
|
||||
an arm disarmed on its own, an arm removed, a dead branch deleted — and assert that nothing
|
||||
notices it. How each of those sentences words its outcome is deliberately not repeated here: a
|
||||
restatement is a second copy of an outcome, free to drift from the site while the entry that
|
||||
executes it stays green, which is the drift a `Claim` exists to close. What they share is the
|
||||
shape — a claim that a mutation is NOT noticed — and for that shape the rule "declare it or do
|
||||
not write it" is unsatisfiable unless the harness can execute a negative. A GREEN entry therefore
|
||||
carries NO `expect`: there is no failing run to read a diagnostic out of, and the three things
|
||||
that could make a green vacuous — a run that errored, a run in which nothing passed, and a proof
|
||||
that never reaches the mutated clause at all — are checked in `verify_claim` instead.
|
||||
|
||||
THE THIRD OF THOSE IS WHAT MAKES A GREEN READABLE, and it needs its own declaration. A red is
|
||||
self-checking: a proof that ignores the mutated file stays green and `verify_mutation` refuses it,
|
||||
naming the clause as not load-bearing. A green has no such property — an unrelated proof produces
|
||||
exactly the green the sentence claims, so exit status and "something passed" together still
|
||||
certify a run that never executed the clause. `reach_replacement` closes that: a SECOND mutation
|
||||
of the SAME clause, declared to REDDEN the same proof with the diagnostic in `reach_expect`, and
|
||||
executed through `verify_mutation` so the red is read exactly as a `MUTATION` row's is. Passing it
|
||||
means the proof reaches this clause and its value changes what the proof observes; only then does
|
||||
the declared green say anything. RED entries leave both fields empty — there is nothing there for
|
||||
them to add.
|
||||
|
||||
What a GREEN claim is worth is exactly what the retired `DETECTOR` survivor was worth: it is
|
||||
re-run and required to KEEP surviving, so the day the mutation starts being noticed the entry
|
||||
goes red and the sentence has to be rewritten. `testing.mutation-claims-are-executed` records the
|
||||
direction that matters — a claim that can no longer decay is a tautology reading like a proof.
|
||||
"""
|
||||
|
||||
RED = "RED"
|
||||
GREEN = "GREEN"
|
||||
|
||||
site: str
|
||||
quote: str
|
||||
target: str
|
||||
clause: str
|
||||
replacement: str
|
||||
proof: str
|
||||
outcome: str
|
||||
expect: str
|
||||
why: str
|
||||
reach_replacement: str = ""
|
||||
reach_expect: str = ""
|
||||
|
||||
@property
|
||||
def node_id(self) -> str:
|
||||
"""A claim may name a whole proof FILE, where the prose says "leaves the suite green"."""
|
||||
return f"scripts/tests/{self.proof}"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
ok: bool
|
||||
@@ -271,35 +334,55 @@ _PYTEST_RED_MEANINGS = {
|
||||
}
|
||||
|
||||
|
||||
def _mutate_and_run(
|
||||
sandbox: Path, target_path: str, clause: str, replacement: str, node_id: str
|
||||
) -> tuple[Verdict | None, subprocess.CompletedProcess | None]:
|
||||
"""Apply one declared clause mutation and run one proof ref under it.
|
||||
|
||||
Split out of `verify_mutation` so the RED and GREEN directions share ONE application path rather
|
||||
than two copies of it. Everything that can go wrong BEFORE the proof runs — an absent target, a
|
||||
clause that has moved or acquired a second site, a replacement identical to the clause — is the
|
||||
same question in both directions and is answered here; each direction then reads the result its
|
||||
own way. A `Verdict` comes back only when the mutation could not be applied at all.
|
||||
"""
|
||||
target = sandbox / target_path
|
||||
if not target.is_file():
|
||||
return Verdict(False, f"the mutation target {target_path} does not exist in the sandbox"), None
|
||||
|
||||
original = target.read_text(encoding="utf-8")
|
||||
occurrences = original.count(clause)
|
||||
if occurrences != 1:
|
||||
return (
|
||||
Verdict(
|
||||
False,
|
||||
f"the declared clause occurs {occurrences} times in {target_path}, not once. "
|
||||
"RETARGET it rather than loosening the match — a clause that has moved, or that now "
|
||||
"matches a second site, means the recorded proof no longer points at what it claims to.",
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
mutated = original.replace(clause, replacement, 1)
|
||||
if mutated == original:
|
||||
return Verdict(False, "the replacement is identical to the clause, so nothing was mutated"), None
|
||||
|
||||
target.write_text(mutated, encoding="utf-8")
|
||||
try:
|
||||
return None, run_pytest(sandbox, [node_id])
|
||||
finally:
|
||||
target.write_text(original, encoding="utf-8")
|
||||
|
||||
|
||||
def verify_mutation(sandbox: Path, mutation: Mutation) -> Verdict:
|
||||
"""Apply one declared mutation in `sandbox` and require its named test to go red.
|
||||
|
||||
The sandbox is left as it was found; callers still `reset_sandbox` between mutations because a
|
||||
driven guard can dirty the tree in ways this function does not know about.
|
||||
"""
|
||||
target = sandbox / mutation.target
|
||||
if not target.is_file():
|
||||
return Verdict(False, f"the mutation target {mutation.target} does not exist in the sandbox")
|
||||
|
||||
original = target.read_text(encoding="utf-8")
|
||||
occurrences = original.count(mutation.clause)
|
||||
if occurrences != 1:
|
||||
return Verdict(
|
||||
False,
|
||||
f"the declared clause occurs {occurrences} times in {mutation.target}, not once. "
|
||||
"RETARGET it rather than loosening the match — a clause that has moved, or that now "
|
||||
"matches a second site, means the recorded proof no longer points at what it claims to.",
|
||||
)
|
||||
|
||||
mutated = original.replace(mutation.clause, mutation.replacement, 1)
|
||||
if mutated == original:
|
||||
return Verdict(False, "the replacement is identical to the clause, so nothing was mutated")
|
||||
|
||||
target.write_text(mutated, encoding="utf-8")
|
||||
try:
|
||||
result = run_pytest(sandbox, [mutation.node_id])
|
||||
finally:
|
||||
target.write_text(original, encoding="utf-8")
|
||||
refusal, result = _mutate_and_run(sandbox, mutation.target, mutation.clause, mutation.replacement, mutation.node_id)
|
||||
if refusal is not None:
|
||||
return refusal
|
||||
assert result is not None # `_mutate_and_run` returns exactly one of the two
|
||||
|
||||
output = result.stdout + result.stderr
|
||||
if result.returncode != 1:
|
||||
@@ -327,3 +410,104 @@ def verify_mutation(sandbox: Path, mutation: Mutation) -> Verdict:
|
||||
f"{output[-3000:]}",
|
||||
)
|
||||
return Verdict(True, "the named test went red under the declared mutation, with the declared diagnostic")
|
||||
|
||||
|
||||
def verify_claim(sandbox: Path, claim: Claim) -> Verdict:
|
||||
"""Execute one declared prose claim: apply its mutation and require its DECLARED direction.
|
||||
|
||||
RED delegates to `verify_mutation`, deliberately and not for brevity: a claim that a mutation IS
|
||||
noticed is the same assertion a `MUTATION` row makes, and giving it a second implementation is
|
||||
how the two would come to disagree about what a red is worth (exit status, diagnostic, an
|
||||
already-red proof).
|
||||
|
||||
GREEN is the direction that has no equivalent above, and it is checked by THREE separate clauses
|
||||
because each alone is satisfiable by a run that proves nothing. Exit status alone accepts a run
|
||||
in which every test SKIPPED — pytest exits 0 for that — "something passed" alone accepts a file
|
||||
that also failed, and both together still accept a proof that never touches the mutated file:
|
||||
an unrelated test run produces the same green as the claimed one. That last one is answered by
|
||||
the entry's declared `reach_replacement`, a second mutation of the SAME clause required to REDDEN
|
||||
the same proof. They are kept apart rather than combined into one condition so each can be
|
||||
disarmed on its own and seen to matter; a single clause covering all three would be a guard
|
||||
nothing can witness failing.
|
||||
|
||||
THE RELEVANCE GATE RUNS LAST BECAUSE IT IS THE EXPENSIVE ONE: it is the only one of the three
|
||||
that costs a second full run of the proof — under the reach mutation, with a sandbox reset in
|
||||
between — so an entry the status or the vacuity gate has already refused never pays for it. What
|
||||
keeps the three worth three is that each carries its own disarm proof, rather than the position
|
||||
it sits in. The sandbox is reset between the two runs because the first proof run can dirty the
|
||||
tree, and a second result that is a function of the first's is what `reset_sandbox` exists to
|
||||
prevent.
|
||||
"""
|
||||
if claim.outcome == Claim.RED:
|
||||
return verify_mutation(
|
||||
sandbox,
|
||||
Mutation(
|
||||
guard=claim.site,
|
||||
target=claim.target,
|
||||
clause=claim.clause,
|
||||
replacement=claim.replacement,
|
||||
proof=claim.proof,
|
||||
granularity=Mutation.CLAUSE,
|
||||
expect=claim.expect,
|
||||
why=claim.why,
|
||||
),
|
||||
)
|
||||
if claim.outcome != Claim.GREEN:
|
||||
return Verdict(
|
||||
False,
|
||||
f"unknown outcome {claim.outcome!r} on the claim at {claim.site} — declare RED (the "
|
||||
"mutation is noticed) or GREEN (it is not). An outcome this function does not recognise "
|
||||
"must not read as a verified claim.",
|
||||
)
|
||||
|
||||
refusal, result = _mutate_and_run(sandbox, claim.target, claim.clause, claim.replacement, claim.node_id)
|
||||
if refusal is not None:
|
||||
return refusal
|
||||
assert result is not None # `_mutate_and_run` returns exactly one of the two
|
||||
|
||||
output = result.stdout + result.stderr
|
||||
if result.returncode != 0:
|
||||
return Verdict(
|
||||
False,
|
||||
f"the prose at {claim.site} says this mutation leaves {claim.proof} GREEN, but the run "
|
||||
f"exited {result.returncode}. If the mutation is now noticed that is good news about the "
|
||||
"code and bad news about the sentence: rewrite the sentence, and regrade this entry RED "
|
||||
f"with the diagnostic it produces.\n--- inner pytest output ---\n{output[-3000:]}",
|
||||
)
|
||||
if "passed" not in output:
|
||||
return Verdict(
|
||||
False,
|
||||
f"the run of {claim.proof} exited 0 but NOTHING PASSED — every test was skipped, "
|
||||
"deselected or absent. A green nothing ran is not evidence that the mutation went "
|
||||
f"unnoticed.\n--- inner pytest output ---\n{output[-3000:]}",
|
||||
)
|
||||
|
||||
reset_sandbox(sandbox)
|
||||
reach = verify_mutation(
|
||||
sandbox,
|
||||
Mutation(
|
||||
guard=claim.site,
|
||||
target=claim.target,
|
||||
clause=claim.clause,
|
||||
replacement=claim.reach_replacement,
|
||||
proof=claim.proof,
|
||||
granularity=Mutation.CLAUSE,
|
||||
expect=claim.reach_expect,
|
||||
why=claim.why,
|
||||
),
|
||||
)
|
||||
if not reach.ok:
|
||||
return Verdict(
|
||||
False,
|
||||
f"{claim.proof} stayed green under the declared mutation, but the DECLARED REACH "
|
||||
"mutation of the same clause did not redden it, so that green is not evidence about "
|
||||
"this clause: a proof that never executes it produces exactly the same result. Either "
|
||||
"the proof no longer reaches the clause — in which case the sentence at "
|
||||
f"{claim.site} is about something this proof cannot see — or the reach mutation itself "
|
||||
f"needs re-declaring.\n--- the reach verdict ---\n{reach.reason}",
|
||||
)
|
||||
return Verdict(
|
||||
True,
|
||||
"the named proof stayed green under the declared mutation, with tests actually run, and "
|
||||
"reddened under the declared reach mutation of the same clause",
|
||||
)
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
"""The DECLARED clause mutations, one per `MUTATION`-graded row of `docs/guard-inventory.md`.
|
||||
"""The DECLARED clause mutations: one per `MUTATION`-graded row of `docs/guard-inventory.md`
|
||||
in `MUTATIONS`, plus one per declared PROSE claim about a mutation's outcome in `CLAIMS`
|
||||
(ersatztv#881). The two populations are derived differently and are kept apart for that reason;
|
||||
the `CLAIMS` section below states its own.
|
||||
|
||||
Data only. The machinery that applies these is `mutation_harness_lib.py`; the checks that keep this
|
||||
file honest are `test_mutation_harness.py`.
|
||||
@@ -22,10 +25,12 @@ and why; no count is kept here, because a count of the entries below is a second
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from scripts.tests.mutation_harness_lib import Mutation
|
||||
from scripts.tests.mutation_harness_lib import Claim, Mutation
|
||||
|
||||
CLAUSE = Mutation.CLAUSE
|
||||
DETECTOR = Mutation.DETECTOR
|
||||
RED = Claim.RED
|
||||
GREEN = Claim.GREEN
|
||||
|
||||
MUTATIONS: tuple[Mutation, ...] = (
|
||||
Mutation(
|
||||
@@ -317,10 +322,20 @@ MUTATIONS: tuple[Mutation, ...] = (
|
||||
"`mutation_harness_lib.py` so a clause of it can be disarmed in an isolated copy at all. The "
|
||||
"clause is the DIAGNOSTIC gate — the check that a failing proof failed with the diagnostic "
|
||||
"its row declares. Disarmed, a red for any unrelated reason is certified as a guard doing "
|
||||
"its job, which is the shape that made two rows in this very file measure nothing. The other "
|
||||
"gate, the one requiring pytest exit code 1, carries its own proof in "
|
||||
"`test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_TEST`; the "
|
||||
"inventory holds one ref per row, so this entry names the stronger of the two.",
|
||||
"its job, which is the shape that made two rows in this very file measure nothing. THE "
|
||||
"INVENTORY HOLDS ONE REF PER ROW, so this entry names the gate verified rows hang on most "
|
||||
"directly, and the gates that read a proof RUN carry disarm proofs of their own: "
|
||||
"`test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_TEST` for the "
|
||||
"exit-status gate and, on the GREEN direction `CLAIMS` uses, the status, vacuity and "
|
||||
"relevance gates. "
|
||||
"So does the PRE-FLIGHT refusal that no later gate can stand in for — a replacement "
|
||||
"identical to its clause mutates nothing, so the proof runs against the original tree and "
|
||||
"an already-red one reddens exactly like a detection: "
|
||||
"`test_MUTATION_disarming_the_IDENTICAL_REPLACEMENT_gate_CERTIFIES_A_ROW_THAT_MUTATED_NOTHING`. "
|
||||
"Named rather than counted, because a count moves silently when one is added — and naming "
|
||||
"is not a claim the list is CLOSED: the two remaining pre-flight refusals, an absent target "
|
||||
"and a clause that does not occur exactly once, decide before any proof runs and this entry "
|
||||
"says nothing about what they are worth.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/ci-toolchain-image-resolves.sh",
|
||||
@@ -544,3 +559,124 @@ UNDECLARED: dict[str, str] = {
|
||||
"have to go into the guarded artifact — the release-path scan job in the workflow. Which "
|
||||
"weakening of that job is THE defect it exists to catch has not been settled.",
|
||||
}
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE PROSE CLAIMS — same rule, wider population (ersatztv#881)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
#
|
||||
# `MUTATIONS` above covers the `MUTATION`-graded rows of `docs/guard-inventory.md` and nothing else,
|
||||
# which is where ersatztv#812's four defects got in: a claim about a mutation's outcome written in a
|
||||
# TEST DOCSTRING, a DECISION RECORD or a CODE COMMENT was outside the executed-claim rule by
|
||||
# construction, so four consecutive review rounds each found one that was simply false. The rule was
|
||||
# already the right rule; its population was its first site.
|
||||
#
|
||||
# So a claim entry is keyed on the PROSE, not on a guard. `site` names the tracked file that makes
|
||||
# the claim and `quote` is the sentence, verbatim: the harness requires the quote to occur exactly
|
||||
# once there, so the entry and the sentence cannot drift apart silently. That binding is the half
|
||||
# #812's defects were missing — every one of them was an assertion with nothing tying it to its
|
||||
# referent.
|
||||
#
|
||||
# THE QUOTE SPANS THE MUTATION AND THE OUTCOME. The mutation half of a sentence is the part that
|
||||
# is uncontroversial; the checkable assertion is the outcome — `reddens <this test>`, `leaves
|
||||
# the suite green`. A quote that stops before it binds the wrong half: the words that make the
|
||||
# sentence a claim could then be reworded, or the test renamed in the prose, while the entry's
|
||||
# binding still resolves.
|
||||
#
|
||||
# THE REFERENT IS DECLARED, NEVER INFERRED — the same doctrine the clause mutations above are built
|
||||
# on, applied one level up. A checker that went looking for sentences that SOUND like mutation claims
|
||||
# would be a string predicate over prose, which `docs/defect-shapes-773.md` §4 and
|
||||
# `docs.no-session-narrative` both refuse to make load-bearing. What is checked here is a claim that
|
||||
# named its own referent; what is NOT checked is a claim that named none, and that limit is stated in
|
||||
# `testing.mutation-claims-are-executed` rather than papered over.
|
||||
#
|
||||
# `outcome` is RED or GREEN. GREEN is new and is the point: a sentence asserting that a mutation is
|
||||
# NOT noticed is one no `MUTATION` row can express, so before this the rule "declare it or do not
|
||||
# write it" had no way to be satisfied for that shape at all. A
|
||||
# GREEN entry carries no `expect` — there is no red to read a diagnostic out of — and is worth
|
||||
# exactly what the retired `DETECTOR` survivor was worth: it is required to KEEP surviving, and the
|
||||
# day the mutation starts being noticed the entry reddens and the sentence has to be rewritten.
|
||||
#
|
||||
# A GREEN ENTRY ALSO DECLARES ITS `reach_replacement`, AND IS WORTHLESS WITHOUT IT. The two
|
||||
# directions are not symmetric: a RED claim checks itself, because a proof that never executes the
|
||||
# mutated file stays green and the harness refuses it in the same words it refuses a clause that has
|
||||
# stopped being load-bearing. A GREEN claim has no such property — an unrelated proof produces
|
||||
# exactly the green the sentence asserts, so a green certifies nothing until the proof is shown to
|
||||
# reach the clause. `reach_replacement` is a SECOND mutation of the SAME clause, declared to redden
|
||||
# the same proof with `reach_expect`, and it is executed through the same `verify_mutation` the rows
|
||||
# above use, so its red is read exactly as theirs is rather than on exit status.
|
||||
#
|
||||
# No count is kept here either, for the reason stated at the top of this file.
|
||||
|
||||
CLAIMS: tuple[Claim, ...] = (
|
||||
Claim(
|
||||
site="scripts/post-review-verdict.sh",
|
||||
# The quote spans the OUTCOME half and the named test, not only the mutation half: the
|
||||
# checkable assertion here is `<mutation> reddens <this test>`, and a quote stopping at
|
||||
# the comma would leave the words that make it a claim free to be reworded — or the test
|
||||
# renamed in the prose — while the entry's binding still resolves.
|
||||
quote=(
|
||||
"rather than asserted: disarming the type test feeds a `null` `.statuses` to "
|
||||
"`map(select(...))`,\n"
|
||||
"# jq refuses to iterate it, and the script dies with the PARSE message instead of "
|
||||
"the shape one — so\n"
|
||||
"# `test_a_readback_whose_statuses_array_is_NULL_is_refused` reddens."
|
||||
),
|
||||
target="scripts/post-review-verdict.sh",
|
||||
clause='if (.statuses | type) == "array" then',
|
||||
replacement="if true then",
|
||||
proof="test_post_review_verdict.py::test_a_readback_whose_statuses_array_is_NULL_is_refused",
|
||||
outcome=RED,
|
||||
expect="Cannot iterate over null (null)",
|
||||
why="The sentence explains why three read-back outcomes are kept apart with separate "
|
||||
"messages, and it asserts that the array-TYPE test is what keeps them apart. Disarmed, a "
|
||||
"`null` `.statuses` reaches `map(select(...))`, jq refuses to iterate it and the `|| die` "
|
||||
"fires with the PARSE message, so the named test fails on the SHAPE message it asserts. The "
|
||||
"declared diagnostic is jq's own, not that assertion's expected text: the expected text also "
|
||||
"appears in the comparison pytest prints, so matching it would certify any red inside the "
|
||||
"same assertion, including one produced by an unrelated change to the script.",
|
||||
),
|
||||
Claim(
|
||||
site="scripts/check-doc-narrative.py",
|
||||
quote="`test_check_doc_narrative.py` stays green with this arm removed",
|
||||
target="scripts/check-doc-narrative.py",
|
||||
clause=' path = None if p == "/dev/null" else (p[2:] if p.startswith("b/") else p)',
|
||||
replacement=' path = p[2:] if p.startswith("b/") else p',
|
||||
proof="test_check_doc_narrative.py",
|
||||
outcome=GREEN,
|
||||
expect="",
|
||||
# THE RELEVANCE GATE, and without it this entry certifies nothing: a proof that never
|
||||
# executes this line produces exactly the green the sentence predicts. The reach mutation
|
||||
# drops the `b/` stripping the same expression performs, so every `+++ b/<path>` header
|
||||
# resolves to a path `is_scanned_path` rejects and the run scans NOTHING — which is what the
|
||||
# declared diagnostic reads, from the ACTUAL half of the comparison rather than the expected
|
||||
# one. Passing it means the proof reaches this clause and what the clause computes changes
|
||||
# what the proof observes, which is the precondition the green half needs and cannot supply.
|
||||
reach_replacement=" path = p",
|
||||
reach_expect="scanned 0 file(s); 0 advisory warning(s)",
|
||||
why="The `/dev/null` arm is declared DEFENSIVE in its own comment — a deleted file "
|
||||
"contributes no `+` lines, so nothing is yielded for it either way. The comment says so and "
|
||||
"then names an outcome, which is a measurement, and until this entry nothing re-took it. The "
|
||||
"mutation removes exactly the arm the comment is about, leaving the `b/` stripping it is "
|
||||
"wrapped around, so a red here would mean the arm has become load-bearing and the comment is "
|
||||
"now wrong. The quote names the proof FILE rather than every test in the repository, because "
|
||||
"the file is what the harness runs: an outcome half broader than the executed scope is the "
|
||||
"same drift in miniature.",
|
||||
),
|
||||
Claim(
|
||||
site="docs/decisions/records/docs/record-wing-parse-guard.md",
|
||||
quote="deleting the single line that WIRES it into `main()` reddens a dedicated test",
|
||||
target="scripts/decisions_validate.py",
|
||||
clause=" wing_faults=record_wing_faults() + yaml_faults,",
|
||||
replacement=" wing_faults=yaml_faults,",
|
||||
proof="test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan",
|
||||
outcome=RED,
|
||||
expect="a wing fault must fail the validator",
|
||||
why="A RED claim, and one made in a DECISION RECORD — the site class two of #812's four "
|
||||
"defects lived in, and the one furthest from anything that executes. The record says the "
|
||||
"wiring line's removal previously left the whole suite green while a real record vanished, "
|
||||
"so the dedicated test is the only thing standing between that defect and a silent return. "
|
||||
"The mutation drops `record_wing_faults()` from the call and keeps `yaml_faults`, which is "
|
||||
"the deletion the sentence describes rather than a caricature of it.",
|
||||
),
|
||||
)
|
||||
|
||||
@@ -110,9 +110,12 @@ def test_a_DELETED_doc_is_not_reported_as_added_content(repo: Path) -> None:
|
||||
"""A deletion contributes no added lines.
|
||||
|
||||
Stated exactly: this pins the BEHAVIOUR, and it is NOT a proof of the `+++ /dev/null` arm, which
|
||||
is defensive — removing that arm reddens nothing, because a deletion yields no `+` lines either
|
||||
way. The script comment says the same. A docstring claiming a proof it does not have is worse
|
||||
than no docstring: it is the thing a later reader trusts instead of re-checking."""
|
||||
is defensive — a deletion yields no `+` lines either way. What removing that arm does to this
|
||||
file is deliberately NOT restated here: it is a declared claim in
|
||||
`scripts/tests/mutation_manifest.py`, re-executed on every run (ersatztv#881), and a second copy
|
||||
of an outcome is free to drift from the entry that executes it. A docstring claiming a proof it
|
||||
does not have is worse than no docstring: it is the thing a later reader trusts instead of
|
||||
re-checking."""
|
||||
(repo / "docs" / "del.md").write_text(f"x\n{NARRATIVE}\n", encoding="utf-8")
|
||||
commit(repo)
|
||||
(repo / "docs" / "del.md").unlink()
|
||||
|
||||
@@ -135,6 +135,11 @@ POPULATION_EXEMPT = {
|
||||
# DIRECTORY — because `<file>/**` and the pathspec `<file>` select different sets while spelling
|
||||
# the same string. Reviewed 2026-08-30.
|
||||
"test_ci_image_paths_pin_agreement.py": "index used for a per-member directory test, not a population",
|
||||
# Its population is `CLAIMS` in `scripts/tests/mutation_manifest.py` — declared entries, not
|
||||
# files. It consults the index only to answer a per-member question about each entry's `site`:
|
||||
# is this named path one git TRACKS, as opposed to a scratch file that would be present on the
|
||||
# author's machine and absent in the sandbox and in CI. Reviewed 2026-09-04.
|
||||
"test_mutation_harness.py": "index used for a per-member tracked test on declared claim sites, not a population",
|
||||
}
|
||||
|
||||
_HELPER = "scripts.tests.tracked_files"
|
||||
@@ -624,9 +629,10 @@ def test_every_index_derived_module_is_registered():
|
||||
WHAT NO ASSERTION CAN DECIDE, dated so it is re-examined rather than assumed: whether an
|
||||
exemption is still WARRANTED. A stale key and an unexplained one are both caught below, but an
|
||||
exempt module that later grows a real derived population stays uncovered and silent. Reviewed
|
||||
2026-08-30 — two entries. One uses the helper only to assemble a tmp fixture copy; the other
|
||||
consults the index per-member, to decide whether a named path is a directory, and derives its
|
||||
population from parsed workflow documents instead. Neither asserts a file-population claim.
|
||||
2026-09-04 — three entries. One uses the helper only to assemble a tmp fixture copy; the other
|
||||
two consult the index per-member, to decide whether a named path is a directory and whether a
|
||||
named path is tracked at all, and derive their populations from parsed workflow documents and
|
||||
from a declared manifest instead. None asserts a file-population claim.
|
||||
"""
|
||||
exempt = {Path(__file__).name} | set(POPULATION_EXEMPT)
|
||||
importers = _modules_importing_the_helper()
|
||||
|
||||
@@ -48,14 +48,17 @@ import pytest
|
||||
|
||||
from scripts.tests.mutation_harness_lib import (
|
||||
_BASELINES,
|
||||
Claim,
|
||||
Mutation,
|
||||
_git,
|
||||
build_sandbox,
|
||||
reset_sandbox,
|
||||
run_pytest,
|
||||
verify_claim,
|
||||
verify_mutation,
|
||||
)
|
||||
from scripts.tests.mutation_manifest import MUTATIONS, UNDECLARED
|
||||
from scripts.tests.mutation_manifest import CLAIMS, MUTATIONS, UNDECLARED
|
||||
from scripts.tests.tracked_files import tracked_file_set
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
|
||||
@@ -201,6 +204,117 @@ def test_every_entry_declares_a_SPECIFIC_diagnostic_it_must_redden_with():
|
||||
assert not clashes, f"{m.guard}'s expectation is contained in {clashes}'s — neither is specific"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE PROSE CLAIMS — the same rule over a population that is not the inventory (ersatztv#881)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_CLAIM_population_is_NOT_EMPTY():
|
||||
"""ANTI-VACUITY for everything below, which is parametrized over `CLAIMS`.
|
||||
|
||||
`@parametrize` over an empty tuple collects one placeholder reported as SKIPPED, not a failure,
|
||||
so an emptied `CLAIMS` would leave the whole prose half of this file passing over nothing while
|
||||
reading exactly like coverage. That is the same reason
|
||||
`test_the_DETECTOR_survivor_set_is_empty_ON_PURPOSE` exists — with the opposite expectation,
|
||||
because an empty survivor set is a recorded decision and an empty claim set is a mechanism that
|
||||
has quietly stopped being used.
|
||||
"""
|
||||
assert CLAIMS, (
|
||||
"CLAIMS is empty, so every check below passes over nothing. An opt-in binding nobody uses is "
|
||||
"indistinguishable from no binding at all — declare the claims, or retire the mechanism "
|
||||
"deliberately and say so in testing.mutation-claims-are-executed."
|
||||
)
|
||||
|
||||
|
||||
def test_every_CLAIM_is_BOUND_to_the_PROSE_it_justifies():
|
||||
"""The binding, both halves, because a claim entry unbound from its sentence proves nothing.
|
||||
|
||||
A claim is an assertion ABOUT a sentence in another file. If the sentence is reworded, moved or
|
||||
deleted, the entry keeps executing a mutation nobody is claiming anything about and reports it as
|
||||
verified — which is precisely the decay `MUTATION` rows were suffering before #790, one level up.
|
||||
|
||||
MEMBERSHIP is answered by the git INDEX, never by the disk
|
||||
(`testing.guard-derives-population-from-source`): a claim pointing at an untracked scratch file
|
||||
would otherwise pass here on the author's machine and be absent in the sandbox and in CI.
|
||||
EXISTENCE on disk is a separate assertion below, because a tracked path with no file is a tree
|
||||
mid-edit and should be reported, not read as an absent site.
|
||||
"""
|
||||
tracked = tracked_file_set()
|
||||
untracked = sorted({c.site for c in CLAIMS if c.site not in tracked})
|
||||
assert not untracked, (
|
||||
f"these claim sites are not tracked by git: {untracked}. The prose has to live in the "
|
||||
"repository for the binding to mean anything."
|
||||
)
|
||||
for claim in CLAIMS:
|
||||
# Existence ASSERTED, not filtered, per `tracked_files.py`'s own contract: a path in the
|
||||
# index with no file on disk means the tree is mid-edit, and a bare `FileNotFoundError`
|
||||
# from the read below would name the path without saying why this file cares.
|
||||
site = REPO_ROOT / claim.site
|
||||
assert site.is_file(), (
|
||||
f"git tracks {claim.site} but there is no file there, so the sentence this entry is "
|
||||
"bound to cannot be read. The tree is mid-edit; this is reported rather than skipped."
|
||||
)
|
||||
text = site.read_text(encoding="utf-8")
|
||||
occurrences = text.count(claim.quote)
|
||||
assert occurrences == 1, (
|
||||
f"{claim.site}: the declared quote occurs {occurrences} times, not once — "
|
||||
f"{claim.quote!r}. RETARGET the entry rather than loosening the match. If the sentence "
|
||||
"was deleted, delete this entry; if it was reworded, re-read it and re-declare, because "
|
||||
"a reworded claim is a NEW claim and this harness has never executed it."
|
||||
)
|
||||
assert claim.quote.strip() == claim.quote and len(claim.quote) >= 20, (
|
||||
f"{claim.site}: the quote is too short or padded to identify a sentence: {claim.quote!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_every_CLAIM_declares_a_DIRECTION_and_the_FIELDS_THAT_DIRECTION_NEEDS():
|
||||
"""RED and GREEN are read differently, and the shape has to say which before anything runs.
|
||||
|
||||
A RED claim is the assertion a `MUTATION` row makes and needs the same specific diagnostic: exit
|
||||
status alone would let a crash certify it. A GREEN claim has no failing run to read a diagnostic
|
||||
out of, so an `expect` on one could only be dead text — and dead text beside a live field is how
|
||||
a reader comes to believe something is checked. `verify_claim` owns the unknown-outcome case, so
|
||||
it is deliberately not re-asserted here; two copies of one rule is how they come to disagree.
|
||||
|
||||
THE REACH FIELDS ARE REQUIRED ON A GREEN AND FORBIDDEN ON A RED, and that asymmetry is the whole
|
||||
difference between the two directions. A red is self-checking: a proof that never executes the
|
||||
mutated file stays green and the harness refuses the entry. A green is not — an unrelated proof
|
||||
produces the same green the sentence claims — so a GREEN entry must declare a second mutation of
|
||||
the same clause that DOES redden its proof, with its own specific diagnostic, or the entry is
|
||||
unreadable. A RED entry declaring them would be dead text of exactly the kind the `expect` rule
|
||||
above rejects.
|
||||
"""
|
||||
for claim in CLAIMS:
|
||||
if claim.outcome == Claim.RED:
|
||||
assert len(claim.expect.strip()) >= 20, (
|
||||
f"{claim.site}: a RED claim needs a specific diagnostic, not {claim.expect!r} — "
|
||||
"otherwise any red at all satisfies it, including a crash."
|
||||
)
|
||||
assert claim.reach_replacement == "" and claim.reach_expect == "", (
|
||||
f"{claim.site}: a RED claim must declare no reach mutation. Its own mutation already "
|
||||
"proves the proof reaches the clause — a green there is what the harness refuses — "
|
||||
"so these fields would be read by nothing."
|
||||
)
|
||||
elif claim.outcome == Claim.GREEN:
|
||||
assert claim.expect == "", (
|
||||
f"{claim.site}: a GREEN claim must carry no `expect` ({claim.expect!r}). There is no "
|
||||
"failing run to match it against, so it would read as a check and be none."
|
||||
)
|
||||
assert claim.reach_replacement and claim.reach_replacement != claim.replacement, (
|
||||
f"{claim.site}: a GREEN claim must declare a `reach_replacement` distinct from the "
|
||||
"mutation it claims is unnoticed. Without one, a proof that never executes this "
|
||||
"clause produces exactly the green the sentence asserts and the entry certifies "
|
||||
"nothing."
|
||||
)
|
||||
assert len(claim.reach_expect.strip()) >= 20, (
|
||||
f"{claim.site}: the reach mutation's diagnostic is {claim.reach_expect!r}. It is read "
|
||||
"by the same gate a RED claim's is, and for the same reason: a red for an unrelated "
|
||||
"reason would certify reach the proof does not have."
|
||||
)
|
||||
assert claim.why.strip(), f"{claim.site}: every claim must say what its mutation does"
|
||||
assert claim.target and claim.clause and claim.replacement, f"{claim.site}: incomplete mutation"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE EXECUTION — one isolated repository, reused, reset between mutations
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
@@ -221,12 +335,26 @@ def sandbox(tmp_path_factory):
|
||||
# simply absent there and the run below reports "file or directory not found" — accurate, and
|
||||
# unreadable as a diagnosis. Named here instead, because this is what a developer adding a guard
|
||||
# hits first.
|
||||
missing = sorted({m.node_id.split("::")[0] for m in MUTATIONS if not (sb / m.node_id.split("::")[0]).is_file()})
|
||||
node_ids = [m.node_id for m in MUTATIONS] + [c.node_id for c in CLAIMS]
|
||||
missing = sorted({n.split("::")[0] for n in node_ids if not (sb / n.split("::")[0]).is_file()})
|
||||
assert not missing, (
|
||||
f"these proof files are not in the sandbox: {missing}. The sandbox is derived from "
|
||||
"`git ls-files`, so an unstaged new file is not in it — `git add` it and re-run."
|
||||
)
|
||||
result = run_pytest(sb, [m.node_id for m in MUTATIONS])
|
||||
# The claim proofs join the control for the reason the row proofs do, and it holds in BOTH
|
||||
# directions: over a proof that was ALREADY red, a RED claim is satisfied by redness its mutation
|
||||
# did not cause, and a GREEN claim is refused for a reason that has nothing to do with its
|
||||
# mutation — reported as a wrong sentence when what is wrong is the proof.
|
||||
# WHAT THIS CONTROL CANNOT DO, so it is not read as more. It cannot see a single ref that
|
||||
# collects nothing — both assertions below are over the AGGREGATE of every ref, so the word
|
||||
# `passed` is supplied by the others. Nor can it see a ref that collects plenty and reaches
|
||||
# NOTHING the claim mutates, which is the larger of the two: such a proof is green here and green
|
||||
# under the mutation, and a GREEN claim would read as verified on a run that never executed its
|
||||
# clause. Both discriminations are per-claim and live in `verify_claim` — its vacuity gate and
|
||||
# its relevance gate — each disarmed and witnessed in
|
||||
# `test_MUTATION_disarming_the_GREEN_VACUITY_gate_accepts_a_run_in_which_NOTHING_PASSED` and
|
||||
# `test_MUTATION_disarming_the_GREEN_RELEVANCE_gate_accepts_a_proof_that_NEVER_REACHES_THE_CLAUSE`.
|
||||
result = run_pytest(sb, node_ids)
|
||||
assert result.returncode == 0, (
|
||||
"the proof tests named by the inventory are NOT green on an unmutated copy of this "
|
||||
"repository, so nothing below can distinguish 'the mutation was noticed' from 'the test was "
|
||||
@@ -280,6 +408,16 @@ def test_MUTATION_the_declared_clause_reddens_the_named_proof(sandbox, mutation)
|
||||
assert verdict.ok, f"{mutation.guard}: {verdict.reason}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("claim", CLAIMS, ids=lambda c: f"{c.site}:{c.outcome}")
|
||||
def test_CLAIM_the_declared_mutation_produces_the_DECLARED_OUTCOME(sandbox, claim):
|
||||
"""The prose, executed. A red here is not necessarily a defect in the code — for a GREEN claim it
|
||||
usually means the mutation is noticed NOW, which is good news about the code and a sentence that
|
||||
has to be rewritten. Either way the claim has stopped being true, which is the whole point."""
|
||||
reset_sandbox(sandbox)
|
||||
verdict = verify_claim(sandbox, claim)
|
||||
assert verdict.ok, f"{claim.site} — {claim.quote!r}: {verdict.reason}"
|
||||
|
||||
|
||||
_SURVIVORS = tuple(m for m in MUTATIONS if m.granularity == Mutation.DETECTOR)
|
||||
|
||||
|
||||
@@ -351,6 +489,17 @@ def test_every_SURVIVING_clause_mutation_still_does(sandbox, mutation):
|
||||
|
||||
EXIT_STATUS_CLAUSE = " if result.returncode != 1:"
|
||||
DIAGNOSTIC_CLAUSE = " if mutation.expect not in diagnostic:"
|
||||
# The GREEN direction's three gates (#881). They are separate clauses precisely so each can be
|
||||
# disarmed alone: a run that ERRORED, a run in which NOTHING PASSED and a proof that never reaches
|
||||
# the mutated clause are different ways for a green to be worthless, and a single condition covering
|
||||
# them is a guard nothing can witness failing.
|
||||
GREEN_STATUS_CLAUSE = " if result.returncode != 0:"
|
||||
GREEN_VACUITY_CLAUSE = ' if "passed" not in output:'
|
||||
GREEN_RELEVANCE_CLAUSE = " if not reach.ok:"
|
||||
# The PRE-FLIGHT refusal that decides before any proof runs and still needs a disarm proof: a
|
||||
# replacement equal to its clause writes the target back unchanged, so what the proof ref then runs
|
||||
# against is the ORIGINAL tree.
|
||||
IDENTICAL_REPLACEMENT_CLAUSE = " if mutated == original:"
|
||||
LIB = REPO_ROOT / "scripts" / "tests" / "mutation_harness_lib.py"
|
||||
|
||||
|
||||
@@ -400,6 +549,11 @@ def _lib_with(tmp_path: Path, clause: str, replacement: str, label: str):
|
||||
spec.loader.exec_module(mutant)
|
||||
finally:
|
||||
sys.modules.pop(spec.name, None)
|
||||
# The copy's `_BASELINES` starts EMPTY — it is a fresh module object — so a sandbox registered
|
||||
# with the real library is invisible to it and its `reset_sandbox` would refuse a tree it has no
|
||||
# baseline for. Share the registry rather than rebuilding the sandbox for the mutant: the subject
|
||||
# of every proof below is one clause of this copy, not where its sandbox came from.
|
||||
mutant._BASELINES.update(_BASELINES)
|
||||
return mutant
|
||||
|
||||
|
||||
@@ -470,6 +624,30 @@ def test_MUTATION_disarming_the_EXIT_STATUS_gate_accepts_a_run_that_NEVER_RAN_A_
|
||||
)
|
||||
|
||||
|
||||
def test_MUTATION_disarming_the_IDENTICAL_REPLACEMENT_gate_CERTIFIES_A_ROW_THAT_MUTATED_NOTHING(tmp_path):
|
||||
"""The pre-flight refusal, isolated the same way, and it is the one no later gate can stand in for.
|
||||
|
||||
A replacement equal to its clause leaves the file byte-identical, so the proof ref runs against an
|
||||
UNMUTATED tree. Over a proof that is already red for its own reasons, that is indistinguishable
|
||||
from a detection — exit 1, carrying the declared diagnostic — so the exit-status and diagnostic
|
||||
gates both accept and this one is what refuses. Which is why the fixture fails on its own: a green
|
||||
fixture would be rejected by the status gate instead and the two would mask each other (#685).
|
||||
"""
|
||||
boom = "a failure this fixture produces with no mutation applied at all"
|
||||
sb, inert = _inert_sandbox(tmp_path, f" raise RuntimeError({boom!r})\n")
|
||||
no_op = replace(inert, replacement=inert.clause, expect=boom)
|
||||
|
||||
verdict = verify_mutation(sb, no_op)
|
||||
assert not verdict.ok, "a replacement identical to its clause was accepted as a mutation"
|
||||
assert "nothing was mutated" in verdict.reason, verdict.reason
|
||||
|
||||
mutant = _lib_with(tmp_path, IDENTICAL_REPLACEMENT_CLAUSE, " if False:", "identical replacement gate")
|
||||
assert mutant.verify_mutation(sb, no_op).ok, (
|
||||
"the identical-replacement gate was replaced with a constant and a row that mutated NOTHING "
|
||||
"was still rejected, so the verdict does not hang on the clause that compares the two texts"
|
||||
)
|
||||
|
||||
|
||||
def test_a_clause_that_no_longer_OCCURS_ONCE_is_reported_rather_than_applied(tmp_path):
|
||||
"""The failure paths, driven directly, because this is where a harness quietly stops harnessing.
|
||||
|
||||
@@ -500,3 +678,133 @@ def test_the_sandbox_is_left_UNCHANGED_by_a_verdict(tmp_path):
|
||||
before = subject.read_text()
|
||||
verify_mutation(sb, inert)
|
||||
assert subject.read_text() == before, "verify_mutation left its mutation in the sandbox"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE GREEN DIRECTION'S OWN PROOFS — three gates, each disarmed ALONE (ersatztv#881)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
# The reach mutation these fixtures declare: it replaces the marker with a test that FAILS, so the
|
||||
# proof reddens with a diagnostic nothing else in the fixture produces. That is what a real reach
|
||||
# declaration does — a second mutation of the same clause whose red is read through the diagnostic
|
||||
# gate, not through exit status.
|
||||
_REACH_DIAGNOSTIC = "a deliberate failure the reach mutation injects"
|
||||
_REACH_REPLACEMENT = f'def test_reach_red():\n assert False, "{_REACH_DIAGNOSTIC}"'
|
||||
|
||||
|
||||
def _inert_claim_sandbox(tmp_path: Path, body: str, reach_replacement: str = _REACH_REPLACEMENT) -> tuple[Path, Claim]:
|
||||
"""A synthetic sandbox holding one test file, and a GREEN claim over it.
|
||||
|
||||
The claim's mutation is an inert comment edit, so whatever the file's tests do under it is what
|
||||
they do without it — which is what isolates the VERDICT's reading of the result from the
|
||||
mutation's effect. `body` decides the outcome each proof below needs, and `reach_replacement`
|
||||
decides what the relevance gate sees; the default reddens, so a caller testing an EARLIER gate
|
||||
is not stopped by this one.
|
||||
|
||||
It is a REAL git repository, unlike `_inert_sandbox` above, because `verify_claim` resets between
|
||||
its two runs and `reset_sandbox` refuses a tree it has no recorded baseline for. Registering the
|
||||
baseline is what `build_sandbox` does for the repo-sized one; here it is three git calls.
|
||||
"""
|
||||
sb = tmp_path / "sb"
|
||||
tests = sb / "scripts" / "tests"
|
||||
tests.mkdir(parents=True)
|
||||
(tests / "test_inert.py").write_text(f"# INERT MARKER\n{body}")
|
||||
_git(sb, "init", "-q", ".")
|
||||
_git(sb, "add", "-A", "-f", ".")
|
||||
_git(sb, "commit", "-qm", "inert claim sandbox")
|
||||
_BASELINES[str(sb.resolve())] = _git(sb, "rev-parse", "HEAD").stdout.decode().strip()
|
||||
return sb, Claim(
|
||||
site="scripts/tests/test_inert.py",
|
||||
quote="a sentence this fixture does not have to contain, since the binding is checked elsewhere",
|
||||
target="scripts/tests/test_inert.py",
|
||||
clause="# INERT MARKER",
|
||||
replacement="# INERT MARKER, CHANGED",
|
||||
proof="test_inert.py",
|
||||
outcome=Claim.GREEN,
|
||||
expect="",
|
||||
why="an inert mutation over a file whose outcome the fixture fixes",
|
||||
reach_replacement=reach_replacement,
|
||||
reach_expect=_REACH_DIAGNOSTIC,
|
||||
)
|
||||
|
||||
|
||||
def test_an_UNKNOWN_outcome_is_REFUSED_rather_than_read_as_either(tmp_path):
|
||||
"""`verify_claim` branches on the outcome, so a value it does not recognise must not fall through
|
||||
to a verdict. A typo'd direction reading as verified is the shape this whole file exists to
|
||||
remove, one level up from the claims themselves."""
|
||||
sb, claim = _inert_claim_sandbox(tmp_path, "\n\ndef test_ok():\n assert True\n")
|
||||
verdict = verify_claim(sb, replace(claim, outcome="green"))
|
||||
assert not verdict.ok, "an unrecognised outcome was accepted"
|
||||
assert "unknown outcome" in verdict.reason, verdict.reason
|
||||
|
||||
|
||||
def test_MUTATION_disarming_the_GREEN_EXIT_STATUS_gate_accepts_a_proof_that_WENT_RED(tmp_path):
|
||||
"""Disarm the status gate alone and a GREEN claim whose proof FAILED must start reporting verified.
|
||||
|
||||
The fixture's file both fails and passes a test, deliberately: a file that only failed would also
|
||||
be rejected by the vacuity gate below, and the two would mask each other — a mutant that reddens
|
||||
because a SECOND clause is doing the work proves nothing about the one under test (#685).
|
||||
"""
|
||||
body = "\n\ndef test_ok():\n assert True\n\n\ndef test_red():\n assert False\n"
|
||||
sb, claim = _inert_claim_sandbox(tmp_path, body)
|
||||
|
||||
verdict = verify_claim(sb, claim)
|
||||
assert not verdict.ok, "a proof that went RED was accepted as a verified GREEN claim"
|
||||
assert "exited 1" in verdict.reason, verdict.reason
|
||||
|
||||
mutant = _lib_with(tmp_path, GREEN_STATUS_CLAUSE, " if False:", "green status gate")
|
||||
assert mutant.verify_claim(sb, claim).ok, (
|
||||
"the green-direction status gate was replaced with a constant and a proof that FAILED was "
|
||||
"still rejected, so the verdict does not hang on the status it reads"
|
||||
)
|
||||
|
||||
|
||||
def test_MUTATION_disarming_the_GREEN_VACUITY_gate_accepts_a_run_in_which_NOTHING_PASSED(tmp_path):
|
||||
"""The second gate, isolated the same way. Pytest exits 0 when every test SKIPS, so exit status
|
||||
alone certifies a green in which nothing was given the chance to notice the mutation — which is
|
||||
exactly the vacuous-completeness failure the inventory itself exists to prevent."""
|
||||
body = (
|
||||
"\n\nimport pytest\n\n\n@pytest.mark.skip(reason='nothing runs here')\ndef test_skipped():\n assert True\n"
|
||||
)
|
||||
sb, claim = _inert_claim_sandbox(tmp_path, body)
|
||||
|
||||
verdict = verify_claim(sb, claim)
|
||||
assert not verdict.ok, "a run in which every test skipped was accepted as a verified GREEN claim"
|
||||
assert "NOTHING PASSED" in verdict.reason, verdict.reason
|
||||
|
||||
mutant = _lib_with(tmp_path, GREEN_VACUITY_CLAUSE, " if False:", "green vacuity gate")
|
||||
assert mutant.verify_claim(sb, claim).ok, (
|
||||
"the green-direction vacuity gate was replaced with a constant and a run in which no test "
|
||||
"passed was still rejected, so the verdict does not hang on the clause that reads it"
|
||||
)
|
||||
|
||||
|
||||
def test_MUTATION_disarming_the_GREEN_RELEVANCE_gate_accepts_a_proof_that_NEVER_REACHES_THE_CLAUSE(tmp_path):
|
||||
"""The third gate, and the one the other two cannot stand in for.
|
||||
|
||||
This is the hole in the shape: a proof that never executes the mutated clause exits 0 with tests
|
||||
genuinely passing, so the status gate and the vacuity gate both accept, and the run is
|
||||
indistinguishable from the green the sentence claims. Only the declared reach mutation separates
|
||||
them — and here it does not redden either, which is exactly what a proof that reads a different
|
||||
file looks like.
|
||||
|
||||
The fixture makes the reach mutation inert too, rather than pointing the proof at a second file:
|
||||
"the proof does not depend on this clause" is the property under test, and an inert mutation of
|
||||
the clause is that property in its smallest form. `verify_mutation` refuses it in the words it
|
||||
uses for a clause that has stopped being load-bearing, which is the same fact from the other end.
|
||||
"""
|
||||
sb, claim = _inert_claim_sandbox(
|
||||
tmp_path,
|
||||
"\n\ndef test_ok():\n assert True\n",
|
||||
reach_replacement="# INERT MARKER, ALSO CHANGED",
|
||||
)
|
||||
|
||||
verdict = verify_claim(sb, claim)
|
||||
assert not verdict.ok, "a green produced by a proof that never reaches the clause was accepted"
|
||||
assert "DECLARED REACH" in verdict.reason, verdict.reason
|
||||
|
||||
mutant = _lib_with(tmp_path, GREEN_RELEVANCE_CLAUSE, " if False:", "green relevance gate")
|
||||
assert mutant.verify_claim(sb, claim).ok, (
|
||||
"the relevance gate was replaced with a constant and a green from a proof that does not "
|
||||
"depend on the clause was still rejected, so the verdict does not hang on the reach mutation"
|
||||
)
|
||||
|
||||
@@ -82,6 +82,17 @@ def _git_ls_files() -> list[str]:
|
||||
return paths
|
||||
|
||||
|
||||
def tracked_file_set() -> set[str]:
|
||||
"""Every tracked path as a set, for guards that test MEMBERSHIP of one NAMED path.
|
||||
|
||||
The other helpers here select a population by directory; this one answers "does git track
|
||||
exactly this path", which is what a guard checking a hand-written reference needs. Same
|
||||
authoritative source, so a reference to a file that was deleted or renamed reports as absent
|
||||
instead of being silently satisfied by whatever is on disk.
|
||||
"""
|
||||
return set(_git_ls_files())
|
||||
|
||||
|
||||
def tracked_children(directory: str, patterns: tuple[str, ...]) -> set[str]:
|
||||
"""Tracked files that are DIRECT children of `directory` and match one of `patterns`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user