Files
ersatztv/scripts/tests/test_prove_fix.py
T
timothyandClaude Opus 5 8d9be0d4b8
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 8s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 21s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 23s
review-verdict/h10 Awaiting review verdict for 8d9be0d
Review verdict / Set review-verdict status (pull_request_target) Successful in 19s
PR Gates / Fix proofs (Proves trailers) (pull_request) Failing after 2m2s
PR Gates / Script tests (pytest) (pull_request) Successful in 4m17s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m45s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m49s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m28s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
fix(794): a status of 1 must come from pytest, not from a failed cd
Round 2: BLOCKED, 1 Blocker + 2 High + 2 Low; 5 of 10 round-1 items verified fixed.

THE BLOCKER IS THE SAFETY CLAIM ITSELF. `( cd X && pytest ); rc=$?` returns the
SUBSHELL's status, so a failed cd or a failed redirection also yields 1 — and the
"only pytest exit 1 counts as red" rule then accepts it as a witnessed test failure
with pytest never having run. The rule I added last round to close a false green had a
false green underneath it.

Fixed structurally: the phase writes pytest's status to a marker file only AFTER pytest
returns. An absent marker means "pytest did not complete" and is refused as a harness
failure, which cannot be confused with a verdict. Proven load-bearing —
test_a_harness_failure_is_NOT_reported_as_PROVEN goes RED when the marker logic is
mutated back to trusting the subshell status.

Two fail-open enumerations, both real:
- the CI job read each commit's trailer with an unchecked command substitution, so a
  failing `git show` produced an empty selector, skipped the commit, and let the job
  exit 0 having been unable to inspect a possible claim;
- `while ... done < <(git diff-tree ...)` cannot see the producer's exit status, so a
  git failure that had already emitted one complete record passed the non-empty check
  and reverted only PART of the commit — overstating "without the fix". The stream is
  now captured to a file with a checked status, and any truncated record is fatal.

The selector is now split once into an array and always expanded as "${SEL[@]}".
Unquoted it also applied PATHNAME expansion, so a node id containing a glob character
could select different tests than the trailer names.

PROVE_FIX_PATH is now gated behind PROVE_FIX_MUTATION_RUN=1: as an unrestricted ambient
override, a stray CI value would silently point the whole suite at another script.

Verified: 14 tests; the blocker fix and the mutation proof each witnessed red against
their own mutants; real commits 84165ab75 PROVEN / UNPROVEN against an unrelated test;
scripts/tests 681 passed, 2 skipped.

Proves: scripts/tests/test_prove_fix.py
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 10:57:44 +02:00

331 lines
16 KiB
Python

"""scripts/prove-fix.sh witnesses a fix's test failing before the fix (ersatztv#794).
THE NEGATIVE CONTROL IS THE POINT OF THIS FILE. A prover that reports PROVEN for
everything is worse than no prover: it manufactures exactly the confidence #794 exists
to withhold. So `test_unrelated_test_is_UNPROVEN` is the load-bearing case here, and the
positive case only tells us the script can distinguish the two.
Each test builds a throwaway git repo rather than pinning real commits from this
repository's history — a test anchored to a real sha rots the moment that sha is rebased
or the file moves, and then it passes for the wrong reason (or is deleted for being
flaky, which is worse).
"""
from __future__ import annotations
import os
import subprocess
import time
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
# Overridable so a MUTANT copy can be driven through these very tests. That is what makes
# the mutation proof below a witnessed red rather than an argument: the unchanged test is
# re-run against the mutant and observed to fail.
# Overridable so a MUTANT copy can be driven through these very tests — that is what makes
# the mutation proof below a witnessed red rather than an argument. Gated behind a sentinel
# so a stray CI value cannot silently point the whole suite at another script.
if os.environ.get("PROVE_FIX_PATH") and os.environ.get("PROVE_FIX_MUTATION_RUN") != "1":
raise RuntimeError(
"PROVE_FIX_PATH is set without PROVE_FIX_MUTATION_RUN=1. That would silently test a "
"different script than the one this suite vouches for."
)
PROVE_FIX = Path(os.environ.get("PROVE_FIX_PATH") or (REPO_ROOT / "scripts" / "prove-fix.sh"))
def _git(repo: Path, *args: str) -> str:
return subprocess.run(
["git", "-C", str(repo), *args],
check=True, capture_output=True, text=True,
).stdout.strip()
def _run(repo: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
["bash", str(PROVE_FIX), "--repo", str(repo), *args],
capture_output=True, text=True, cwd=str(repo),
)
@pytest.fixture
def fixrepo(tmp_path: Path) -> Path:
"""A repo whose HEAD is a fix: code change + a test that discriminates.
It also carries an UNRELATED test, present from the first commit, which passes with
or without the fix. That test is the negative control's subject.
"""
repo = tmp_path / "r"
(repo / "scripts" / "tests").mkdir(parents=True)
subprocess.run(["git", "init", "-q", str(repo)], check=True)
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
# --- commit 1: the bug, plus a test that cannot see it
(repo / "calc.py").write_text("def add(a, b):\n return a - b # bug\n")
(repo / "scripts" / "tests" / "test_unrelated.py").write_text(
"def test_unrelated():\n assert 1 + 1 == 2\n"
)
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "initial: buggy add, unrelated test")
# --- commit 2: the fix + a test that discriminates against its absence
(repo / "calc.py").write_text("def add(a, b):\n return a + b\n")
(repo / "scripts" / "tests" / "test_add.py").write_text(
"from calc import add\n\n\ndef test_add():\n assert add(2, 3) == 5\n"
)
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m",
"fix: add() returned a difference\n\nProves: scripts/tests/test_add.py")
return repo
def test_reverting_the_fix_reddens_its_test_PROVEN(fixrepo: Path) -> None:
r = _run(fixrepo, "HEAD", "scripts/tests/test_add.py")
assert r.returncode == 0, f"expected PROVEN (0), got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "PROVEN" in r.stdout
assert "UNPROVEN" not in r.stdout
def test_unrelated_test_is_UNPROVEN(fixrepo: Path) -> None:
"""THE NEGATIVE CONTROL. A test that passes without the fix must be refused.
Without this, every other assertion in this file is compatible with a script that
prints PROVEN unconditionally.
"""
r = _run(fixrepo, "HEAD", "scripts/tests/test_unrelated.py")
assert r.returncode == 1, f"expected UNPROVEN (1), got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "UNPROVEN" in r.stdout
assert "do not discriminate" in r.stdout
def test_selector_comes_from_the_Proves_trailer(fixrepo: Path) -> None:
"""No selector argument: it must read `Proves:` rather than guess."""
r = _run(fixrepo, "HEAD")
assert r.returncode == 0, f"expected PROVEN via trailer, got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "scripts/tests/test_add.py" in r.stdout
def test_no_selector_and_no_trailer_REFUSES(fixrepo: Path) -> None:
"""Refusing beats guessing: a heuristic silently proves nothing when a fix edits an
existing test, which is the failure mode being mechanised against."""
(fixrepo / "calc.py").write_text("def add(a, b):\n return a + b # touched\n")
_git(fixrepo, "add", "-A")
_git(fixrepo, "commit", "-q", "-m", "fix: no trailer here")
r = _run(fixrepo, "HEAD")
assert r.returncode == 3, f"expected 3 (no selector), got {r.returncode}\n{r.stderr}"
assert "Proves:" in r.stderr
def test_test_only_commit_REFUSES(fixrepo: Path) -> None:
"""A commit with no code side cannot be proven this way — it must say so, not pass."""
(fixrepo / "scripts" / "tests" / "test_extra.py").write_text("def test_x():\n assert True\n")
_git(fixrepo, "add", "-A")
_git(fixrepo, "commit", "-q", "-m", "test: add a test only\n\nProves: scripts/tests/test_extra.py")
r = _run(fixrepo, "HEAD")
assert r.returncode == 4, f"expected 4 (nothing to revert), got {r.returncode}\n{r.stderr}"
assert "nothing to revert" in r.stderr
def test_selector_matching_no_tests_REFUSES(fixrepo: Path) -> None:
"""`pytest` exits 5 when it collects nothing. Treating that as red would prove every
fix — a check that examined nothing reporting success."""
r = _run(fixrepo, "HEAD", "scripts/tests/test_does_not_exist.py")
assert r.returncode == 5, f"expected 5 (no tests collected), got {r.returncode}\n{r.stderr}"
assert "NO tests" in r.stderr
def test_added_code_file_is_removed_not_checked_out(tmp_path: Path) -> None:
"""A file the fix ADDED does not exist in the parent. `git checkout parent -- <new>`
fails there, and if that failure were swallowed the fix would stay in place and every
run would report a false PROVEN."""
repo = tmp_path / "r2"
(repo / "scripts" / "tests").mkdir(parents=True)
subprocess.run(["git", "init", "-q", str(repo)], check=True)
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "scripts" / "tests" / "keep.py").write_text("# placeholder\n")
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
# the fix ADDS helper.py (it has no parent version) and a test that needs it
(repo / "helper.py").write_text("def shout(s):\n return s.upper()\n")
(repo / "scripts" / "tests" / "test_helper.py").write_text(
"from helper import shout\n\n\ndef test_shout():\n assert shout('a') == 'A'\n"
)
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "feat: add shout\n\nProves: scripts/tests/test_helper.py")
r = _run(repo, "HEAD")
assert r.returncode == 0, f"expected PROVEN, got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "PROVEN" in r.stdout
def test_root_commit_REFUSES(tmp_path: Path) -> None:
repo = tmp_path / "r3"
repo.mkdir()
subprocess.run(["git", "init", "-q", str(repo)], check=True)
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "a.py").write_text("x = 1\n")
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "root")
r = _run(repo, "HEAD", "scripts/tests")
assert r.returncode == 5
assert "root commit" in r.stderr
def test_runs_under_the_system_bash_3_2(fixrepo: Path) -> None:
"""macOS ships /bin/bash 3.2, where `mapfile` is absent AND yields an empty array
instead of erroring. Assert by EXECUTING under that bash, not by grepping the source
for "mapfile" — the first version of this test did the latter and matched the comment
explaining why mapfile is avoided, which is a string predicate failing exactly as
docs/defect-shapes-773.md §3.7 says they do."""
system_bash = Path("/bin/bash")
if not system_bash.exists():
pytest.skip("/bin/bash not present")
ver = subprocess.run([str(system_bash), "--version"], capture_output=True, text=True).stdout
r = subprocess.run(
[str(system_bash), str(PROVE_FIX), "--repo", str(fixrepo), "HEAD",
"scripts/tests/test_add.py"],
capture_output=True, text=True, cwd=str(fixrepo),
)
assert r.returncode == 0, (
f"prove-fix.sh must work under the system bash ({ver.splitlines()[0] if ver else '?'}); "
f"got {r.returncode}\n{r.stdout}\n{r.stderr}"
)
assert "PROVEN" in r.stdout
def test_control_failure_REFUSES(fixrepo: Path) -> None:
"""A test that is ALREADY red with the fix in place proves nothing by being red after
a revert. Without this control, half a discrimination claim reads as the whole one."""
(fixrepo / "scripts" / "tests" / "test_broken.py").write_text(
"from calc import add\n\n\ndef test_broken():\n assert add(2, 3) == 99\n"
)
(fixrepo / "calc.py").write_text("def add(a, b):\n return a + b # unchanged\n")
_git(fixrepo, "add", "-A")
_git(fixrepo, "commit", "-q", "-m",
"fix: with an already-failing test\n\nProves: scripts/tests/test_broken.py")
r = _run(fixrepo, "HEAD")
assert r.returncode == 6, f"expected 6 (control failed), got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "control FAILED" in r.stderr
def test_control_green_then_red_is_reported_as_both(fixrepo: Path) -> None:
"""The PROVEN line must state both halves — green with, red without — because that
pair is the claim. Reporting only the red half is the overclaim being mechanised out."""
r = _run(fixrepo, "HEAD", "scripts/tests/test_add.py")
assert r.returncode == 0
assert "control" in r.stdout.lower()
assert "Green with the fix (control), red without it" in r.stdout
def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test(
tmp_path: Path,
) -> None:
"""Clause-level mutation, EXECUTED, with the unchanged test WITNESSED RED against it.
`docs/guard-inventory.md` is explicit that MUTATION means a mutation was executed and
the named test was *witnessed red* — feeding the real script a rejecting input is
BEHAVIOUR-ONLY, and the column "is not a grading curve". An earlier version of this
test deleted the clause and then asserted the MUTANT misbehaved, which left this test
green and proved nothing about whether the clause is load-bearing. Cold review caught
that, and it was right.
So: delete the `RC -eq 0 -> UNPROVEN` clause (the CLAUSE, not the file — #510), then
re-run the UNCHANGED `test_unrelated_test_is_UNPROVEN` against the mutant in a nested
pytest run. That test must go RED. Its red is the proof.
"""
real = REPO_ROOT / "scripts" / "prove-fix.sh"
src = real.read_text()
marker = 'if [ "$RC" -eq 0 ]; then'
assert marker in src, "the clause under mutation is gone — regrade the inventory row"
end = src.index(" exit 1\nfi\n", src.index(marker)) + len(" exit 1\nfi\n")
mutant = tmp_path / "prove-fix-mutant.sh"
mutant.write_text(src[: src.index(marker)] + src[end:])
assert marker not in mutant.read_text(), "mutation did not remove the clause"
env = {**os.environ, "PROVE_FIX_PATH": str(mutant), "PROVE_FIX_MUTATION_RUN": "1"}
nested = subprocess.run(
["python3", "-m", "pytest", f"{Path(__file__).name}::test_unrelated_test_is_UNPROVEN",
"-q", "-p", "no:cacheprovider"],
cwd=str(Path(__file__).parent), env=env, capture_output=True, text=True,
)
out = nested.stdout + nested.stderr
assert nested.returncode == 1, (
"the unchanged refusal test must go RED against the mutant (pytest exit 1). "
f"got {nested.returncode} — exit 2/3/4/5 would mean the nested run broke rather "
f"than the test failing, which proves nothing.\n{out[-2000:]}"
)
# Require a real reported FAILURE of that specific test. `returncode != 0` alone would
# be satisfied by a collection error — the vacuous shape this whole file is against.
assert "FAILED" in out and "test_unrelated_test_is_UNPROVEN" in out, (
"expected a reported failure of test_unrelated_test_is_UNPROVEN; the nested run "
f"failed for some other reason.\n{out[-2000:]}"
)
assert "1 failed" in out, f"expected exactly one failing test.\n{out[-1200:]}"
def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
"""A killed run must not look like evidence.
Cold review measured the first version of this script printing PROVEN and exiting 0
after a SIGTERM: the trap cleaned up but did not exit, so the previous status stood.
Cancellation masquerading as proof is the exact failure this tool exists to prevent,
so the signal path gets its own test rather than an argument.
"""
repo = tmp_path / "slow"
(repo / "scripts" / "tests").mkdir(parents=True)
subprocess.run(["git", "init", "-q", str(repo)], check=True)
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "mod.py").write_text("VALUE = 1\n")
(repo / "scripts" / "tests" / "test_slow.py").write_text(
"import time\nfrom mod import VALUE\n\n\n"
"def test_slow():\n time.sleep(20)\n assert VALUE == 2\n"
)
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
(repo / "mod.py").write_text("VALUE = 2\n")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "fix: bump\n\nProves: scripts/tests/test_slow.py")
proc = subprocess.Popen(
["bash", str(PROVE_FIX), "--repo", str(repo), "HEAD"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd=str(repo),
)
time.sleep(4) # inside the control run, which sleeps 20s
assert proc.poll() is None, "the run finished before it could be signalled; test is void"
proc.terminate()
out, err = proc.communicate(timeout=60)
assert proc.returncode != 0, f"a signalled run must exit non-zero, got {proc.returncode}"
assert "PROVEN" not in out, f"a signalled run must not print a verdict:\n{out}"
def test_a_harness_failure_is_NOT_reported_as_PROVEN(fixrepo: Path, tmp_path: Path) -> None:
"""pytest must actually RUN for its status to mean anything.
`( cd X && pytest ); rc=$?` returns the SUBSHELL's status, so a failed cd or a failed
redirection also yields 1 — which "only exit 1 is red" would accept as a witnessed test
failure with pytest never having run. Cold review found that; the script now reads
pytest's status from a marker written only after it returns.
Simulated by making python3 unavailable to the run: the phase cannot produce a status,
so the script must refuse rather than infer.
"""
fake_bin = tmp_path / "bin"
fake_bin.mkdir()
(fake_bin / "python3").write_text("#!/bin/sh\nexit 127\n")
(fake_bin / "python3").chmod(0o755)
env = {**os.environ, "PATH": f"{fake_bin}:{os.environ['PATH']}"}
r = subprocess.run(
["bash", str(PROVE_FIX), "--repo", str(fixrepo), "HEAD", "scripts/tests/test_add.py"],
capture_output=True, text=True, cwd=str(fixrepo), env=env,
)
assert "PROVEN" not in r.stdout, (
f"a run where pytest could not execute must not produce a verdict:\n{r.stdout}\n{r.stderr}"
)
assert r.returncode != 0, f"expected a refusal, got {r.returncode}"