Files
ersatztv/scripts/tests/test_build_catalog_check_path.py
T
timothyandClaude Opus 5 b2b651779a fix(785): close the re-review's BLOCKER — the leak guard pinned the FIX, not the property
Third cold review, this one scoped to the previous two fix commits. Its central finding is
the one worth reading, because the guard it faults is the guard I added to stop exactly
this class of thing.

BLOCKER — `test_the_subprocess_env_CARRIES_the_isolated_hook_fire_log_dir` asserted on
`_env()`'s RETURN VALUE. `_env()` can be perfectly correct while a call site passes
something else, and that is the reintroduction a future tidy-up actually produces.
Demonstrated: leave `_env()` exactly as shipped, restore a module-level snapshot and point
one `env=` at it — 13 tests pass and 58 records leak. I reproduced it before acting.

The guard was pinned to the SHAPE OF THE FIX rather than to the property, which is
`verify-against-the-REAL-predecessor`: it was witnessed red against a hand-written revert,
so "pins it" and #809's "witnessed red by reintroducing the snapshot" were both overclaims.

`test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir` asserts the EFFECT instead —
drive a real hook through the real `_run()`, records must land in the fixture's dir — with
an anti-vacuity check that the hook reached a decision first. Measured: green at head, RED
under the call-site reintroduction that the old guard passed.

MEDIUM x3 — the workflow extractor was a text scanner and review broke it three ways in
one pass. A `run: |` block scalar was invisible, so reformatting the step to the form its
own sibling in that job already uses produced a FALSE RED whose message asserts a CI
regression that has not happened. A job or step switched off with `if: false` or
`continue-on-error: true` still read as wired — 7 green with the guard not gating at all.
And `run:` inside block-scalar TEXT was extracted and executed, so an `echo we no longer
run: PYTHONPATH=. python3 ... --check` left the subprocess proof green while the real step
was deleted. Two of the three are false greens.

`fixing-a-parser-bug-introduces-the-next-one` says the second round of a string predicate
is not the last. The structure is available — pyyaml is installed by the `script-tests` job
and four sibling suites already parse this workflow with it — so the scanner is replaced by
`yaml.safe_load`, and "would actually execute" is now three explicit conditions: the job
exists, the job is not disabled, the step is neither disabled nor allowed to fail. The six
ways it can stop gating are a parametrised test, and the block-scalar REFORMAT is a
negative control, because a detector that cannot tell a reformat from a removal trains its
readers to ignore it.

LOW — `_hook_calls` skipped whole-line comments but not inline ones, and broadening the
marker in the previous commit is what made that reachable: a trailing
`# ... .claude/hooks/removed-helper.sh` reddened a correct `pre-push`. The fix for one
over-match introduced another. Verified both directions.

NIT — a section header still said "three clauses" after the fourth mutation was added.

Docs. Two accuracy fixes, both verified against the files rather than reasoned about:
`test_hook_fire_log.py` proves THREE clauses (coverage, transparency, placement), not two,
and its row's Blocks column already claims two of them — the single `Proof ref` is a column
limit, not a coverage statement. And the "1,488 stray records" figure was unreproducible in
a durable doc, the same commit having deleted the evidence. Replaced with the reproduction
and a figure that survives it: 58 records per run, measured identically on macOS/git 2.55
and Linux/git 2.47.3.

Also corrects a count in 88832047b's own message: it says the inversion reddens "8 of 12";
that file had 13 tests at that commit. Third mislabelled figure in this branch, all three
in messages about mislabelled figures.

Refs: #785
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 21:29:00 +02:00

430 lines
20 KiB
Python

"""`scripts/build_decisions_catalog.py`'s `--check` path — the one CI actually runs.
`decisions-guard` (`.gitea/workflows/pr-checks.yml`, the "Active catalog in sync" step) runs
`python3 scripts/build_decisions_catalog.py --check`. `scripts/tests/test_build_catalog.py` covers
`render_catalog()` directly and never calls `main()` at all, so nothing there proves:
* that `main()`'s stale-detection comparison (`want.strip() != have.strip()`) is load-bearing —
a version that always agreed would pass every existing test;
* that the `if __name__ == "__main__": raise SystemExit(main())` wiring actually turns a stale
catalog into a non-zero process exit code, which is the only thing CI's `run:` step reads.
`testing.guard-ships-with-mutation-proof` (#775) is explicit that a guard is not tested because a
test *involving* it passes: it ships with a mutation proof — disarm the guard's own clause, alone,
and a named test must go red. This file is that proof for the catalog-guard, plus the subprocess
proof that the `__main__` wiring is connected (the #751/#719 shape the decision record names: a
green `main()` behind dead wiring).
"""
from __future__ import annotations
import importlib.util
import subprocess
import sys
from pathlib import Path
import pytest
import yaml
import scripts.build_decisions_catalog as bc
import scripts.decisions_lib as dl
REPO_ROOT = Path(__file__).resolve().parents[2]
CATALOG_SCRIPT = REPO_ROOT / "scripts" / "build_decisions_catalog.py"
DECISIONS_LIB = REPO_ROOT / "scripts" / "decisions_lib.py"
SCRIPTS_INIT = REPO_ROOT / "scripts" / "__init__.py"
def _current_catalog_text() -> str:
return bc.render_catalog(dl.all_active_records())
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY — if the real corpus is empty, every assertion below passes for nothing.
# ------------------------------------------------------------------------------------------------
def test_the_real_corpus_is_non_empty_and_renders_a_real_catalog():
records = dl.all_active_records()
active = [r for r in records if r.status == "active" and r.key]
assert active, "no active decision records were parsed — every test below would be vacuous"
text = _current_catalog_text()
assert bc.BANNER in text, "render_catalog produced no banner — not a real catalog document"
assert f"`{active[0].key}`" in text, (
"render_catalog produced no row for a known active record — not a real catalog document"
)
# ------------------------------------------------------------------------------------------------
# `main(["--check"])` — the comparison CI reads
# ------------------------------------------------------------------------------------------------
def test_check_returns_0_when_OUTPUT_matches_render_catalog(tmp_path, monkeypatch):
fresh = _current_catalog_text()
output = tmp_path / "README.md"
output.write_text(fresh.rstrip("\n") + "\n", encoding="utf-8")
monkeypatch.setattr(bc, "OUTPUT", output)
assert bc.main(["--check"]) == 0
def test_check_returns_1_when_OUTPUT_is_stale(tmp_path, monkeypatch):
fresh = _current_catalog_text()
output = tmp_path / "README.md"
# Append a line: the on-disk file no longer matches what render_catalog would produce.
output.write_text(fresh.rstrip("\n") + "\nEXTRA STALE LINE\n", encoding="utf-8")
monkeypatch.setattr(bc, "OUTPUT", output)
assert bc.main(["--check"]) == 1
def test_generate_then_check_round_trips(tmp_path, monkeypatch):
"""The no-argument path WRITES the catalog, and a subsequent --check must then pass.
This pins the property CI depends on: generate and check agree. If they ever diverged, `main([])`
would produce a file that `main(["--check"])` immediately rejects — a self-contradiction that
would make the generator useless for fixing the exact problem `--check` reports.
"""
output = tmp_path / "README.md"
assert not output.exists()
monkeypatch.setattr(bc, "OUTPUT", output)
assert bc.main([]) == 0
assert output.exists(), "main([]) with no --check must write OUTPUT"
written = output.read_text(encoding="utf-8")
assert written.strip() == _current_catalog_text().strip()
assert bc.main(["--check"]) == 0, "the file main([]) just wrote must satisfy main(['--check'])"
# ------------------------------------------------------------------------------------------------
# THE `__main__` WIRING — proof CI's subprocess invocation actually surfaces staleness
# ------------------------------------------------------------------------------------------------
def _seed_decisions_copy(root: Path) -> None:
"""Copy only what build_decisions_catalog.py + decisions_lib.py need to resolve a real corpus."""
(root / "scripts").mkdir(parents=True)
(root / "scripts" / "__init__.py").write_bytes(SCRIPTS_INIT.read_bytes())
(root / "scripts" / "build_decisions_catalog.py").write_bytes(CATALOG_SCRIPT.read_bytes())
(root / "scripts" / "decisions_lib.py").write_bytes(DECISIONS_LIB.read_bytes())
docs = root / "docs"
docs.mkdir()
(docs / "decisions.md").write_bytes((REPO_ROOT / "docs" / "decisions.md").read_bytes())
dst_decisions = docs / "decisions"
src_decisions = REPO_ROOT / "docs" / "decisions"
dst_decisions.mkdir()
for item in src_decisions.iterdir():
if item.is_dir():
_copy_tree(item, dst_decisions / item.name)
else:
(dst_decisions / item.name).write_bytes(item.read_bytes())
def _copy_tree(src: Path, dst: Path) -> None:
dst.mkdir(parents=True, exist_ok=True)
for item in src.rglob("*"):
rel = item.relative_to(src)
target = dst / rel
if item.is_dir():
target.mkdir(parents=True, exist_ok=True)
else:
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(item.read_bytes())
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "pr-checks.yml"
def _decisions_guard_run_lines(workflow: Path | None = None) -> list[str]:
"""Every command line the `decisions-guard` job would ACTUALLY execute.
Parsed with `yaml.safe_load`, not scanned as text. The first version of this scanned lines for
the literal `run:`, and cold review broke it three ways in one pass: a `run: |` block scalar was
invisible (its content lines contain no `run:` — and the sibling "Validate decision lifecycle"
step in this very job uses that form); a job or step switched off with `if: false` or
`continue-on-error: true` still read as wired; and `run:` appearing inside block-scalar *text*
(`echo we no longer run: ...`) was extracted and executed as though it were the step's command.
Two of those are false GREENS.
`fixing-a-parser-bug-introduces-the-next-one` says a string predicate over structured text needs
several rounds and then fails in a way nobody anticipated. The structure is available — `pyyaml`
is installed by the `script-tests` job and four sibling suites already parse this workflow with
it — so this reads the structure instead of guessing at it.
"Actually execute" is the operative phrase, and it is three separate conditions: the job exists,
the job is not disabled, and the step is neither disabled nor allowed to fail. A guard that is
present but switched off is the `guard-ships-with-mutation-proof` distinction between a guard
being WIRED and a guard RUNNING.
"""
doc = yaml.safe_load((workflow or WORKFLOW).read_text())
jobs = (doc or {}).get("jobs") or {}
job = jobs.get("decisions-guard")
assert job is not None, (
"no `decisions-guard` job in pr-checks.yml. Either it was renamed or it was removed — the "
f"second is the far more serious finding. Jobs present: {sorted(jobs)}"
)
assert not _disabled(job), (
"the `decisions-guard` job is disabled at the job level "
f"(if: {job.get('if')!r}, continue-on-error: {job.get('continue-on-error')!r}), so nothing "
"in it runs — including the catalog guard"
)
lines: list[str] = []
for step in job.get("steps") or []:
if _disabled(step):
continue
run = step.get("run")
if not run:
continue
lines.extend(ln.strip() for ln in str(run).splitlines() if ln.strip())
return lines
def _disabled(node: dict) -> bool:
"""A job or step that cannot fail the run: switched off, or allowed to fail.
`if:` is normally an expression string that is truthy at parse time (`github.event_name == ...`)
and only a literal false/`"false"` means "never". `continue-on-error: true` is the subtler one —
the step still runs and still reports, but its failure does not fail the job, so it is not a
gate.
"""
cond = node.get("if", True)
if cond is False or (isinstance(cond, str) and cond.strip().lower() == "false"):
return True
coe = node.get("continue-on-error", False)
return coe is True or (isinstance(coe, str) and coe.strip().lower() == "true")
def _ci_check_command(workflow: Path | None = None) -> list[str]:
"""The command the `decisions-guard` job runs, DERIVED from the workflow, not restated here.
`testing.guard-derives-population-from-source`: a hand-copied command is a second copy of the
workflow that drifts silently, and this test's entire value is that it runs what CI runs.
The match requires the line to BEGIN with `PYTHONPATH=.` rather than merely to contain the
script name, because a block scalar's text can mention the command without invoking it — an
`echo we no longer run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check` was
extracted and executed by the previous version, which is a guard proving a command nobody runs.
"""
runs = _decisions_guard_run_lines(workflow)
matches = [r for r in runs if r.startswith("PYTHONPATH=.") and "build_decisions_catalog.py" in r and "--check" in r]
assert matches, (
"the `decisions-guard` job has no ACTIVE step running "
"`build_decisions_catalog.py --check`. The catalog guard has stopped running in CI — that "
f"is the finding, not this test's failure. Active command lines in that job: {runs}"
)
assert len(matches) == 1, f"expected exactly one such command, found {matches}"
return ["bash", "-c", matches[0].replace("python3", sys.executable, 1)]
@pytest.mark.parametrize(
"label,mutate",
[
(
"the step is COMMENTED OUT",
lambda s: s.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
" # - name: Active catalog in sync\n"
" # run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
1,
),
),
(
"the step is DELETED",
lambda s: s.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
"",
1,
),
),
(
"the step is switched off with `if: false`",
lambda s: s.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
" - name: Active catalog in sync\n"
" if: false\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
1,
),
),
(
"the step is allowed to fail with `continue-on-error: true`",
lambda s: s.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
" - name: Active catalog in sync\n"
" continue-on-error: true\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
1,
),
),
(
"the whole JOB is switched off with `if: false`",
lambda s: s.replace(
" decisions-guard:\n name: decisions lifecycle\n runs-on: small\n"
" if: github.event_name == 'pull_request'\n",
" decisions-guard:\n name: decisions lifecycle\n runs-on: small\n if: false\n",
1,
),
),
(
"the command survives only as TEXT inside another step's block scalar",
lambda s: s.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
" - name: Note\n"
" run: |\n"
" echo we no longer run: PYTHONPATH=. python3 "
"scripts/build_decisions_catalog.py --check\n",
1,
),
),
],
)
def test_a_guard_that_stopped_RUNNING_is_DETECTED(tmp_path, label, mutate):
"""Six ways the catalog guard can stop gating, each of which must be caught.
Commenting out is one of them and was the only one the first version detected. `if: false`,
`continue-on-error: true` and a job-level `if: false` all leave the text in place, and a mention
inside another step's block scalar leaves it in place AND parseable — that one had the extractor
executing a command out of an `echo`.
"""
raw = WORKFLOW.read_text()
mutated = mutate(raw)
assert mutated != raw, f"the mutation for {label!r} matched nothing; RETARGET it"
alt = tmp_path / "pr-checks.yml"
alt.write_text(mutated)
try:
_ci_check_command(alt)
except AssertionError:
return
raise AssertionError(
f"the extractor still reported the catalog guard as running when {label}. The subprocess "
"proof below would stay green over a guard CI does not gate on."
)
def test_the_extractor_accepts_the_step_in_BLOCK_SCALAR_form(tmp_path):
"""The negative control for the six above: a legitimate reformatting must NOT be flagged.
`run: |` is idiomatic in this very job (the sibling "Validate decision lifecycle" step uses it),
and the text-scanning version reported that form as "the guard has stopped running in CI" — a
false red whose message asserts a serious regression that has not happened. A detector that
cannot tell a reformat from a removal trains its readers to ignore it.
"""
raw = WORKFLOW.read_text()
mutated = raw.replace(
" - name: Active catalog in sync\n"
" run: PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
" - name: Active catalog in sync\n"
" run: |\n"
" PYTHONPATH=. python3 scripts/build_decisions_catalog.py --check\n",
1,
)
assert mutated != raw, "retarget this reformatting; the step's text has changed"
alt = tmp_path / "pr-checks.yml"
alt.write_text(mutated)
cmd = _ci_check_command(alt)
assert "build_decisions_catalog.py" in cmd[-1], cmd
def test_CLI_subprocess_exits_nonzero_on_a_stale_catalog(tmp_path):
"""The real CI invocation, as a subprocess, against a real corpus.
This is the `__main__` → `SystemExit(main())` wiring proof: `main()` returning 1 is worthless if
the process still exits 0, and the workflow step reads nothing but the exit code. It is the
#751/#719 shape the decision record names — a green result behind wiring that is not connected.
The FRESH half is not optional decoration. Without it this test passes whenever the subprocess
dies for any reason at all — an import error, a missing file in the copy, a syntax error — none
of which is the guard detecting anything. `arbitrary-sample-gives-false-negatives` in reverse:
a non-zero exit is only evidence when the same harness is shown to exit zero on a clean corpus.
"""
repo = tmp_path / "repo-copy"
repo.mkdir()
_seed_decisions_copy(repo)
cmd = _ci_check_command()
fresh = subprocess.run(cmd, cwd=str(repo), capture_output=True, text=True, timeout=60)
assert fresh.returncode == 0, (
"the copied corpus does not even pass --check when untouched, so a non-zero exit below "
f"would be the harness failing rather than the guard firing: {fresh.stdout!r} {fresh.stderr!r}"
)
readme = repo / "docs" / "decisions" / "README.md"
readme.write_text(readme.read_text(encoding="utf-8") + "\nSTALE INJECTED LINE\n", encoding="utf-8")
stale = subprocess.run(cmd, cwd=str(repo), capture_output=True, text=True, timeout=60)
assert stale.returncode != 0, (
"the CLI wiring did not surface a stale catalog as a non-zero exit — CI would report "
f"green over a stale README.md: stdout={stale.stdout!r} stderr={stale.stderr!r}"
)
# ------------------------------------------------------------------------------------------------
# THE MUTATION PROOF — disarm `main()`'s stale-detection clause alone, --check must stop detecting
# ------------------------------------------------------------------------------------------------
def _load_mutated_module(tmp_path: Path, mutated_source: str):
mutated_path = tmp_path / "mutated_build_decisions_catalog.py"
mutated_path.write_text(mutated_source, encoding="utf-8")
spec = importlib.util.spec_from_file_location("mutated_build_decisions_catalog", mutated_path)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_MUTATION_disarming_the_stale_comparison_stops_detection(tmp_path, monkeypatch):
"""`testing.guard-ships-with-mutation-proof` (#775).
`main()`'s ONLY stale-detection logic is the clause `want.strip() != have.strip()`. Disarm that
clause alone — replace it with the constant `False` in an isolated copy of the module, changing
nothing else — and `--check` must stop reporting staleness. If it still returns 1, the deny is
coming from somewhere other than the clause the guard is supposed to hang on, and every test
above proves nothing about it.
"""
text = CATALOG_SCRIPT.read_text(encoding="utf-8")
clause = "want.strip() != have.strip()"
assert clause in text, (
"the stale-detection clause has moved or been reworded; RETARGET this mutation at its new "
"location rather than loosening the string match — a mutation that silently stops mutating "
"is the exact failure this file exists to catch"
)
mutated_source = text.replace(clause, "False", 1)
assert mutated_source != text and clause not in mutated_source, (
"the replacement did not change the source, so the mutant is the subject"
)
fresh = _current_catalog_text()
output = tmp_path / "README.md"
output.write_text(fresh.rstrip("\n") + "\nEXTRA STALE LINE\n", encoding="utf-8")
# POSITIVE CONTROL FIRST. Without this, the mutation assertion below would pass just as well if
# `main(["--check"])` never detected anything at all on this input — "the mutant is silent"
# proves nothing unless the real guard is first shown to be loud on the exact same input.
monkeypatch.setattr(bc, "OUTPUT", output)
assert bc.main(["--check"]) == 1, (
"the UNMUTATED guard did not detect the staleness on this input, so a silent mutant below "
"would prove nothing about the clause"
)
mutant = _load_mutated_module(tmp_path, mutated_source)
mutant.OUTPUT = output
assert mutant.main(["--check"]) == 0, (
"disarming `want.strip() != have.strip()` alone did not stop --check from reporting "
"staleness, so that clause is not what the guard's exit code hangs on"
)