From 6e79c155f1997c8539e7edadc26d57555b15ca86 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 16 Aug 2026 11:49:26 +0200 Subject: [PATCH] fix(794): the multi-trailer guard missed the path that actually enforces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cold review of the squashed head: 1 High, 1 Medium, 2 Low, 2 Nits. It confirmed by measurement the claims that matter — both discrimination tests go RED against 587edbecc (and the old script prints "red without it (pytest exit 2)", a witnessed false PROVEN), the MUTATION grade is earned, 681 passed / 2 skipped, and the decision to omit this branch's own `Proves:` trailer is sound rather than a dodge. THE HIGH IS MY OWN RECORDED LESSON, VIOLATED. `fix-one-path-then-check-its-twin`: I made a second `Proves:` trailer fatal in prove-fix.sh, but that guard only runs when the script reads the trailer ITSELF — and the CI job does `head -1` and passes the selector explicitly, so the guard was bypassed on the ONE path that enforces. Measured on a two-trailer commit: the explicit-selector path returns PROVEN rc=0 with the second claim never run. The job now refuses multi-trailer commits before it gets there, and I reproduced the refusal. That also makes the previous commit message's "a second `Proves:` trailer was silently ignored, now fatal" FALSE as it stood — true of the script, not of CI. Correcting it here rather than rewriting the pushed commit. Same correction for the retracted SIGTERM claim: the record was fixed last round but the sentence survived in TWO other copies (the script docstring and the test docstring), where the test docstring was contradicted by its own inline comment thirty lines below. Swept by subject this time, not by the one file I happened to be editing — `stale-comments-sweep-by-subject-not-verb`. Also: the previous message said "15 tests"; there are 14. Small, but that message is otherwise an evidence record and an unscoped figure in one is how `state-the-invariant-not-the-measurement` starts. docs/ci-cd.md now documents the job — it was missing from the `small`-lane enumeration (a resource-safety list, and this is the lane's heaviest member: two `git worktree add`s plus two pytest runs per claimed commit) and had no per-job section while every other PR gate does. Docs-update is part of done, and I had skipped it. Nits: duplicated comment block in the test file; inventory row sorted. Verified: two-trailer refusal reproduced on the CI logic; 14 tests; scripts/tests 681 passed, 2 skipped; decisions-validate OK. Co-Authored-By: Claude Opus 5 (1M context) --- .gitea/workflows/pr-checks.yml | 12 ++++++++++++ docs/ci-cd.md | 33 ++++++++++++++++++++++++++++++++- docs/guard-inventory.md | 2 +- scripts/prove-fix.sh | 5 +++-- scripts/tests/test_prove_fix.py | 11 ++++------- 5 files changed, 52 insertions(+), 11 deletions(-) diff --git a/.gitea/workflows/pr-checks.yml b/.gitea/workflows/pr-checks.yml index 906b5d145..7c2366a11 100644 --- a/.gitea/workflows/pr-checks.yml +++ b/.gitea/workflows/pr-checks.yml @@ -266,6 +266,18 @@ jobs: "job cannot assert anything about it. Refusing to pass." exit 1 fi + # Refuse MORE THAN ONE `Proves:` here too. prove-fix.sh has this guard, but it + # only fires when it reads the trailer itself — and this job passes the selector + # explicitly, so the guard was bypassed on the one path that actually enforces. + # Measured: a commit with two trailers reported PROVEN while the second was never + # run. Fixing the script and not its twin is how a guard reads as coverage. + if [ "$(printf '%s\n' "$raw" | grep -c .)" -gt 1 ]; then + claimed=$((claimed + 1)); failed=$((failed + 1)) + echo "::error::commit $sha carries more than one 'Proves:' trailer; only the" \ + "first would be checked, so the rest would read as proven without ever" \ + "running. Use a single selector." + continue + fi sel="$(printf '%s\n' "$raw" | head -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" [ -n "$sel" ] || continue claimed=$((claimed + 1)) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index d3b38eda7..6d2cf5d33 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -200,7 +200,10 @@ rather than in `docker-build.yml` — see that section (ersatztv#535). **`small` is git-only, and that is load-bearing (server-management#639).** Everything in the lane is a checkout plus a `git diff`: `decisions-guard`, `ci-image-pin`, -`docs-reminder` — plus `script-tests`, which is a checkout plus a `pytest` run needing only +`docs-reminder` — plus `prove-fix` (ersatztv#794), the heaviest member: per commit carrying a +`Proves:` trailer it makes two `git worktree add`s and runs an arbitrary pytest selection twice, +so a PR claiming many proofs costs proportionally more than the rest of the lane combined — plus +`script-tests`, which is a checkout plus a `pytest` run needing only `pytest` and `pyyaml` (ersatztv#631; it is NOT stdlib-only — that assumption is what turned the job red on its first CI run, see below) — plus **`scan`** (ersatztv#767), the same lightweight-Python shape. `scan` is the lane member to think hardest about before changing anything @@ -871,6 +874,34 @@ compiler/docker build), so it doesn't violate the "small is git-only" lane rule. `docs-reminder`, otherwise a seconds-long `git diff` + parse with no dotnet/node setup (`runs-on: small`). +### `prove-fix` job (`Fix proofs (Proves trailers)`, PR-only — in `pr-checks.yml`) + +Runs `scripts/prove-fix.sh` for **every commit in the PR that carries a `Proves: ` +trailer**, and fails the PR if a claimed proof does not hold. The rule and its rationale are +`testing.fix-ships-a-witnessed-red-test`; this section is the CI-side contract. + +**Opt-in by trailer, enforced when present.** Requiring `Proves:` on every commit would block docs, +CI and refactor commits that have no code side to revert, and a gate that blocks ordinary work gets +switched off — which is how a check ends up running nowhere (ersatztv#631). So the trailer is the +author's *claim* and this job checks claims. **When a PR carries none, the job emits a `::notice::` +saying it asserted nothing** — its green must not be read as fix coverage. + +**It needs full history** (`fetch-depth: 0`): the script reverts each commit against its PARENT, and +a shallow clone cannot resolve `^`. + +Three refusals worth knowing before you write a trailer: +- a **merge** commit is rejected — several parents means "the code before this change" is ambiguous; + put the trailer on the commit carrying the fix; +- **more than one** `Proves:` trailer is rejected — only the first would be checked, so the rest + would read as proven without ever running (the job checks this itself, because passing the + selector explicitly bypasses the script's own guard); +- a **test-only** commit is rejected — there is no code side to revert. + +**Only pytest exit 1 counts as red.** 2 is an interrupted collection, 3 internal, 4 usage, 5 nothing +collected, 143 a signal; `--continue-on-collection-errors` converts a genuine collection failure to +1 so add-a-file fixes stay provable. The job inherits that, so a red here means a claim failed, not +that the runner wobbled. + ### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`) > Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the diff --git a/docs/guard-inventory.md b/docs/guard-inventory.md index 747bb0f11..87ba95f98 100644 --- a/docs/guard-inventory.md +++ b/docs/guard-inventory.md @@ -104,8 +104,8 @@ all. | `scripts/hook-fire-log.sh` | nothing (records that each hook fired, and reports it) | TOOLING | NONE | — | | `scripts/jq-preflight.sh` | the `script-tests` job, on a jq version change | GUARD | BEHAVIOUR-ONLY | `test_jq_preflight.py::test_below_the_floor_is_LOUD` | | `scripts/post-review-verdict.sh` | nothing (writes the verdict status) | GUARD | BEHAVIOUR-ONLY | `test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head` | -| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` | | `scripts/pr-changed-files.sh` | the verdict exemption, on an incomplete enumeration | GUARD | BEHAVIOUR-ONLY | `test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration` | +| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` | | `scripts/update-openapi.sh` | nothing (regenerates the spec) | TOOLING | NONE | — | | `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — | | `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — | diff --git a/scripts/prove-fix.sh b/scripts/prove-fix.sh index e3ae20509..aa1ab0c41 100755 --- a/scripts/prove-fix.sh +++ b/scripts/prove-fix.sh @@ -15,8 +15,9 @@ # ONLY pytest exit 1 COUNTS AS RED, and that is the whole safety argument. "Non-zero" is # not "the test failed": 2 is an interrupted collection, 3 an internal error, 4 a usage # error, 5 nothing collected, and a killed run gives 143. Cold review measured a SIGTERM -# producing PROVEN from the first version of this script — cancellation masquerading as -# evidence. `--continue-on-collection-errors` is passed so a genuine collection failure +# being read as red in an early DRAFT — cancellation masquerading as evidence. On the first +# COMMITTED version the reproducible false PROVEN is the marker case (a failed `cd` giving +# the subshell status 1), not the signal one; see the decision record. `--continue-on-collection-errors` is passed so a genuine collection failure # (what happens when the fix ADDED the module the test imports) is reported as a test # error and exits 1 instead of vanishing into the ambiguous 2/3 band. Measured: pass 0, # fail 1, collection-error 2 (1 with the flag), SIGTERM 143. diff --git a/scripts/tests/test_prove_fix.py b/scripts/tests/test_prove_fix.py index c3674c5d5..a29954631 100644 --- a/scripts/tests/test_prove_fix.py +++ b/scripts/tests/test_prove_fix.py @@ -21,9 +21,6 @@ from pathlib import Path import pytest REPO_ROOT = Path(__file__).resolve().parents[2] -# Overridable so a MUTANT copy can be driven through these very tests. That is what makes -# the mutation proof below a witnessed red rather than an argument: the unchanged test is -# re-run against the mutant and observed to fail. # Overridable so a MUTANT copy can be driven through these very tests — that is what makes # the mutation proof below a witnessed red rather than an argument. Gated behind a sentinel # so a stray CI value cannot silently point the whole suite at another script. @@ -273,10 +270,10 @@ def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test( def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None: """A killed run must not look like evidence. - Cold review measured the first version of this script printing PROVEN and exiting 0 - after a SIGTERM: the trap cleaned up but did not exit, so the previous status stood. - Cancellation masquerading as proof is the exact failure this tool exists to prevent, - so the signal path gets its own test rather than an argument. + An early DRAFT printed PROVEN and exited 0 after a SIGTERM: the trap cleaned up but did + not exit, so the previous status stood. On the first COMMITTED version (587edbecc) the + run reaches rc 5 by a different route entirely, so this test earns its keep only via the + assertions below — see the comment there. """ repo = tmp_path / "slow" (repo / "scripts" / "tests").mkdir(parents=True)