diff --git a/.gitea/workflows/pr-checks.yml b/.gitea/workflows/pr-checks.yml index 906b5d145..7c2366a11 100644 --- a/.gitea/workflows/pr-checks.yml +++ b/.gitea/workflows/pr-checks.yml @@ -266,6 +266,18 @@ jobs: "job cannot assert anything about it. Refusing to pass." exit 1 fi + # Refuse MORE THAN ONE `Proves:` here too. prove-fix.sh has this guard, but it + # only fires when it reads the trailer itself — and this job passes the selector + # explicitly, so the guard was bypassed on the one path that actually enforces. + # Measured: a commit with two trailers reported PROVEN while the second was never + # run. Fixing the script and not its twin is how a guard reads as coverage. + if [ "$(printf '%s\n' "$raw" | grep -c .)" -gt 1 ]; then + claimed=$((claimed + 1)); failed=$((failed + 1)) + echo "::error::commit $sha carries more than one 'Proves:' trailer; only the" \ + "first would be checked, so the rest would read as proven without ever" \ + "running. Use a single selector." + continue + fi sel="$(printf '%s\n' "$raw" | head -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" [ -n "$sel" ] || continue claimed=$((claimed + 1)) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index d3b38eda7..6d2cf5d33 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -200,7 +200,10 @@ rather than in `docker-build.yml` — see that section (ersatztv#535). **`small` is git-only, and that is load-bearing (server-management#639).** Everything in the lane is a checkout plus a `git diff`: `decisions-guard`, `ci-image-pin`, -`docs-reminder` — plus `script-tests`, which is a checkout plus a `pytest` run needing only +`docs-reminder` — plus `prove-fix` (ersatztv#794), the heaviest member: per commit carrying a +`Proves:` trailer it makes two `git worktree add`s and runs an arbitrary pytest selection twice, +so a PR claiming many proofs costs proportionally more than the rest of the lane combined — plus +`script-tests`, which is a checkout plus a `pytest` run needing only `pytest` and `pyyaml` (ersatztv#631; it is NOT stdlib-only — that assumption is what turned the job red on its first CI run, see below) — plus **`scan`** (ersatztv#767), the same lightweight-Python shape. `scan` is the lane member to think hardest about before changing anything @@ -871,6 +874,34 @@ compiler/docker build), so it doesn't violate the "small is git-only" lane rule. `docs-reminder`, otherwise a seconds-long `git diff` + parse with no dotnet/node setup (`runs-on: small`). +### `prove-fix` job (`Fix proofs (Proves trailers)`, PR-only — in `pr-checks.yml`) + +Runs `scripts/prove-fix.sh` for **every commit in the PR that carries a `Proves: ` +trailer**, and fails the PR if a claimed proof does not hold. The rule and its rationale are +`testing.fix-ships-a-witnessed-red-test`; this section is the CI-side contract. + +**Opt-in by trailer, enforced when present.** Requiring `Proves:` on every commit would block docs, +CI and refactor commits that have no code side to revert, and a gate that blocks ordinary work gets +switched off — which is how a check ends up running nowhere (ersatztv#631). So the trailer is the +author's *claim* and this job checks claims. **When a PR carries none, the job emits a `::notice::` +saying it asserted nothing** — its green must not be read as fix coverage. + +**It needs full history** (`fetch-depth: 0`): the script reverts each commit against its PARENT, and +a shallow clone cannot resolve `^`. + +Three refusals worth knowing before you write a trailer: +- a **merge** commit is rejected — several parents means "the code before this change" is ambiguous; + put the trailer on the commit carrying the fix; +- **more than one** `Proves:` trailer is rejected — only the first would be checked, so the rest + would read as proven without ever running (the job checks this itself, because passing the + selector explicitly bypasses the script's own guard); +- a **test-only** commit is rejected — there is no code side to revert. + +**Only pytest exit 1 counts as red.** 2 is an interrupted collection, 3 internal, 4 usage, 5 nothing +collected, 143 a signal; `--continue-on-collection-errors` converts a genuine collection failure to +1 so add-a-file fixes stay provable. The job inherits that, so a red here means a claim failed, not +that the runner wobbled. + ### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`) > Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the diff --git a/docs/guard-inventory.md b/docs/guard-inventory.md index 747bb0f11..87ba95f98 100644 --- a/docs/guard-inventory.md +++ b/docs/guard-inventory.md @@ -104,8 +104,8 @@ all. | `scripts/hook-fire-log.sh` | nothing (records that each hook fired, and reports it) | TOOLING | NONE | — | | `scripts/jq-preflight.sh` | the `script-tests` job, on a jq version change | GUARD | BEHAVIOUR-ONLY | `test_jq_preflight.py::test_below_the_floor_is_LOUD` | | `scripts/post-review-verdict.sh` | nothing (writes the verdict status) | GUARD | BEHAVIOUR-ONLY | `test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head` | -| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` | | `scripts/pr-changed-files.sh` | the verdict exemption, on an incomplete enumeration | GUARD | BEHAVIOUR-ONLY | `test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration` | +| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` | | `scripts/update-openapi.sh` | nothing (regenerates the spec) | TOOLING | NONE | — | | `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — | | `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — | diff --git a/scripts/prove-fix.sh b/scripts/prove-fix.sh index e3ae20509..aa1ab0c41 100755 --- a/scripts/prove-fix.sh +++ b/scripts/prove-fix.sh @@ -15,8 +15,9 @@ # ONLY pytest exit 1 COUNTS AS RED, and that is the whole safety argument. "Non-zero" is # not "the test failed": 2 is an interrupted collection, 3 an internal error, 4 a usage # error, 5 nothing collected, and a killed run gives 143. Cold review measured a SIGTERM -# producing PROVEN from the first version of this script — cancellation masquerading as -# evidence. `--continue-on-collection-errors` is passed so a genuine collection failure +# being read as red in an early DRAFT — cancellation masquerading as evidence. On the first +# COMMITTED version the reproducible false PROVEN is the marker case (a failed `cd` giving +# the subshell status 1), not the signal one; see the decision record. `--continue-on-collection-errors` is passed so a genuine collection failure # (what happens when the fix ADDED the module the test imports) is reported as a test # error and exits 1 instead of vanishing into the ambiguous 2/3 band. Measured: pass 0, # fail 1, collection-error 2 (1 with the flag), SIGTERM 143. diff --git a/scripts/tests/test_prove_fix.py b/scripts/tests/test_prove_fix.py index c3674c5d5..a29954631 100644 --- a/scripts/tests/test_prove_fix.py +++ b/scripts/tests/test_prove_fix.py @@ -21,9 +21,6 @@ from pathlib import Path import pytest REPO_ROOT = Path(__file__).resolve().parents[2] -# Overridable so a MUTANT copy can be driven through these very tests. That is what makes -# the mutation proof below a witnessed red rather than an argument: the unchanged test is -# re-run against the mutant and observed to fail. # Overridable so a MUTANT copy can be driven through these very tests — that is what makes # the mutation proof below a witnessed red rather than an argument. Gated behind a sentinel # so a stray CI value cannot silently point the whole suite at another script. @@ -273,10 +270,10 @@ def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test( def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None: """A killed run must not look like evidence. - Cold review measured the first version of this script printing PROVEN and exiting 0 - after a SIGTERM: the trap cleaned up but did not exit, so the previous status stood. - Cancellation masquerading as proof is the exact failure this tool exists to prevent, - so the signal path gets its own test rather than an argument. + An early DRAFT printed PROVEN and exited 0 after a SIGTERM: the trap cleaned up but did + not exit, so the previous status stood. On the first COMMITTED version (587edbecc) the + run reaches rc 5 by a different route entirely, so this test earns its keep only via the + assertions below — see the comment there. """ repo = tmp_path / "slow" (repo / "scripts" / "tests").mkdir(parents=True)