fix(794): the multi-trailer guard missed the path that actually enforces
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 8s
PR Gates / Docs update reminder (pull_request) Successful in 10s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 20s
PR Gates / decisions lifecycle (pull_request) Successful in 26s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 21s
review-verdict/h10 Awaiting review verdict for 6e79c15
Review verdict / Set review-verdict status (pull_request_target) Successful in 14s
PR Gates / Script tests (pytest) (pull_request) Successful in 3m44s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m23s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Canceled after 1m37s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 0s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Canceled after 0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s

Cold review of the squashed head: 1 High, 1 Medium, 2 Low, 2 Nits. It confirmed by
measurement the claims that matter — both discrimination tests go RED against
587edbecc (and the old script prints "red without it (pytest exit 2)", a witnessed
false PROVEN), the MUTATION grade is earned, 681 passed / 2 skipped, and the decision
to omit this branch's own `Proves:` trailer is sound rather than a dodge.

THE HIGH IS MY OWN RECORDED LESSON, VIOLATED. `fix-one-path-then-check-its-twin`:
I made a second `Proves:` trailer fatal in prove-fix.sh, but that guard only runs when
the script reads the trailer ITSELF — and the CI job does `head -1` and passes the
selector explicitly, so the guard was bypassed on the ONE path that enforces. Measured
on a two-trailer commit: the explicit-selector path returns PROVEN rc=0 with the second
claim never run. The job now refuses multi-trailer commits before it gets there, and
I reproduced the refusal.

That also makes the previous commit message's "a second `Proves:` trailer was silently
ignored, now fatal" FALSE as it stood — true of the script, not of CI. Correcting it
here rather than rewriting the pushed commit.

Same correction for the retracted SIGTERM claim: the record was fixed last round but
the sentence survived in TWO other copies (the script docstring and the test
docstring), where the test docstring was contradicted by its own inline comment thirty
lines below. Swept by subject this time, not by the one file I happened to be editing —
`stale-comments-sweep-by-subject-not-verb`.

Also: the previous message said "15 tests"; there are 14. Small, but that message is
otherwise an evidence record and an unscoped figure in one is how
`state-the-invariant-not-the-measurement` starts.

docs/ci-cd.md now documents the job — it was missing from the `small`-lane enumeration
(a resource-safety list, and this is the lane's heaviest member: two `git worktree add`s
plus two pytest runs per claimed commit) and had no per-job section while every other PR
gate does. Docs-update is part of done, and I had skipped it.

Nits: duplicated comment block in the test file; inventory row sorted.

Verified: two-trailer refusal reproduced on the CI logic; 14 tests; scripts/tests 681
passed, 2 skipped; decisions-validate OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-16 11:49:26 +02:00
co-authored by Claude Opus 5
parent 4f41134349
commit 6e79c155f1
5 changed files with 52 additions and 11 deletions
+12
View File
@@ -266,6 +266,18 @@ jobs:
"job cannot assert anything about it. Refusing to pass."
exit 1
fi
# Refuse MORE THAN ONE `Proves:` here too. prove-fix.sh has this guard, but it
# only fires when it reads the trailer itself — and this job passes the selector
# explicitly, so the guard was bypassed on the one path that actually enforces.
# Measured: a commit with two trailers reported PROVEN while the second was never
# run. Fixing the script and not its twin is how a guard reads as coverage.
if [ "$(printf '%s\n' "$raw" | grep -c .)" -gt 1 ]; then
claimed=$((claimed + 1)); failed=$((failed + 1))
echo "::error::commit $sha carries more than one 'Proves:' trailer; only the" \
"first would be checked, so the rest would read as proven without ever" \
"running. Use a single selector."
continue
fi
sel="$(printf '%s\n' "$raw" | head -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
[ -n "$sel" ] || continue
claimed=$((claimed + 1))
+32 -1
View File
@@ -200,7 +200,10 @@ rather than in `docker-build.yml` — see that section (ersatztv#535).
**`small` is git-only, and that is load-bearing (server-management#639).** Everything in
the lane is a checkout plus a `git diff`: `decisions-guard`, `ci-image-pin`,
`docs-reminder` — plus `script-tests`, which is a checkout plus a `pytest` run needing only
`docs-reminder` — plus `prove-fix` (ersatztv#794), the heaviest member: per commit carrying a
`Proves:` trailer it makes two `git worktree add`s and runs an arbitrary pytest selection twice,
so a PR claiming many proofs costs proportionally more than the rest of the lane combined — plus
`script-tests`, which is a checkout plus a `pytest` run needing only
`pytest` and `pyyaml` (ersatztv#631; it is NOT stdlib-only — that assumption is what turned the
job red on its first CI run, see below) — plus **`scan`** (ersatztv#767), the same
lightweight-Python shape. `scan` is the lane member to think hardest about before changing anything
@@ -871,6 +874,34 @@ compiler/docker build), so it doesn't violate the "small is git-only" lane rule.
`docs-reminder`, otherwise a seconds-long `git diff` + parse with no dotnet/node setup
(`runs-on: small`).
### `prove-fix` job (`Fix proofs (Proves trailers)`, PR-only — in `pr-checks.yml`)
Runs `scripts/prove-fix.sh` for **every commit in the PR that carries a `Proves: <pytest selector>`
trailer**, and fails the PR if a claimed proof does not hold. The rule and its rationale are
`testing.fix-ships-a-witnessed-red-test`; this section is the CI-side contract.
**Opt-in by trailer, enforced when present.** Requiring `Proves:` on every commit would block docs,
CI and refactor commits that have no code side to revert, and a gate that blocks ordinary work gets
switched off — which is how a check ends up running nowhere (ersatztv#631). So the trailer is the
author's *claim* and this job checks claims. **When a PR carries none, the job emits a `::notice::`
saying it asserted nothing** — its green must not be read as fix coverage.
**It needs full history** (`fetch-depth: 0`): the script reverts each commit against its PARENT, and
a shallow clone cannot resolve `<sha>^`.
Three refusals worth knowing before you write a trailer:
- a **merge** commit is rejected — several parents means "the code before this change" is ambiguous;
put the trailer on the commit carrying the fix;
- **more than one** `Proves:` trailer is rejected — only the first would be checked, so the rest
would read as proven without ever running (the job checks this itself, because passing the
selector explicitly bypasses the script's own guard);
- a **test-only** commit is rejected — there is no code side to revert.
**Only pytest exit 1 counts as red.** 2 is an interrupted collection, 3 internal, 4 usage, 5 nothing
collected, 143 a signal; `--continue-on-collection-errors` converts a genuine collection failure to
1 so add-a-file fixes stay provable. The job inherits that, so a red here means a claim failed, not
that the runner wobbled.
### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`)
> Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the
+1 -1
View File
@@ -104,8 +104,8 @@ all.
| `scripts/hook-fire-log.sh` | nothing (records that each hook fired, and reports it) | TOOLING | NONE | — |
| `scripts/jq-preflight.sh` | the `script-tests` job, on a jq version change | GUARD | BEHAVIOUR-ONLY | `test_jq_preflight.py::test_below_the_floor_is_LOUD` |
| `scripts/post-review-verdict.sh` | nothing (writes the verdict status) | GUARD | BEHAVIOUR-ONLY | `test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head` |
| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` |
| `scripts/pr-changed-files.sh` | the verdict exemption, on an incomplete enumeration | GUARD | BEHAVIOUR-ONLY | `test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration` |
| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` |
| `scripts/update-openapi.sh` | nothing (regenerates the spec) | TOOLING | NONE | — |
| `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — |
+3 -2
View File
@@ -15,8 +15,9 @@
# ONLY pytest exit 1 COUNTS AS RED, and that is the whole safety argument. "Non-zero" is
# not "the test failed": 2 is an interrupted collection, 3 an internal error, 4 a usage
# error, 5 nothing collected, and a killed run gives 143. Cold review measured a SIGTERM
# producing PROVEN from the first version of this script — cancellation masquerading as
# evidence. `--continue-on-collection-errors` is passed so a genuine collection failure
# being read as red in an early DRAFT — cancellation masquerading as evidence. On the first
# COMMITTED version the reproducible false PROVEN is the marker case (a failed `cd` giving
# the subshell status 1), not the signal one; see the decision record. `--continue-on-collection-errors` is passed so a genuine collection failure
# (what happens when the fix ADDED the module the test imports) is reported as a test
# error and exits 1 instead of vanishing into the ambiguous 2/3 band. Measured: pass 0,
# fail 1, collection-error 2 (1 with the flag), SIGTERM 143.
+4 -7
View File
@@ -21,9 +21,6 @@ from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
# Overridable so a MUTANT copy can be driven through these very tests. That is what makes
# the mutation proof below a witnessed red rather than an argument: the unchanged test is
# re-run against the mutant and observed to fail.
# Overridable so a MUTANT copy can be driven through these very tests — that is what makes
# the mutation proof below a witnessed red rather than an argument. Gated behind a sentinel
# so a stray CI value cannot silently point the whole suite at another script.
@@ -273,10 +270,10 @@ def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test(
def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
"""A killed run must not look like evidence.
Cold review measured the first version of this script printing PROVEN and exiting 0
after a SIGTERM: the trap cleaned up but did not exit, so the previous status stood.
Cancellation masquerading as proof is the exact failure this tool exists to prevent,
so the signal path gets its own test rather than an argument.
An early DRAFT printed PROVEN and exited 0 after a SIGTERM: the trap cleaned up but did
not exit, so the previous status stood. On the first COMMITTED version (587edbecc) the
run reaches rc 5 by a different route entirely, so this test earns its keep only via the
assertions below — see the comment there.
"""
repo = tmp_path / "slow"
(repo / "scripts" / "tests").mkdir(parents=True)