From 4f41134349ceba7c5abc525ba76ac77c7c3b1973 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 16 Aug 2026 11:32:08 +0200 Subject: [PATCH 1/3] feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mechanises the defect that took #776 and #793 six review rounds each: a fix's test written to confirm the fix, not to discriminate against its absence. testing.guard-ships-with-mutation-proof generalised from guards to fixes. Closes #794. `scripts/prove-fix.sh [selector]` checks a RED-GREEN PAIR — the selector must pass at the commit (control) and fail with the commit's non-test files reverted, in a SEPARATE fresh worktree. RED means pytest exit 1 exactly: 2 is an interrupted collection, 3 internal, 4 usage, 5 nothing collected, 143 a signal, and treating "non-zero" as red let a cancelled run masquerade as evidence. `--continue-on-collection-errors` turns a genuine collection failure into 1 so add-a-file fixes stay provable. pytest's status is read from a marker written only after pytest returns, because `( cd X && pytest ); rc=$?` returns the SUBSHELL's status and a failed cd yields 1 with pytest never having run. Opt-in by trailer, enforced when present: requiring `Proves:` everywhere would block docs/CI/refactor commits and get switched off, which is how a check ends up running nowhere (#631). The trailer is the author's claim; CI checks claims, and says out loud when a PR carries none that its green asserts nothing. THE TOOL CAUGHT ITS OWN AUTHOR, AND THAT IS WHY THIS COMMIT CARRIES NO `Proves:` TRAILER. Three earlier commits on this branch each claimed `Proves: scripts/tests/test_prove_fix.py`. The new CI job ran on its own PR and returned UNPROVEN for all three: reverting the script restored a working earlier version that the suite ALSO passed, so those tests did not discriminate the fixes they claimed to pin. I had "verified" two of them against hand-written mutants that did not correspond to the code that actually existed. Cold review then measured the same thing independently and found why: - `test_SIGTERM_mid_run_never_reports_PROVEN` passed on the old script by accident — its trap deleted $TMP, a later step died 5, and "non-zero and no PROVEN" held either way. Now asserts exit 5 AND the handler's own message, which separates them. - `test_a_harness_failure_is_NOT_reported_as_PROVEN` never reached the marker branch: stubbing python3 to 127 still WRITES a marker and exits via control-failure. Now shims `git` so the reverted `worktree add` succeeds without creating the directory, on an add-file fixture whose revert is `rm -f` — the exact old false-green shape. Both now go RED against 587edbecc's script; the old version prints "red without it (pytest exit 2)" — a false PROVEN, witnessed. A trailer here would pass only because reverting DELETES prove-fix.sh, which is an add-file smoke check rather than a proof of its logic. Claiming that would be the overclaim this tool exists to catch, so the claim is omitted and the logic proof stays where it belongs: a clause-level mutation that re-runs the UNCHANGED refusal test against a mutant and witnesses it red (graded MUTATION in docs/guard-inventory.md). Also from review: the verdict's fallthrough default was PROVEN for a non-numeric status (both numeric tests return 2), now refused; a second `Proves:` trailer was silently ignored, now fatal; CI enumeration and trailer extraction were fail-open, now status-checked; `diff-tree` truncation is fatal; one cleanup and one EXIT trap after a second trap leaked a temp dir on signals; the record's "SIGTERM producing PROVEN from the first version" was not reproducible on that sha and now names the draft and the reproducible case instead. Verified: 15 tests; two of them witnessed red against the real pre-fix script; real commits 84165ab75 PROVEN and UNPROVEN against an unrelated test; merge and root commits refused; scripts/tests 681 passed, 2 skipped. Co-Authored-By: Claude Opus 5 (1M context) --- .gitea/workflows/pr-checks.yml | 91 +++++ docs/decisions/README.md | 1 + .../testing/fix-ships-a-witnessed-red-test.md | 68 ++++ docs/guard-inventory.md | 4 +- scripts/prove-fix.sh | 301 ++++++++++++++ scripts/tests/test_prove_fix.py | 373 ++++++++++++++++++ 6 files changed, 837 insertions(+), 1 deletion(-) create mode 100644 docs/decisions/records/testing/fix-ships-a-witnessed-red-test.md create mode 100755 scripts/prove-fix.sh create mode 100644 scripts/tests/test_prove_fix.py diff --git a/.gitea/workflows/pr-checks.yml b/.gitea/workflows/pr-checks.yml index 315cdc6c2..906b5d145 100644 --- a/.gitea/workflows/pr-checks.yml +++ b/.gitea/workflows/pr-checks.yml @@ -211,6 +211,97 @@ jobs: # `.claude/hooks/pretooluse-merge-consent.sh`, so its true input set spans at least two top-level # directories. A `scripts/**` filter would silently miss a `.claude/hooks/**` edit — and at ~10s a # filter buys nothing but drift. + prove-fix: + name: "Fix proofs (Proves trailers)" + runs-on: small + if: github.event_name == 'pull_request' + steps: + - name: Checkout + # Full history: prove-fix.sh reverts each commit against its PARENT, so a shallow + # clone would leave it unable to resolve `^` and it would refuse every commit. + uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + - name: Install test dependencies + run: python3 -m pip install --disable-pip-version-check --quiet pytest pyyaml + # OPT-IN BY TRAILER, deliberately. Requiring `Proves:` on every commit would block + # docs, CI and refactor commits that have no code side to revert, and a gate that + # blocks ordinary work gets disabled — which is how a check ends up running nowhere + # (#631). So the trailer is the AUTHOR'S CLAIM, and this job checks claims: write + # one and it must hold. Coverage is therefore honest rather than assumed, and + # `docs/decisions/records/testing/fix-ships-a-witnessed-red-test.md` says so. + - name: Prove every commit that claims a proof + run: | + set -uo pipefail + base="${{ github.event.pull_request.base.sha }}" + head="${{ github.event.pull_request.head.sha }}" + echo "range: $base..$head" + + # Capture and VALIDATE the enumeration before looping. `for sha in $(git ...)` + # swallows a git failure: the command substitution yields nothing, the loop body + # never runs, and the job reports "0 claims" green. Fail-open enumeration in the + # thing that decides what gets checked is the defect this job exists to catch. + if ! shas="$(git rev-list "$base".."$head")"; then + echo "::error::git rev-list failed for $base..$head — cannot enumerate commits," \ + "so this job cannot assert anything. Refusing to pass." + exit 1 + fi + + claimed=0; proven=0; failed=0 + while IFS= read -r sha; do + [ -n "$sha" ] || continue + # Trim whitespace only — NOT `xargs`, which applies quote parsing and turns a + # legitimate parametrised node id like test_x[can't] into an empty selector, + # silently dropping a real claim. + # Extract with a CHECKED status. `sel="$(git show ... )"` under `set -uo + # pipefail` but no `-e` yields an empty selector when git fails, the commit is + # skipped, and the job exits 0 having been unable to inspect a possible claim — + # fail-open in the step that decides what gets checked. + if ! raw="$(git show -s --format='%(trailers:key=Proves,valueonly)' "$sha")"; then + echo "::error::git show failed for $sha — cannot read its trailers, so this" \ + "job cannot assert anything about it. Refusing to pass." + exit 1 + fi + sel="$(printf '%s\n' "$raw" | head -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" + [ -n "$sel" ] || continue + claimed=$((claimed + 1)) + + # A merge commit has several parents, so "before this change" is ambiguous. + # prove-fix.sh refuses them; catch it here with a clearer message rather than + # letting the trailer be silently skipped (which --no-merges used to do). + if [ "$(git rev-list --parents -n 1 "$sha" | wc -w)" -gt 2 ]; then + failed=$((failed + 1)) + echo "::error::commit $sha is a MERGE carrying 'Proves: $sel'. Put the trailer" \ + "on the commit that carries the fix — a merge has no single 'before'." + continue + fi + + echo "::group::prove $sha -> $sel" + if bash ./scripts/prove-fix.sh "$sha" "$sel"; then + proven=$((proven + 1)); echo "PROVEN $sha" + else + rc=$? + failed=$((failed + 1)) + echo "::error::commit $sha claims 'Proves: $sel' but prove-fix.sh exited $rc." \ + "A claimed proof that does not hold is worse than none — it reads as" \ + "coverage. Strengthen the test until reverting the fix reddens it, or" \ + "drop the trailer." + fi + echo "::endgroup::" + done <<< "$shas" + + echo "commits claiming a proof: $claimed (proven $proven, failed $failed)" + if [ "$claimed" -eq 0 ]; then + echo "::notice::No commit in this PR carries a 'Proves:' trailer, so nothing was" \ + "verified here. That is allowed — the trailer is opt-in — but it means this" \ + "job asserts NOTHING about this PR. Do not read its green as fix coverage." + fi + [ "$failed" -eq 0 ] + script-tests: name: Script tests (pytest) runs-on: small diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 4510d26ad..d409fd4b1 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -194,6 +194,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera | `testing.e2e-cleanup-scope-by-pid` | An E2E harness or agent may only kill processes whose PIDs it captured at launch — capture the PID; whoever owns the lifecycle releases it from a `trap ... EXIT INT TERM`. Never `pkill -f "dotnet ErsatzTV.dll"` (or any pattern that can match a process this run did not start). A foreign listener is reported, not reaped. | 2026-07-25 | [link](records/testing/e2e-cleanup-scope-by-pid.md) | | `testing.e2e-local-fresh-config-dir` | Always point `scripts/e2e-local.sh` at a fresh config dir — leftover channels/schedules/DB rows bleed state between runs and corrupt assertions. (The *readiness-probe hang* this record was originally written about was fixed in #533; the fresh-dir rule stands on state-bleed grounds alone.) | 2026-07-21 | [link](records/testing/e2e-local-fresh-config-dir.md) | | `testing.enumerating-guard-identity-not-position` | A guard that cross-checks a hand-reviewed registry against call sites discovered across the whole repo must key each entry on properties INTRINSIC to the site — file, kind, and the value source text — and never on its absolute line or column. A registry keyed on position is a function of every other file in the repo, so a branch that never touches the guard can invalidate it; and because each PR is green against its own base, that failure is structurally invisible pre-merge and lands on `main` after review and after the merge gate. Dropping the position keeps every mutation the guard exists for — a NEW site, a REMOVED site and a CHANGED value each still fail, since each changes the identity multiset — and costs exactly ONE case, which must be stated rather than implied: a SAME-IDENTITY SUBSTITUTION within one file (delete a registered site, add a different unreviewed one with the same kind and value token, net-zero count) now passes. A REPORTED failure still prints the discovered line:column, because identity and diagnostics need not share a format. Comparison stays a MULTISET count rather than set membership, so two sites in one file sharing an identity must be discovered exactly that many times and a third occurrence still fails. A SCANNER test that asserts real AST positions against FIXED inline fixtures is the opposite case and keeps its line/column identity — it has no churn, because its input does not move. | 2026-07-27 | [link](records/testing/enumerating-guard-identity-not-position.md) | +| `testing.fix-ships-a-witnessed-red-test` | A commit claiming to fix something may carry a `Proves: ` trailer; when it does, `scripts/prove-fix.sh` must show that selector GREEN with the fix and RED with the code side reverted, and CI enforces it per-PR. The trailer is opt-in — an unproven commit is allowed — but a claimed proof that does not hold fails the build. | 2026-08-16 | [link](records/testing/fix-ships-a-witnessed-red-test.md) | | `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) | | `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file; clause-level grading is tracked in #790. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. What stays with review, and is stated rather than papered over: nothing checks that a row claiming `MUTATION` is telling the truth. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) | | `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin