diff --git a/.gitea/workflows/pr-checks.yml b/.gitea/workflows/pr-checks.yml index 315cdc6c2..45bd89954 100644 --- a/.gitea/workflows/pr-checks.yml +++ b/.gitea/workflows/pr-checks.yml @@ -211,6 +211,123 @@ jobs: # `.claude/hooks/pretooluse-merge-consent.sh`, so its true input set spans at least two top-level # directories. A `scripts/**` filter would silently miss a `.claude/hooks/**` edit — and at ~10s a # filter buys nothing but drift. + prove-fix: + name: "Fix proofs (Proves trailers)" + runs-on: small + if: github.event_name == 'pull_request' + steps: + - name: Checkout + # Full history: prove-fix.sh reverts each commit against its PARENT, so a shallow + # clone would leave it unable to resolve `^` and it would refuse every commit. + uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + - name: Install test dependencies + run: python3 -m pip install --disable-pip-version-check --quiet pytest pyyaml + # OPT-IN BY TRAILER, deliberately. Requiring `Proves:` on every commit would block + # docs, CI and refactor commits that have no code side to revert, and a gate that + # blocks ordinary work gets disabled — which is how a check ends up running nowhere + # (#631). So the trailer is the AUTHOR'S CLAIM, and this job checks claims: write + # one and it must hold. Coverage is therefore honest rather than assumed, and + # `docs/decisions/records/testing/fix-ships-a-witnessed-red-test.md` says so. + - name: Prove every commit that claims a proof + run: | + set -uo pipefail + base="${{ github.event.pull_request.base.sha }}" + head="${{ github.event.pull_request.head.sha }}" + echo "range: $base..$head" + + # Capture and VALIDATE the enumeration before looping. `for sha in $(git ...)` + # swallows a git failure: the command substitution yields nothing, the loop body + # never runs, and the job reports "0 claims" green. Fail-open enumeration in the + # thing that decides what gets checked is the defect this job exists to catch. + if ! shas="$(git rev-list "$base".."$head")"; then + echo "::error::git rev-list failed for $base..$head — cannot enumerate commits," \ + "so this job cannot assert anything. Refusing to pass." + exit 1 + fi + + claimed=0; proven=0; failed=0 + while IFS= read -r sha; do + [ -n "$sha" ] || continue + # Trim whitespace only — NOT `xargs`, which applies quote parsing and turns a + # legitimate parametrised node id like test_x[can't] into an empty selector, + # silently dropping a real claim. + # Extract with a CHECKED status. `sel="$(git show ... )"` under `set -uo + # pipefail` but no `-e` yields an empty selector when git fails, the commit is + # skipped, and the job exits 0 having been unable to inspect a possible claim — + # fail-open in the step that decides what gets checked. + if ! raw="$(git show -s --format='%(trailers:key=Proves,valueonly)' "$sha")"; then + echo "::error::git show failed for $sha — cannot read its trailers, so this" \ + "job cannot assert anything about it. Refusing to pass." + exit 1 + fi + # Refuse MORE THAN ONE `Proves:` here too. prove-fix.sh has this guard, but it + # only fires when it reads the trailer itself — and this job passes the selector + # explicitly, so the guard was bypassed on the one path that actually enforces. + # Measured: a commit with two trailers reported PROVEN while the second was never + # run. Fixing the script and not its twin is how a guard reads as coverage. + # Count trailer PRESENCE, not non-empty values: `%(...valueonly)` renders a bare + # `Proves:` as an empty line, so counting non-empty lines misses a commit whose + # FIRST trailer is empty — `sel` then comes out empty and the commit is skipped + # in silence, with a real second selector never checked. Fail-open in CI while + # the script is fail-closed is the same asymmetry this guard exists to remove. + present="$(git show -s --format='%(trailers:key=Proves)' "$sha")" + if [ "$(printf '%s\n' "$present" | grep -c .)" -gt 1 ]; then + claimed=$((claimed + 1)); failed=$((failed + 1)) + echo "::error::commit $sha carries more than one 'Proves:' trailer; only the" \ + "first would be checked, so the rest would read as proven without ever" \ + "running. Use a single selector." + continue + fi + sel="$(printf '%s\n' "$raw" | head -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" + # A trailer that is PRESENT but empty is a claim with no selector. Refuse it + # loudly; skipping it silently would let the job report "no claims" for a PR that + # made one. + if [ -n "$present" ] && [ -z "$sel" ]; then + claimed=$((claimed + 1)); failed=$((failed + 1)) + echo "::error::commit $sha carries a 'Proves:' trailer with no selector." + continue + fi + [ -n "$sel" ] || continue + claimed=$((claimed + 1)) + + # A merge commit has several parents, so "before this change" is ambiguous. + # prove-fix.sh refuses them; catch it here with a clearer message rather than + # letting the trailer be silently skipped (which --no-merges used to do). + if [ "$(git rev-list --parents -n 1 "$sha" | wc -w)" -gt 2 ]; then + failed=$((failed + 1)) + echo "::error::commit $sha is a MERGE carrying 'Proves: $sel'. Put the trailer" \ + "on the commit that carries the fix — a merge has no single 'before'." + continue + fi + + echo "::group::prove $sha -> $sel" + if bash ./scripts/prove-fix.sh "$sha" "$sel"; then + proven=$((proven + 1)); echo "PROVEN $sha" + else + rc=$? + failed=$((failed + 1)) + echo "::error::commit $sha claims 'Proves: $sel' but prove-fix.sh exited $rc." \ + "A claimed proof that does not hold is worse than none — it reads as" \ + "coverage. Strengthen the test until reverting the fix reddens it, or" \ + "drop the trailer." + fi + echo "::endgroup::" + done <<< "$shas" + + echo "commits claiming a proof: $claimed (proven $proven, failed $failed)" + if [ "$claimed" -eq 0 ]; then + echo "::notice::No commit in this PR carries a 'Proves:' trailer, so nothing was" \ + "verified here. That is allowed — the trailer is opt-in — but it means this" \ + "job asserts NOTHING about this PR. Do not read its green as fix coverage." + fi + [ "$failed" -eq 0 ] + script-tests: name: Script tests (pytest) runs-on: small diff --git a/docs/ci-cd.md b/docs/ci-cd.md index d3b38eda7..5ce0c456d 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -200,7 +200,10 @@ rather than in `docker-build.yml` — see that section (ersatztv#535). **`small` is git-only, and that is load-bearing (server-management#639).** Everything in the lane is a checkout plus a `git diff`: `decisions-guard`, `ci-image-pin`, -`docs-reminder` — plus `script-tests`, which is a checkout plus a `pytest` run needing only +`docs-reminder` — plus `prove-fix` (ersatztv#794), the heaviest member: per commit carrying a +`Proves:` trailer it makes two `git worktree add`s and runs an arbitrary pytest selection twice, +so a PR claiming many proofs costs proportionally more than the rest of the lane combined — plus +`script-tests`, which is a checkout plus a `pytest` run needing only `pytest` and `pyyaml` (ersatztv#631; it is NOT stdlib-only — that assumption is what turned the job red on its first CI run, see below) — plus **`scan`** (ersatztv#767), the same lightweight-Python shape. `scan` is the lane member to think hardest about before changing anything @@ -871,6 +874,36 @@ compiler/docker build), so it doesn't violate the "small is git-only" lane rule. `docs-reminder`, otherwise a seconds-long `git diff` + parse with no dotnet/node setup (`runs-on: small`). +### `prove-fix` job (`Fix proofs (Proves trailers)`, PR-only — in `pr-checks.yml`) + +Runs `scripts/prove-fix.sh` for **every commit in the PR that carries a `Proves: ` +trailer**, and fails the PR if a claimed proof does not hold. The rule and its rationale are +`testing.fix-ships-a-witnessed-red-test`; this section is the CI-side contract. + +**Opt-in by trailer, enforced when present.** Requiring `Proves:` on every commit would block docs, +CI and refactor commits that have no code side to revert, and a gate that blocks ordinary work gets +switched off — which is how a check ends up running nowhere (ersatztv#631). So the trailer is the +author's *claim* and this job checks claims. **When a PR carries none, the job emits a `::notice::` +saying it asserted nothing** — its green must not be read as fix coverage. + +**It needs full history** (`fetch-depth: 0`): the script reverts each commit against its PARENT, and +a shallow clone cannot resolve `^`. + +Three refusals worth knowing before you write a trailer: +- a **merge** commit is rejected — several parents means "the code before this change" is ambiguous; + put the trailer on the commit carrying the fix; +- **more than one** `Proves:` trailer is rejected — only the first would be checked, so the rest + would read as proven without ever running (the job checks this itself, because passing the + selector explicitly bypasses the script's own guard); +- a **test-only** commit is rejected — there is no code side to revert. + +**Only pytest exit 1 counts as red.** 2 is an interrupted collection, 3 internal, 4 usage, 5 nothing +collected, 143 a signal; `--continue-on-collection-errors` converts a genuine collection failure to +1 so add-a-file fixes stay provable. The job inherits that. Note the direction: a wobble DOES redden +this job (`prove-fix.sh` exits 5 on a harness/git failure or a signal, and the job turns that into an +error), so what the exit-code discipline buys is the other way round — a **green** here means a claim +was witnessed, never that a run was cancelled or broke. + ### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`) > Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the @@ -964,7 +997,7 @@ assumed. **File:** `.gitea/workflows/pr-checks.yml` — `on: pull_request` only. -The four git-only PR gates — `ci-image-pin`, `docs-reminder`, `decisions-guard`, `script-tests` +The five PR gates — `ci-image-pin`, `docs-reminder`, `decisions-guard`, `script-tests`, and `prove-fix` (the one member that is NOT merely checkout + `git diff`: it installs pytest and creates worktrees) (all described above) — live here, **not** in `docker-build.yml`, and that separation is the fix for **ersatztv#535**. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 4510d26ad..d409fd4b1 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -194,6 +194,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera | `testing.e2e-cleanup-scope-by-pid` | An E2E harness or agent may only kill processes whose PIDs it captured at launch — capture the PID; whoever owns the lifecycle releases it from a `trap ... EXIT INT TERM`. Never `pkill -f "dotnet ErsatzTV.dll"` (or any pattern that can match a process this run did not start). A foreign listener is reported, not reaped. | 2026-07-25 | [link](records/testing/e2e-cleanup-scope-by-pid.md) | | `testing.e2e-local-fresh-config-dir` | Always point `scripts/e2e-local.sh` at a fresh config dir — leftover channels/schedules/DB rows bleed state between runs and corrupt assertions. (The *readiness-probe hang* this record was originally written about was fixed in #533; the fresh-dir rule stands on state-bleed grounds alone.) | 2026-07-21 | [link](records/testing/e2e-local-fresh-config-dir.md) | | `testing.enumerating-guard-identity-not-position` | A guard that cross-checks a hand-reviewed registry against call sites discovered across the whole repo must key each entry on properties INTRINSIC to the site — file, kind, and the value source text — and never on its absolute line or column. A registry keyed on position is a function of every other file in the repo, so a branch that never touches the guard can invalidate it; and because each PR is green against its own base, that failure is structurally invisible pre-merge and lands on `main` after review and after the merge gate. Dropping the position keeps every mutation the guard exists for — a NEW site, a REMOVED site and a CHANGED value each still fail, since each changes the identity multiset — and costs exactly ONE case, which must be stated rather than implied: a SAME-IDENTITY SUBSTITUTION within one file (delete a registered site, add a different unreviewed one with the same kind and value token, net-zero count) now passes. A REPORTED failure still prints the discovered line:column, because identity and diagnostics need not share a format. Comparison stays a MULTISET count rather than set membership, so two sites in one file sharing an identity must be discovered exactly that many times and a third occurrence still fails. A SCANNER test that asserts real AST positions against FIXED inline fixtures is the opposite case and keeps its line/column identity — it has no churn, because its input does not move. | 2026-07-27 | [link](records/testing/enumerating-guard-identity-not-position.md) | +| `testing.fix-ships-a-witnessed-red-test` | A commit claiming to fix something may carry a `Proves: ` trailer; when it does, `scripts/prove-fix.sh` must show that selector GREEN with the fix and RED with the code side reverted, and CI enforces it per-PR. The trailer is opt-in — an unproven commit is allowed — but a claimed proof that does not hold fails the build. | 2026-08-16 | [link](records/testing/fix-ships-a-witnessed-red-test.md) | | `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) | | `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file; clause-level grading is tracked in #790. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. What stays with review, and is stated rather than papered over: nothing checks that a row claiming `MUTATION` is telling the truth. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) | | `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin