feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI #801

Merged

timothy merged 3 commits from fix/794-prove-fix into main 2026-08-16 12:25:00 +02:00

Conversation 1 Commits 3 Files Changed 7

+897 -3

timothy commented 2026-08-16 10:16:46 +02:00

Owner

Copy Link

Copy Source

fixes #794

Mechanises the defect that took #776 and #793 six review rounds each: a fix's test written to confirm the fix, not to discriminate against its absence. testing.guard-ships-with-mutation-proof generalised from guards to fixes. #796 is the same finding from the other side.

It checks a RED-GREEN PAIR, not a red

scripts/prove-fix.sh runs the selector twice — at the commit (control, must be green), then with the commit's non-test files reverted (must be non-zero). The control is what makes the second run mean anything: a test that is already failing proves nothing by failing again.

That hole was real. The first version had no control, and this file's own tests found it.

Two exit-code traps, both found by tests rather than reasoned about

• pytest exits 4 for a path that does not exist and 5 when it collects nothing. The first version handled only 5 — so a mistyped selector fell through to "non-zero implies red" and would have PROVEN every fix.
• exit 2 (collection error) after a green control is accepted: when the fix adds the module the test imports, reverting makes it unloadable. That is the strongest discrimination available, and refusing it would leave every add-a-file fix unprovable.

Opt-in by trailer, enforced when present

Requiring Proves: on every commit would block docs/CI/refactor commits with no code side — and a gate that blocks ordinary work gets switched off, which is how a check ends up running nowhere (#631). The trailer is the author's claim; CI checks claims. When a PR carries none, the job says out loud that its green asserts nothing.

Graded MUTATION honestly

docs/guard-inventory.md is explicit that feeding a script a rejecting input is BEHAVIOUR-ONLY, not MUTATION. So test_unrelated_test_is_UNPROVEN does not earn the grade. The row cites a test that deletes the UNPROVEN clause and witnesses the prover wrongly reporting PROVEN — clause, not file (#510).

Verification

• Self-proof: this PR's own commit carries Proves: and passes — green with the fix, red without
• Real commits: 84165ab75 PROVEN against its own test, UNPROVEN against an unrelated one (the load-bearing control), b8f60bdec refused as test-only (rc=4)
• 12 tests covering PROVEN / UNPROVEN / trailer / no-selector-refuses / test-only-refuses / bad-selector-refuses / added-file / root-commit / control-failure / clause mutation
• scripts/tests: 679 passed, 2 skipped
• mapfile avoided (absent in bash 3.2, where it yields an empty array rather than erroring) and asserted by executing under /bin/bash, not by grepping the source — the first version grepped for "mapfile" and matched the comment explaining why it is avoided

Stated limits

It shows a test can go red, not that it asserts the right property — #776's test_output_survives_a_SIGTERM would have passed while still missing returncode. That judgement stays with review. Python-only; no coverage claimed for C# or web/.

fixes #794 Mechanises the defect that took #776 and #793 six review rounds each: **a fix's test written to confirm the fix, not to discriminate against its absence.** `testing.guard-ships-with-mutation-proof` generalised from guards to fixes. #796 is the same finding from the other side. ## It checks a RED-GREEN PAIR, not a red `scripts/prove-fix.sh` runs the selector twice — at the commit (**control, must be green**), then with the commit's non-test files reverted (**must be non-zero**). The control is what makes the second run mean anything: a test that is already failing proves nothing by failing again. That hole was real. The first version had no control, and this file's own tests found it. ## Two exit-code traps, both found by tests rather than reasoned about - pytest exits **4** for a path that does not exist and **5** when it collects nothing. The first version handled only 5 — so a mistyped selector fell through to "non-zero implies red" and would have **PROVEN every fix**. - exit **2** (collection error) after a green control **is** accepted: when the fix adds the module the test imports, reverting makes it unloadable. That is the strongest discrimination available, and refusing it would leave every add-a-file fix unprovable. ## Opt-in by trailer, enforced when present Requiring `Proves:` on every commit would block docs/CI/refactor commits with no code side — and a gate that blocks ordinary work gets switched off, which is how a check ends up running nowhere (#631). The trailer is the author's **claim**; CI checks claims. When a PR carries none, the job says out loud that its green asserts nothing. ## Graded MUTATION honestly `docs/guard-inventory.md` is explicit that feeding a script a rejecting input is BEHAVIOUR-ONLY, not MUTATION. So `test_unrelated_test_is_UNPROVEN` does not earn the grade. The row cites a test that **deletes the UNPROVEN clause and witnesses the prover wrongly reporting PROVEN** — clause, not file (#510). ## Verification - **Self-proof**: this PR's own commit carries `Proves:` and passes — green with the fix, red without - **Real commits**: `84165ab75` PROVEN against its own test, **UNPROVEN against an unrelated one** (the load-bearing control), `b8f60bdec` refused as test-only (rc=4) - 12 tests covering PROVEN / UNPROVEN / trailer / no-selector-refuses / test-only-refuses / bad-selector-refuses / added-file / root-commit / control-failure / clause mutation - `scripts/tests`: **679 passed, 2 skipped** - `mapfile` avoided (absent in bash 3.2, where it yields an empty array rather than erroring) and asserted by **executing** under `/bin/bash`, not by grepping the source — the first version grepped for "mapfile" and matched the comment explaining why it is avoided ## Stated limits It shows a test **can** go red, not that it asserts the **right** property — #776's `test_output_survives_a_SIGTERM` would have passed while still missing `returncode`. That judgement stays with review. Python-only; no coverage claimed for C# or `web/`.

timothy added 1 commit 2026-08-16 11:32:21 +02:00

feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI ... 4f41134349

Mechanises the defect that took #776 and #793 six review rounds each: a fix's test
written to confirm the fix, not to discriminate against its absence.
testing.guard-ships-with-mutation-proof generalised from guards to fixes. Closes #794.

`scripts/prove-fix.sh <commit> [selector]` checks a RED-GREEN PAIR — the selector must
pass at the commit (control) and fail with the commit's non-test files reverted, in a
SEPARATE fresh worktree. RED means pytest exit 1 exactly: 2 is an interrupted
collection, 3 internal, 4 usage, 5 nothing collected, 143 a signal, and treating
"non-zero" as red let a cancelled run masquerade as evidence.
`--continue-on-collection-errors` turns a genuine collection failure into 1 so
add-a-file fixes stay provable. pytest's status is read from a marker written only
after pytest returns, because `( cd X && pytest ); rc=$?` returns the SUBSHELL's status
and a failed cd yields 1 with pytest never having run.

Opt-in by trailer, enforced when present: requiring `Proves:` everywhere would block
docs/CI/refactor commits and get switched off, which is how a check ends up running
nowhere (#631). The trailer is the author's claim; CI checks claims, and says out loud
when a PR carries none that its green asserts nothing.

THE TOOL CAUGHT ITS OWN AUTHOR, AND THAT IS WHY THIS COMMIT CARRIES NO `Proves:`
TRAILER. Three earlier commits on this branch each claimed
`Proves: scripts/tests/test_prove_fix.py`. The new CI job ran on its own PR and
returned UNPROVEN for all three: reverting the script restored a working earlier
version that the suite ALSO passed, so those tests did not discriminate the fixes they
claimed to pin. I had "verified" two of them against hand-written mutants that did not
correspond to the code that actually existed. Cold review then measured the same thing
independently and found why:

- `test_SIGTERM_mid_run_never_reports_PROVEN` passed on the old script by accident —
  its trap deleted $TMP, a later step died 5, and "non-zero and no PROVEN" held either
  way. Now asserts exit 5 AND the handler's own message, which separates them.
- `test_a_harness_failure_is_NOT_reported_as_PROVEN` never reached the marker branch:
  stubbing python3 to 127 still WRITES a marker and exits via control-failure. Now
  shims `git` so the reverted `worktree add` succeeds without creating the directory,
  on an add-file fixture whose revert is `rm -f` — the exact old false-green shape.

Both now go RED against 587edbecc's script; the old version prints "red without it
(pytest exit 2)" — a false PROVEN, witnessed.

A trailer here would pass only because reverting DELETES prove-fix.sh, which is an
add-file smoke check rather than a proof of its logic. Claiming that would be the
overclaim this tool exists to catch, so the claim is omitted and the logic proof stays
where it belongs: a clause-level mutation that re-runs the UNCHANGED refusal test
against a mutant and witnesses it red (graded MUTATION in docs/guard-inventory.md).

Also from review: the verdict's fallthrough default was PROVEN for a non-numeric status
(both numeric tests return 2), now refused; a second `Proves:` trailer was silently
ignored, now fatal; CI enumeration and trailer extraction were fail-open, now
status-checked; `diff-tree` truncation is fatal; one cleanup and one EXIT trap after a
second trap leaked a temp dir on signals; the record's "SIGTERM producing PROVEN from
the first version" was not reproducible on that sha and now names the draft and the
reproducible case instead.

Verified: 15 tests; two of them witnessed red against the real pre-fix script; real
commits 84165ab75 PROVEN and UNPROVEN against an unrelated test; merge and root commits
refused; scripts/tests 681 passed, 2 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy force-pushed fix/794-prove-fix from 8d9be0d4b8 to 4f41134349 2026-08-16 11:32:21 +02:00 Compare

timothy added 1 commit 2026-08-16 11:49:30 +02:00

fix(794): the multi-trailer guard missed the path that actually enforces ... 6e79c155f1

Cold review of the squashed head: 1 High, 1 Medium, 2 Low, 2 Nits. It confirmed by
measurement the claims that matter — both discrimination tests go RED against
587edbecc (and the old script prints "red without it (pytest exit 2)", a witnessed
false PROVEN), the MUTATION grade is earned, 681 passed / 2 skipped, and the decision
to omit this branch's own `Proves:` trailer is sound rather than a dodge.

THE HIGH IS MY OWN RECORDED LESSON, VIOLATED. `fix-one-path-then-check-its-twin`:
I made a second `Proves:` trailer fatal in prove-fix.sh, but that guard only runs when
the script reads the trailer ITSELF — and the CI job does `head -1` and passes the
selector explicitly, so the guard was bypassed on the ONE path that enforces. Measured
on a two-trailer commit: the explicit-selector path returns PROVEN rc=0 with the second
claim never run. The job now refuses multi-trailer commits before it gets there, and
I reproduced the refusal.

That also makes the previous commit message's "a second `Proves:` trailer was silently
ignored, now fatal" FALSE as it stood — true of the script, not of CI. Correcting it
here rather than rewriting the pushed commit.

Same correction for the retracted SIGTERM claim: the record was fixed last round but
the sentence survived in TWO other copies (the script docstring and the test
docstring), where the test docstring was contradicted by its own inline comment thirty
lines below. Swept by subject this time, not by the one file I happened to be editing —
`stale-comments-sweep-by-subject-not-verb`.

Also: the previous message said "15 tests"; there are 14. Small, but that message is
otherwise an evidence record and an unscoped figure in one is how
`state-the-invariant-not-the-measurement` starts.

docs/ci-cd.md now documents the job — it was missing from the `small`-lane enumeration
(a resource-safety list, and this is the lane's heaviest member: two `git worktree add`s
plus two pytest runs per claimed commit) and had no per-job section while every other PR
gate does. Docs-update is part of done, and I had skipped it.

Nits: duplicated comment block in the test file; inventory row sorted.

Verified: two-trailer refusal reproduced on the CI logic; 14 tests; scripts/tests 681
passed, 2 skipped; decisions-validate OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-16 12:00:38 +02:00

fix(794): an EMPTY Proves: trailer was skipped in silence ... 163522fb3a

Cold review of the delta returned MERGEABLE with one MEDIUM it judged below the
blocking bar but would fold in before merge. Folding it in, because it is the same
script/CI asymmetry the previous commit set out to close.

`%(trailers:key=Proves,valueonly)` renders a bare `Proves:` as an EMPTY line, so
counting non-empty value lines missed a commit whose FIRST trailer is empty: the
multi-trailer guard did not fire, `sel` came out empty, and `[ -n "$sel" ] || continue`
dropped the commit without a word — with a real second selector never checked. If such
a commit were a PR's only claim, the job would print "No commit in this PR carries a
'Proves:' trailer", which is then simply false. No false PROVEN, but a claim that reads
as absent is still a claim that never ran.

Counting trailer PRESENCE (`%(trailers:key=Proves)`) catches it, and a present-but-empty
trailer is now refused loudly. Verified on both of the reviewer's scenarios: the
empty-first case counts 1 by the old expression and 2 by the new one, and a lone bare
trailer now errors instead of being skipped.

Also from the same review:
- docs/ci-cd.md had the direction backwards — a wobble DOES redden this job (exit 5 on a
  harness failure or signal); what the exit-code discipline buys is that a GREEN can
  never come from a cancelled or broken run. Reworded.
- the PR-gates enumeration still said "four git-only gates"; there are five, and
  prove-fix is the one that is not merely checkout + git diff.
- the last unqualified copy of the retracted SIGTERM claim now says "in the draft", and
  a 150-char comment line was rewrapped.

Verified: scripts/tests 681 passed, 2 skipped; yaml parses; bash -n clean; no line over
100 chars in the script.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy commented 2026-08-16 12:00:49 +02:00

Author

Owner

Copy Link

Copy Source

Review-verdict: MERGEABLE @ 163522f

Cold review of 6e79c155f returned MERGEABLE. Its one MEDIUM (an empty-valued Proves: trailer skipped silently by CI) was judged below the blocking bar but recommended for folding in before merge; 163522fb3 applies exactly that reviewer-prescribed fix and nothing else, verified on both scenarios the reviewer named (empty-first counts 1 old vs 2 new; a lone bare trailer now errors instead of being skipped), plus its two LOW doc corrections and two nits. Earlier rounds confirmed by measurement: both discrimination tests go RED against 587edbecc, the pre-fix script emits a witnessed false PROVEN, the MUTATION grade is earned per guard-inventory's own definition, and the verdict chain is fail-closed in every branch. scripts/tests 681 passed, 2 skipped. Note: this branch deliberately carries NO Proves: trailer — the only one that would pass does so because reverting deletes prove-fix.sh, an add-file smoke check rather than a proof of its logic.

Review-verdict: MERGEABLE @ 163522f Cold review of 6e79c155f returned MERGEABLE. Its one MEDIUM (an empty-valued Proves: trailer skipped silently by CI) was judged below the blocking bar but recommended for folding in before merge; 163522fb3 applies exactly that reviewer-prescribed fix and nothing else, verified on both scenarios the reviewer named (empty-first counts 1 old vs 2 new; a lone bare trailer now errors instead of being skipped), plus its two LOW doc corrections and two nits. Earlier rounds confirmed by measurement: both discrimination tests go RED against 587edbecc, the pre-fix script emits a witnessed false PROVEN, the MUTATION grade is earned per guard-inventory's own definition, and the verdict chain is fail-closed in every branch. scripts/tests 681 passed, 2 skipped. Note: this branch deliberately carries NO Proves: trailer — the only one that would pass does so because reverting deletes prove-fix.sh, an add-file smoke check rather than a proof of its logic.

timothy merged commit 15d2439915 into main 2026-08-16 12:25:00 +02:00

timothy deleted branch fix/794-prove-fix 2026-08-16 12:25:03 +02:00

timothy referenced this pull request 2026-08-16 12:25:13 +02:00

A fix ships a test witnessed failing BEFORE the fix — mechanise it by reverting the code side and asserting red #794

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

ad-hoc

One-off / ad-hoc work not tracked by a dedicated issue

api

REST API / HTTP endpoints

bug

Something isn't working

ci-cd

Build, test, deploy pipeline

content

Channel content / schedules / playlists

dependencies

Dependency updates (Renovate)

enhancement

New feature or improvement

frontend

ChicoryTV React SPA frontend

in-progress

Claimed by an active session — do not pick up

jellyfin

Jellyfin tuner / IPTV integration

parked

Excluded from automatic queue pickup; work only when explicitly selected

priority: high

priority: low

priority: medium

review

Adversarial review finding

security

Security / vulnerability fix

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate timothy (Timothy)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: timothy/ersatztv#801