feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI #801
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
163522fb3a |
fix(794): an EMPTY Proves: trailer was skipped in silence
review-verdict/h10 Review-verdict: MERGEABLE @ 163522f (base: main)
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 11s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 20s
PR Gates / decisions lifecycle (pull_request) Successful in 14s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 11s
Review verdict / Set review-verdict status (pull_request_target) Successful in 12s
PR Gates / Script tests (pytest) (pull_request) Successful in 3m38s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m18s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m45s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m37s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
Cold review of the delta returned MERGEABLE with one MEDIUM it judged below the blocking bar but would fold in before merge. Folding it in, because it is the same script/CI asymmetry the previous commit set out to close. `%(trailers:key=Proves,valueonly)` renders a bare `Proves:` as an EMPTY line, so counting non-empty value lines missed a commit whose FIRST trailer is empty: the multi-trailer guard did not fire, `sel` came out empty, and `[ -n "$sel" ] || continue` dropped the commit without a word — with a real second selector never checked. If such a commit were a PR's only claim, the job would print "No commit in this PR carries a 'Proves:' trailer", which is then simply false. No false PROVEN, but a claim that reads as absent is still a claim that never ran. Counting trailer PRESENCE (`%(trailers:key=Proves)`) catches it, and a present-but-empty trailer is now refused loudly. Verified on both of the reviewer's scenarios: the empty-first case counts 1 by the old expression and 2 by the new one, and a lone bare trailer now errors instead of being skipped. Also from the same review: - docs/ci-cd.md had the direction backwards — a wobble DOES redden this job (exit 5 on a harness failure or signal); what the exit-code discipline buys is that a GREEN can never come from a cancelled or broken run. Reworded. - the PR-gates enumeration still said "four git-only gates"; there are five, and prove-fix is the one that is not merely checkout + git diff. - the last unqualified copy of the retracted SIGTERM claim now says "in the draft", and a 150-char comment line was rewrapped. Verified: scripts/tests 681 passed, 2 skipped; yaml parses; bash -n clean; no line over 100 chars in the script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6e79c155f1 |
fix(794): the multi-trailer guard missed the path that actually enforces
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 8s
PR Gates / Docs update reminder (pull_request) Successful in 10s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 20s
PR Gates / decisions lifecycle (pull_request) Successful in 26s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 21s
review-verdict/h10 Awaiting review verdict for 6e79c15
Review verdict / Set review-verdict status (pull_request_target) Successful in 14s
PR Gates / Script tests (pytest) (pull_request) Successful in 3m44s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m23s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Canceled after 1m37s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 0s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Canceled after 0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Cold review of the squashed head: 1 High, 1 Medium, 2 Low, 2 Nits. It confirmed by
measurement the claims that matter — both discrimination tests go RED against
|
||
|
|
4f41134349 |
feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 16s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 18s
PR Gates / Docs update reminder (pull_request) Successful in 18s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 14s
review-verdict/h10 Awaiting review verdict for 4f41134
Review verdict / Set review-verdict status (pull_request_target) Successful in 15s
PR Gates / Script tests (pytest) (pull_request) Successful in 3m49s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m16s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m33s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 1m15s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Mechanises the defect that took #776 and #793 six review rounds each: a fix's test
written to confirm the fix, not to discriminate against its absence.
testing.guard-ships-with-mutation-proof generalised from guards to fixes. Closes #794.
`scripts/prove-fix.sh <commit> [selector]` checks a RED-GREEN PAIR — the selector must
pass at the commit (control) and fail with the commit's non-test files reverted, in a
SEPARATE fresh worktree. RED means pytest exit 1 exactly: 2 is an interrupted
collection, 3 internal, 4 usage, 5 nothing collected, 143 a signal, and treating
"non-zero" as red let a cancelled run masquerade as evidence.
`--continue-on-collection-errors` turns a genuine collection failure into 1 so
add-a-file fixes stay provable. pytest's status is read from a marker written only
after pytest returns, because `( cd X && pytest ); rc=$?` returns the SUBSHELL's status
and a failed cd yields 1 with pytest never having run.
Opt-in by trailer, enforced when present: requiring `Proves:` everywhere would block
docs/CI/refactor commits and get switched off, which is how a check ends up running
nowhere (#631). The trailer is the author's claim; CI checks claims, and says out loud
when a PR carries none that its green asserts nothing.
THE TOOL CAUGHT ITS OWN AUTHOR, AND THAT IS WHY THIS COMMIT CARRIES NO `Proves:`
TRAILER. Three earlier commits on this branch each claimed
`Proves: scripts/tests/test_prove_fix.py`. The new CI job ran on its own PR and
returned UNPROVEN for all three: reverting the script restored a working earlier
version that the suite ALSO passed, so those tests did not discriminate the fixes they
claimed to pin. I had "verified" two of them against hand-written mutants that did not
correspond to the code that actually existed. Cold review then measured the same thing
independently and found why:
- `test_SIGTERM_mid_run_never_reports_PROVEN` passed on the old script by accident —
its trap deleted $TMP, a later step died 5, and "non-zero and no PROVEN" held either
way. Now asserts exit 5 AND the handler's own message, which separates them.
- `test_a_harness_failure_is_NOT_reported_as_PROVEN` never reached the marker branch:
stubbing python3 to 127 still WRITES a marker and exits via control-failure. Now
shims `git` so the reverted `worktree add` succeeds without creating the directory,
on an add-file fixture whose revert is `rm -f` — the exact old false-green shape.
Both now go RED against 587edbecc's script; the old version prints "red without it
(pytest exit 2)" — a false PROVEN, witnessed.
A trailer here would pass only because reverting DELETES prove-fix.sh, which is an
add-file smoke check rather than a proof of its logic. Claiming that would be the
overclaim this tool exists to catch, so the claim is omitted and the logic proof stays
where it belongs: a clause-level mutation that re-runs the UNCHANGED refusal test
against a mutant and witnesses it red (graded MUTATION in docs/guard-inventory.md).
Also from review: the verdict's fallthrough default was PROVEN for a non-numeric status
(both numeric tests return 2), now refused; a second `Proves:` trailer was silently
ignored, now fatal; CI enumeration and trailer extraction were fail-open, now
status-checked; `diff-tree` truncation is fatal; one cleanup and one EXIT trap after a
second trap leaked a temp dir on signals; the record's "SIGTERM producing PROVEN from
the first version" was not reproducible on that sha and now names the draft and the
reproducible case instead.
Verified: 15 tests; two of them witnessed red against the real pre-fix script; real
commits
|