8d9be0d4b8a18869d62167e4d5c1d58294dfffc2
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8d9be0d4b8 |
fix(794): a status of 1 must come from pytest, not from a failed cd
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 8s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 21s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 23s
review-verdict/h10 Awaiting review verdict for 8d9be0d
Review verdict / Set review-verdict status (pull_request_target) Successful in 19s
PR Gates / Fix proofs (Proves trailers) (pull_request) Failing after 2m2s
PR Gates / Script tests (pytest) (pull_request) Successful in 4m17s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m45s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m49s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m28s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
Round 2: BLOCKED, 1 Blocker + 2 High + 2 Low; 5 of 10 round-1 items verified fixed.
THE BLOCKER IS THE SAFETY CLAIM ITSELF. `( cd X && pytest ); rc=$?` returns the
SUBSHELL's status, so a failed cd or a failed redirection also yields 1 — and the
"only pytest exit 1 counts as red" rule then accepts it as a witnessed test failure
with pytest never having run. The rule I added last round to close a false green had a
false green underneath it.
Fixed structurally: the phase writes pytest's status to a marker file only AFTER pytest
returns. An absent marker means "pytest did not complete" and is refused as a harness
failure, which cannot be confused with a verdict. Proven load-bearing —
test_a_harness_failure_is_NOT_reported_as_PROVEN goes RED when the marker logic is
mutated back to trusting the subshell status.
Two fail-open enumerations, both real:
- the CI job read each commit's trailer with an unchecked command substitution, so a
failing `git show` produced an empty selector, skipped the commit, and let the job
exit 0 having been unable to inspect a possible claim;
- `while ... done < <(git diff-tree ...)` cannot see the producer's exit status, so a
git failure that had already emitted one complete record passed the non-empty check
and reverted only PART of the commit — overstating "without the fix". The stream is
now captured to a file with a checked status, and any truncated record is fatal.
The selector is now split once into an array and always expanded as "${SEL[@]}".
Unquoted it also applied PATHNAME expansion, so a node id containing a glob character
could select different tests than the trailer names.
PROVE_FIX_PATH is now gated behind PROVE_FIX_MUTATION_RUN=1: as an unrestricted ambient
override, a stray CI value would silently point the whole suite at another script.
Verified: 14 tests; the blocker fix and the mutation proof each witnessed red against
their own mutants; real commits
|
||
|
|
efe3dda983 |
fix(794): only pytest exit 1 is red — a killed run was printing PROVEN
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Failing after 3s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 8s
PR Gates / Docs update reminder (pull_request) Successful in 11s
PR Gates / decisions lifecycle (pull_request) Successful in 19s
review-verdict/h10 Awaiting review verdict for efe3dda
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
PR Gates / Fix proofs (Proves trailers) (pull_request) Failing after 1m11s
PR Gates / Script tests (pytest) (pull_request) Successful in 4m22s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m57s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Canceled after 3m21s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 0s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Canceled after 0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Cold review returned BLOCKED with 1 Blocker, 4 High, 4 Medium, 1 Low. The Blocker is
this tool committing the defect it exists to catch: nearly every non-zero status was
read as "the test failed", so a SIGTERM (143) produced PROVEN and exit 0. Cancellation
masquerading as evidence.
Fixed by measurement, not assumption. Exit codes on this machine: pass 0, fail 1,
collection-error 2, SIGTERM 143 — and with --continue-on-collection-errors a collection
error becomes 1. So the script now passes that flag and accepts ONLY exit 1 as red,
which closes the ambiguous 2/3 band without needing to parse output, and still supports
the add-a-file case (reverting the module the test imports). INT/TERM handlers now exit
explicitly; without that the trap fell through and the previous status stood.
Proven load-bearing: test_SIGTERM_mid_run_never_reports_PROVEN goes RED when the
exit-code strictness and the signal exit are mutated out.
The MUTATION grade was NOT earned and is now earned. Review was right: the old test
deleted the clause and asserted the MUTANT misbehaved, leaving the named test green,
which is an argument rather than a witnessed red — and docs/guard-inventory.md says
plainly that the column is not a grading curve. The test now re-runs the UNCHANGED
refusal test against the mutant via PROVE_FIX_PATH and requires it to FAIL (exit 1, one
reported failure, named). Loose assertions were tightened after checking WHY the nested
run failed rather than that it failed.
Other findings, all real:
- control and reverted phases now get SEPARATE worktrees and TMPDIRs, so state written
by the control run cannot decide the second one;
- revert is driven by git's own --name-status letters (NUL-delimited) instead of probing
the parent blob and reading error text — `cat-file -e parent:new.py` writes a non-empty
message for a legitimately added file, so the old logic both misclassified that and
could not distinguish a real git failure;
- the CI loop no longer parses selectors with `xargs` (it mangles a parametrised node id
like test_x[can't] into an empty selector, silently dropping a claim) and no longer
enumerates commits fail-open (`for sha in $(git rev-list ...)` succeeds with zero
commits when git fails);
- merge commits carrying `Proves:` are refused with a reason instead of being skipped by
`--no-merges`;
- PROVEN now prints the observed failures, since the docstring asks review to judge
whether the RIGHT property failed and hiding the report made that impossible.
Also corrected an overclaim of my own: the self-proof on the previous commit goes red
because the negative phase DELETES prove-fix.sh, so it demonstrates the file is required,
not that the logic discriminates. The clause mutation is what carries that.
Verified: 13 tests; real commits
|
||
|
|
587edbecc0 |
feat(794): witness a fix's test failing BEFORE the fix, and check the claim in CI
PR Gates / Docs update reminder (pull_request) Successful in 12s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 13s
PR Gates / decisions lifecycle (pull_request) Successful in 19s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 26s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 21s
review-verdict/h10 Awaiting review verdict for 587edbe
Review verdict / Set review-verdict status (pull_request_target) Successful in 15s
PR Gates / Script tests (pytest) (pull_request) Successful in 3m57s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m15s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Canceled after 6m3s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 0s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Canceled after 0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Closes the mechanised half of the defect that took #776 and #793 six review rounds each: a fix's test written to confirm the fix rather than to discriminate against its absence. This is testing.guard-ships-with-mutation-proof generalised from guards to fixes — refs #794, and #796 is the same finding measured from the other side. scripts/prove-fix.sh checks a RED-GREEN PAIR, not a red. It runs the selector at the commit (control, must be green), then reverts only the commit's non-test files and runs it again (must be non-zero). The control is what makes the second run mean anything: a test already failing proves nothing by failing again. That hole was real — the first version had no control, and this file's own tests found it. Two exit-code traps, both found by the tests rather than reasoned about: - pytest exits 4 for a path that does not exist and 5 when it collects nothing. The first version handled only 5, so a mistyped selector fell through to "non-zero implies red" and would have PROVEN every fix. - exit 2 (collection error) after a green control IS accepted: when the fix adds the module the test imports, reverting makes it unloadable, which is the strongest discrimination available. Refusing it would leave every add-a-file fix unprovable. Opt-in by trailer, enforced when present. Requiring Proves: on every commit would block docs/CI/refactor commits with no code side, and a gate that blocks ordinary work gets switched off — that is how a check ends up running nowhere (#631). The trailer is the author's claim; CI checks claims. When a PR carries none the job says out loud that its green asserts nothing. Graded MUTATION honestly. test_unrelated_test_is_UNPROVEN only shows the script reacts to its input, which docs/guard-inventory.md calls BEHAVIOUR-ONLY. The MUTATION row cites a test that deletes the UNPROVEN clause and witnesses the prover wrongly reporting PROVEN — clause, not file (#510). Also avoided mapfile: absent from bash 3.2 (macOS /bin/bash), where a missing builtin leaves the array EMPTY rather than erroring, and the test asserts by EXECUTING under /bin/bash rather than grepping the source — the first version grepped for "mapfile" and matched the comment explaining why it is avoided. Run against real commits: |