fix(787): derive the dropped-step guard's scope, and reconcile its snapshot against the server
Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 11s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 23s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 17s
PR Gates / Docs update reminder (pull_request) Successful in 17s
review-verdict/h10 Review-verdict: MERGEABLE @ b1b9c1a (base: main)
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 11s
PR Gates / decisions lifecycle (pull_request) Successful in 21s
Review verdict / Set review-verdict status (pull_request_target) Successful in 9s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 6m36s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m38s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m15s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 5m51s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 10s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 11s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 23s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 17s
PR Gates / Docs update reminder (pull_request) Successful in 17s
review-verdict/h10 Review-verdict: MERGEABLE @ b1b9c1a (base: main)
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 11s
PR Gates / decisions lifecycle (pull_request) Successful in 21s
Review verdict / Set review-verdict status (pull_request_target) Successful in 9s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 6m36s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m38s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m15s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 5m51s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 10s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
`scripts/tests/test_ci_dropped_step_guard.py` decides which jobs must carry per-step execution
markers from `MARKED_JOBS = ("test", "migrations")` — a hand-written mirror of the required status
contexts on `main`, annotated `Read live on 2026-08-10` and compared against the server by nothing.
A dated comment is a claim about the past. Add a fourth required context and it acquired no
dropped-step guard, silently, in the file the decision record names as its own model.
The constraint that made this its own issue: `pr-checks.yml::script-tests` checks out with
`persist-credentials: false`, so it cannot ask Gitea, and the issue's own candidate fixes were each
unsatisfying — a snapshot "just moves the hand-maintenance", a credentialed check that skips without
creds is the #756 fail-open.
SPLIT THE INVARIANT AT THE CREDENTIAL BOUNDARY and check each half where it can be checked:
* `.gitea/required-status-contexts.json` — the mirror, committed verbatim from the API.
* The guard DERIVES `MARKED_JOBS` from it offline. Each candidate context string is SYNTHESIZED from
the workflow (`<workflow name> / <job name> (<event>)`) and matched by equality, never parsed —
the delimiters occur inside the names they delimit (`Build & test (.NET)` ends in a parenthesis of
its own). A required context mapping to neither a job nor an explicit `ACCOUNTED_ELSEWHERE`
disposition is a HARD failure at import, which is what makes a fourth required check impossible to
add in silence. Both directions are asserted, so a disposition that stops being required is
reported too.
* `scripts/check-required-contexts.sh` reconciles the mirror against live `status_check_contexts`,
called from `pretooluse-merge-consent.sh`.
So the maintenance is now a verbatim copy with an equality check on each side, and updating it to
match the server makes CI red until the newly required job is marked.
Three decisions worth keeping:
* The hook arm ASKS, never denies. Drift does not make that merge unsafe — Gitea enforces the live
required set server-side — so a deny would state something false about the change in front of the
reader. What is stale is a guard's scope.
* It runs LAST, so it can only downgrade an otherwise-satisfied auto-grant and never masks a more
serious refusal (asserted, not described).
* It runs on BOTH merge paths. The pre-existing branch-protection read sits inside the
`merge_when_checks_succeed` branch and never fires on an immediate merge — the common case. This
hook already records that exact shape once: the base re-read "first landed inside the
scheduled-auto-merge branch only", and cold review found scheduled+retarget denied while
immediate+retarget auto-GRANTED.
The scope index is built from `docker-build.yml` alone and deliberately NOT from the git index. An
earlier draft derived it with `tracked_paths` and broke the RELEASE PATH: `docker-build.yml`'s `scan`
job runs this guard inside a poisoned COPY of the tree where `git ls-files` exits 128, which is the
trap `test_ci_release_path_scan_job.py` documents in the very paragraph this change edits. A narrow
index is safe here because the failure direction is: the completeness claim is over the SNAPSHOT's
contexts, so a missing workflow makes a context UNRESOLVABLE and the guard strictly louder, never
quieter. That harness now stages the snapshot into its copy, named explicitly rather than by glob.
Rule selection was EXTRACTED, not copied. `scripts/lib/branch-rule-classifier.jq` is now loaded by
both the hook and the checker: this is a security predicate, and the hook's own history is the
argument — the docs-only file enumeration was written twice, four rounds of hardening landed on one
copy and never reached the other, and the copy with real authority ended up strictly weaker.
An earlier draft of the checker had its own blanket "any glob metacharacter anywhere" test, which
called a rule named `a[b` undecidable for `main`; the shared classifier's `superset` test does not.
A structural test now pins that both callers reach the one file, replacing a first attempt that
banned the token "gobwas" and flagged the comment saying the logic is DELEGATED.
Cold cross-family review (Codex) returned BLOCKED with seven findings; all are fixed and each fix's
red was witnessed:
* THE MUTATION PROOF WAS VACUOUS. The mutant was copied without the classifier it loads beside
itself, so it exited 2 with EMPTY stdout — and the assertion `stdout != "drift"` was satisfied by
that empty string. The proof never reached the clause, so the `MUTATION` grade was unsupported. It
now asserts the mutant's EXACT verdict `(0, "match")` and ships a positive control proving the tmp
layout reproduces the real one.
* Duplicate synthesized contexts silently overwrote in a dict. Two jobs sharing a `name:` collide on
one context string, the later wins, and the other's steps go unguarded — branch protection cannot
tell them apart either. Now a multimap with a uniqueness assertion.
* `enable_status_check: false` was type-checked and its VALUE ignored, so protection with checks
DISABLED certified as `match`. The comparison is now against the EFFECTIVE required set.
* The checker and snapshot were resolved through `$CLAUDE_PROJECT_DIR`. An env var is not a sound
input to a security decision — a wrong value pointing at a tree containing an executable checker
returns `match` about a different checkout. Both now bind to the hook-derived `$repo_root`, and the
missing-checker test runs a COPY of the hook, the only way to move that root.
* `unreadable` is a DECLARED class that fell to the catch-all, which told the operator the checker
had returned a word the hook did not understand. It has its own arm. Its test asserts on the
IMMEDIATE path: on the scheduled path the h10 arm answers first, and the first draft passed
without reaching the code it names.
* `scripts/lib/*.jq` was outside the remote-state population, whose claim is that it has no content
filter. Scope widened and the classifier given an `N/A` row.
* The release-path Blocker it raised was already fixed in this branch before the review landed.
A second cold review round (a different model family, and a re-review of the fix commit) returned
BLOCKED on two findings, both fixed:
* THE STATED INVARIANT WENT UNENFORCED THE MOMENT ONE SIDE STARTED MOVING. The file says "the
delimiter ban is WIDER than the marker set", and while both were literals edited together that
held. `MARKED_JOBS` now DERIVES, so on the day a fourth required context arrives the marker half
goes red, the author adds markers, CI greens — and that job's `run:` bodies were never checked for
the `${{` delimiter that CAUSES the drop the markers detect. `DELIMITER_BAN_JOBS` is now derived
from `MARKED_JOBS`, and the property is asserted so a future edit back to a literal cannot
re-open it silently. Measured: a planted 4th context grows both sets together.
* `docs/guard-inventory.md`'s row claimed the context index came from the git index — true of the
reverted draft, false of the shipped code, and in the artifact whose job is to be trusted about
exactly that. Nothing asserts that column, so it would not have self-corrected.
The cross-workflow half moved to its own guard, `scripts/tests/test_ci_status_context_uniqueness.py`.
It has to be separate: a cross-workflow claim needs the whole workflow population, whose authoritative
source is the git index, and the dropped-step guard cannot touch the index because the release-path
`scan` job runs it in a non-git copy. Split, each half gets the right source. It is MUTATION-graded
via the checker-guard shape — the guard IS a test, so disarming its own assertion makes it absent
rather than red, and the declared mutation goes into the guarded ARTIFACT instead: a workflow job
renamed to collide with the required `Build & test (.NET)` context, which is precisely the evasion.
Folded in from the same round: the arm now runs only for the repo the snapshot NAMES (it was
comparing any repo's live contexts against this repo's mirror — `server-management` returns `[]`,
which classified as `nomatch` and would have reported a flatly false finding about it); the hook
guards the shared classifier's readability itself, rather than inheriting a `jq -f` failure that
blamed the payload for a missing local file; `unreadable` and the usage exits match their stated
contracts; and `docs/ci-cd.md` records that the immediate merge path now needs a repo-admin-scoped
credential, which it did not before.
A fifth cold round returned MERGEABLE — no Blocker, no High, no Medium. It independently re-ran
round 4's exact re-inlining attack and confirmed the suite now reddens on it, disarmed twelve further
clauses and found each reddened only its own named test, and verified #857 survived the rebase
intact. Its one Low and two nits were fixed rather than deferred, because the first made a claim in
this message false: the sweep of the retracted "or a dated staleness marker" wording had covered
`docs/` but missed a quoting site in `scripts/tests/test_optional_request_members.py`. The checker's
stderr is no longer discarded either — a usage error used to reach the operator as "returned
'nothing'", naming no cause.
Two prose overclaims of my own were corrected rather than defended: "neither half is checkable
alone; composed, they are" now names the reconciliation as ONE bypassable client-side hook, and
"updating it to match the server makes CI RED" now says ADD, because hand-NARROWING the snapshot
narrows the derived scope and stays green exactly as the literal it replaced did.
A third cold round (the cross-family reviewer had hit a usage limit, so this was the strongest
available reasoning on a cold brief) returned BLOCKED. Its findings, all fixed and each red
witnessed:
* THE CENTRAL CLAUSE OF THIS CHANGE SHIPPED WITH NO PROOF. `assert not unaccounted` is what makes a
fourth required check impossible to add in silence, and the decision record sells it as exactly
that — but disarming it and planting a fourth context left the derived scope silently behind with
51 tests still green. `_derive_marked_jobs` now takes an injectable context list SOLELY so that
clause and the anti-vacuity clause beside it can be proven; production calls it with no argument.
* The `enable_status_check:false` fix from round 2 had ZERO coverage: removing that one clause
returned `match` for disabled protection and all 71 tests stayed green, because the `rule()`
helper's `enabled=` parameter had no call site that ever passed False — this repo's own recorded
"a fixture that omits a field tests only the default".
* A FIFTH test passing for the wrong reason: the missing-checker test asserted the substring
`check-required-contexts.sh`, which the CATCH-ALL arm's reason also contains, so deleting the
`[ ! -x ]` clause left it green — two guards masking each other (#685). It now asserts wording
unique to the arm AND that no branch-protection fetch happened, which only the early clause
prevents.
* The repo gate was byte-exact while Gitea resolves owner/repo case-insensitively (verified live:
`/repos/TIMOTHY/ErsatzTV` answers 200), so a case variant would skip the freshness arm entirely —
the gate failing open on a spelling.
* The new `branch_protections` read had no `docs/remote-state-inventory.md` row, and the checker's
row delegated its fetch to "the caller's row" that did not exist. Added, graded `UNSAFE-KNOWN`
with what it authorizes (nothing: every outcome is `ask` or silence).
A fourth cold round returned BLOCKED on two independent grounds, both fixed:
* THE SEVENTH TEST PASSING FOR THE WRONG REASON, and the one that mattered most: the "one classifier,
two callers" property was asserted with a SUBSTRING over source text, which a COMMENT satisfies.
Review proved it by re-inlining a byte-identical copy of the classifier into the hook with a
comment naming the shared file — the entire suite stayed green at exactly 1062. The byte-identical
inline is the refactor that matters, precisely because it agrees on day one; a weakened copy would
have been caught behaviourally. Both callers are now proven by SWAPPING the shared file for a
sentinel that can only answer `undecidable` and requiring their behaviour to change. Re-running
review's exact attack against the fixed suite now reddens it.
* THE HEAD WAS STALE AND THE REBASE WAS RED. `origin/main` had moved to #857, which touches five of
the same files; this branch did not apply cleanly, and the guard-inventory counts were computed off
the dead base. Rebased onto `45b17e58e`, counts resolved to 42/6/21 and 20/6/16, the generated
catalog REGENERATED rather than hand-resolved. The earlier green vouched for a tree that no longer
existed.
Also fixed from that round: the required-`branch` clause was masked by the mismatch guard one line
below it, so its test could not see it (#685 again — it now asserts the arm-unique stderr); three of
the four freshness arms had no proof and were carried by the catch-all; the checker's own
classifier-readability guard was the untested twin of a fix the hook made loudly; and the
unaccounted-context diagnostic offered a remediation that is impossible for a context produced by
another workflow.
Two coverage overclaims were corrected rather than defended: the arm does NOT run for a docs/process
-only PR, so a PR editing only the snapshot — the narrowing direction — never triggers it, and it
could not have judged one anyway, since it compares the live server against the snapshot in the local
checkout rather than the version the PR proposes.
So SEVEN of this change's own tests were caught passing for the wrong reason across four rounds, and
the pattern is one class — the mutation proof (a mutant that never ran), the `unreadable` arm (an earlier arm answering), the
never-deny assertion, and the repo gate (a stub serving a matching payload either way). Every one
asserted the ABSENCE of something. They now assert a specific positive outcome, and each fix's red
was witnessed by disarming the clause it names.
Self-found in the same round: the snapshot names its own `branch`, and a mismatch against `--branch`
is now a usage error rather than a class — comparing one branch's live contexts against another's
mirror would otherwise produce a confident verdict about a pair never meant to be compared.
Verified: the mutation is witnessed, not asserted — unmutated reports `drift` on an added context,
the mutant with the set comparison disarmed reports `match`, a permanent no-op. The checker was run
against the LIVE branch_protections payload (`match`) and every other class exercised. Full
`scripts/tests` suite green; ruff check + format clean.
Sweep by subject: every site claiming this gap is open was updated — `docs/guard-inventory.md`,
`test_ci_release_path_scan_job.py`, `test_guard_populations_derive_from_git.py`,
`test_ci_image_pin_population.py` (whose remaining `TOOLCHAIN_JOBS` literal is #789), and by clause,
the rule's "or a dated staleness marker" alternative, which the new guidance rejects.
fixes #787
Decisions-Edit: yes
Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
This commit is contained in:
@@ -472,25 +472,17 @@ else
|
||||
# directions, and it is rare in practice: as of 2026-08-19 this repo's only rule is the plain
|
||||
# name `main`, which the classifier resolves to `exact` on every run. That is a dated
|
||||
# observation about mutable remote config, not a property to rely on.
|
||||
bp_verdict=$(printf '%s' "$bp_list" | jq --arg b "$base_ref" -c '
|
||||
def esc: gsub("(?<c>[.+?^${}()|\\[\\]\\\\])"; "\\" + .c);
|
||||
def offs: [match("[*?\\[\\]{}\\\\]"; "g").offset];
|
||||
def superset: . as $n | (offs) as $o
|
||||
| ($n[0:$o[0]] | esc) + ".*" + ($n[($o[-1]+1):] | esc);
|
||||
def nonascii: explode | any(. > 127);
|
||||
. as $rules | $b as $base |
|
||||
($rules | map(select((.branch_name // .rule_name // "") as $n
|
||||
| (($n|offs|length) == 0)
|
||||
and (($n|ascii_downcase) == ($base|ascii_downcase))))) as $exacts |
|
||||
(($base|nonascii) or ($rules | any((.branch_name // .rule_name // "") as $n
|
||||
| ($n|offs|length) == 0 and ($n|nonascii)))) as $unfoldable |
|
||||
if ($rules | any((.branch_name // .rule_name // "") as $n
|
||||
| (($n|offs|length) > 0)
|
||||
and ($base | test("^" + ($n|superset) + "$")))) then {verdict:"undecidable"}
|
||||
elif $unfoldable then {verdict:"undecidable"}
|
||||
elif ($exacts | length) > 1 then {verdict:"undecidable"}
|
||||
elif ($exacts | length) == 1 then {verdict:"exact", rule:($exacts | first)}
|
||||
else {verdict:"none"} end' 2>/dev/null || true)
|
||||
# The classifier is a FILE now (ersatztv#787), so its absence is a new failure mode: `jq -f` on a
|
||||
# missing program exits 2 with empty stdout, which reaches the `*)` arm below and asks that "this
|
||||
# repo's branch-protection rules came back in a shape this hook could not parse" — blaming the
|
||||
# payload for a missing local file. That is precisely the states-a-cause-that-did-not-happen defect
|
||||
# the two comments beside that arm were written to fix, so it is checked here rather than inherited.
|
||||
classifier="$repo_root/scripts/lib/branch-rule-classifier.jq"
|
||||
if [ ! -r "$classifier" ]; then
|
||||
rm -f "$bp_file"
|
||||
decide ask "H6/H10 merge gate: the shared branch-protection rule classifier is missing or unreadable at $classifier, so which rule governs '$base_ref' — and therefore whether 'review-verdict/h10' is required on it — could not be derived (ersatztv#787). Restore the file, or confirm the required checks manually."
|
||||
fi
|
||||
bp_verdict=$(printf '%s' "$bp_list" | jq --arg b "$base_ref" -c -f "$classifier" 2>/dev/null || true)
|
||||
case $(printf '%s' "$bp_verdict" | jq -r '.verdict // ""' 2>/dev/null || true) in
|
||||
exact) bp=$(printf '%s' "$bp_verdict" | jq -c '.rule' 2>/dev/null || true); bp_code=200 ;;
|
||||
undecidable) rm -f "$bp_file"
|
||||
@@ -614,6 +606,119 @@ case "$class" in
|
||||
decide ask "H10 merge gate: unrecognized verdict classification '$class' for PR #$pr. Confirm the review covered the latest commit ($short) before merging." ;;
|
||||
esac
|
||||
|
||||
# --- (d) Guard-scope freshness (ersatztv#787): the committed mirror of `main`'s required status
|
||||
# checks must still match the server. ------------------------------------------------------
|
||||
# ORDERED LAST, and that is a severity argument rather than a stylistic one. Every check above
|
||||
# can DENY; this one can only ever downgrade an otherwise-satisfied auto-grant to a prompt. Run
|
||||
# earlier it would preempt those verdicts and report a stale guard scope at a reader whose merge
|
||||
# is blocked for a completely different and more serious reason, and it would ask on payloads the
|
||||
# checks above are about to reject anyway. Placed here it is also PAST the point where the two
|
||||
# merge paths converge, so it covers both without duplicating anything.
|
||||
# `scripts/tests/test_ci_dropped_step_guard.py` DERIVES which jobs must carry per-step execution
|
||||
# markers from `.gitea/required-status-contexts.json`, because its CI job checks out with
|
||||
# `persist-credentials: false` and cannot ask Gitea. That makes the snapshot the single
|
||||
# hand-maintained input in the chain: a fourth required context added on the server leaves the
|
||||
# snapshot — and therefore the guard's scope — silently behind, which is the whole of #787.
|
||||
#
|
||||
# THIS RUNS ON BOTH MERGE PATHS, deliberately, and it is placed here rather than beside the
|
||||
# branch-protection read in the scheduled-auto-merge branch for that reason.
|
||||
#
|
||||
# WHAT IT DOES NOT COVER, said here rather than left to be discovered: a PR whose changed files are
|
||||
# all docs/process — `.gitea/` included — exits at the docs-only passthrough far above, so this arm
|
||||
# never runs for it. A PR that edits ONLY `.gitea/required-status-contexts.json` is docs-only BY
|
||||
# CONSTRUCTION, and that is exactly the snapshot-NARROWING direction the decision record names as
|
||||
# this design's residual. Excluding that path from the allow-list would not buy the protection it
|
||||
# looks like it would: this arm compares the live server against the snapshot in the LOCAL CHECKOUT,
|
||||
# not against the version the PR proposes, so it cannot see a narrowing that has not landed yet.
|
||||
# What does hold is that the passthrough is a passthrough — a human prompt, never an auto-grant —
|
||||
# which is the `.gitea/` treatment ersatztv#317 asked for. That read is inside
|
||||
# `else` (mwcs = true) and never executes on an immediate merge, which is the common case; hanging
|
||||
# the freshness check off it would fire it only when an auto-merge is armed. This file already
|
||||
# records that exact defect one section up — the base re-read "first landed inside the
|
||||
# scheduled-auto-merge branch only", and cold review found scheduled+retarget denied while
|
||||
# immediate+retarget auto-GRANTED. Same shape, so it is not repeated here.
|
||||
#
|
||||
# It reads `main` (the branch the snapshot names), NOT `$base_ref`. That is a DIFFERENT question
|
||||
# from the one the scheduled branch asks — "is review-verdict/h10 required on the base I am merging
|
||||
# into" — so this is not a second copy of that classifier and the two cannot drift into disagreeing:
|
||||
# they consume different fields of different rules for different decisions.
|
||||
#
|
||||
# ASK, NEVER DENY. Drift does not make THIS merge unsafe: Gitea enforces the live required set
|
||||
# server-side, so a newly required context with no status blocks the merge on its own. What has gone
|
||||
# stale is a guard's scope — a different artifact, on a different clock. Denying would state
|
||||
# something false about the change in front of the reader. Every non-`match` class asks, so a
|
||||
# comparison that could not be made is surfaced rather than skipped (`unknown` is not `fine`).
|
||||
# ONE base for both the checker and the snapshot, and it is `$repo_root` — derived from this file's
|
||||
# own location — rather than `$CLAUDE_PROJECT_DIR`. Two reasons, and the second is the load-bearing
|
||||
# one. Resolving them from different roots would let the hook classify one checkout's snapshot with
|
||||
# another checkout's script, mismatched halves of a comparison whose whole job is to detect a
|
||||
# mismatch. And an ENV VAR is not a sound input to a security decision: a wrong value pointing at a
|
||||
# tree that happens to contain an executable checker returns `match` about a different checkout
|
||||
# entirely, which silently authorizes the grant. A missing path only asks, so the failure is quiet
|
||||
# exactly where it is worst.
|
||||
ctx_base="$repo_root"
|
||||
ctx_snapshot="$ctx_base/.gitea/required-status-contexts.json"
|
||||
ctx_script="$ctx_base/scripts/check-required-contexts.sh"
|
||||
|
||||
# THIS ARM IS ABOUT ONE REPO, and the merge tool is not. Every other check here reads
|
||||
# `$owner/$repo` from the tool input and is repo-agnostic; this one compares a HARDCODED branch
|
||||
# against a snapshot committed in THIS checkout. Merging a PR in another repo from a session opened
|
||||
# here would otherwise weigh that repo's live contexts against this repo's mirror and report a
|
||||
# confident, flatly false finding about it — measured: server-management returns `[]`, which
|
||||
# classifies as `nomatch`. So the snapshot names the repo it describes and the arm runs only for it.
|
||||
# An unreadable snapshot cannot answer "is this my repo?" either, so it asks rather than skipping.
|
||||
ctx_repo=$(jq -r 'if (.repo | type) == "string" then .repo else "" end' "$ctx_snapshot" 2>/dev/null || true)
|
||||
if [ -z "$ctx_repo" ]; then
|
||||
decide ask "H6 merge gate: $ctx_snapshot is missing, unreadable, or names no \`repo\`, so the dropped-step guard's scope could not be checked against branch protection — nor could it be established whether this snapshot even describes $owner/$repo (ersatztv#787). Restore the file, or check the required checks manually."
|
||||
fi
|
||||
# CASE-FOLDED, because Gitea resolves owner/repo case-insensitively: verified live, both
|
||||
# `/repos/timothy/ersatztv` and `/repos/TIMOTHY/ErsatzTV` answer 200. A byte-exact compare would let
|
||||
# any case variant sail through every other arm and SKIP this one, so drift would go unreported with
|
||||
# no ask — the gate failing open on a spelling. The hook already treats case folding as
|
||||
# decision-relevant one section up, where `MAIN` vs `main` makes the governing rule undecidable.
|
||||
ctx_repo_fold=$(printf '%s' "$ctx_repo" | tr '[:upper:]' '[:lower:]')
|
||||
target_repo_fold=$(printf '%s' "$owner/$repo" | tr '[:upper:]' '[:lower:]')
|
||||
if [ "$ctx_repo_fold" = "$target_repo_fold" ]; then
|
||||
if [ ! -x "$ctx_script" ]; then
|
||||
decide ask "H6 merge gate: the required-contexts checker is missing or not executable at $ctx_script, so whether the dropped-step guard's scope still matches branch protection on 'main' could not be derived (ersatztv#787). Check it manually, or restore the script."
|
||||
fi
|
||||
bpf=$(mktemp) || decide ask "H6 merge gate: could not allocate a temp file to read branch protection for the guard-scope freshness check (ersatztv#787)."
|
||||
if [ -n "${ETV_GITEA_TOKEN:-}" ]; then
|
||||
ctx_code=$(curl -s -o "$bpf" -w '%{http_code}' -H "Authorization: token $ETV_GITEA_TOKEN" "$base_url/repos/$owner/$repo/branch_protections" 2>/dev/null || true)
|
||||
else
|
||||
ctx_code=$(curl -s -o "$bpf" -w '%{http_code}' -u "$ETV_GITEA_BASICAUTH" "$base_url/repos/$owner/$repo/branch_protections" 2>/dev/null || true)
|
||||
fi
|
||||
if [ "$ctx_code" = "200" ]; then
|
||||
# stderr is KEPT, not sent to /dev/null. The checker exits 2 with a diagnostic on a usage error —
|
||||
# an unreadable snapshot, a branch mismatch, a missing classifier — and discarding it made all of
|
||||
# those arrive at the operator as the catch-all's "returned 'nothing'", which names no cause. That
|
||||
# is the same states-a-cause-that-did-not-happen shape this arm was careful about elsewhere.
|
||||
ctx_class=$("$ctx_script" --branch main --snapshot "$ctx_snapshot" < "$bpf" 2>"$bpf.err" || true)
|
||||
ctx_diag=$(tr '\n' ' ' < "$bpf.err" 2>/dev/null | cut -c1-300 || true)
|
||||
else
|
||||
ctx_class=readfail
|
||||
ctx_diag=""
|
||||
fi
|
||||
rm -f "$bpf" "$bpf.err"
|
||||
case "$ctx_class" in
|
||||
match) : ;;
|
||||
drift)
|
||||
decide ask "H6 merge gate: the required status checks on 'main' no longer match .gitea/required-status-contexts.json (ersatztv#787). scripts/tests/test_ci_dropped_step_guard.py derives its marked-job scope from that snapshot, so until it is reconciled a required context may have NO dropped-step guard — a step the runner drops would conclude success and take that check green having done no work (ersatztv#756). Re-read the live list and update the snapshot in a PR (the guard will then demand markers for any newly required job, or an ACCOUNTED_ELSEWHERE entry naming what covers it). This does not make the merge in front of you unsafe — Gitea enforces the live required set server-side — so approve if you have judged it unrelated." ;;
|
||||
nomatch)
|
||||
decide ask "H6 merge gate: no branch-protection rule governs 'main' at all, so the required status checks the dropped-step guard scopes itself to could not be confirmed (ersatztv#787). Branch protection on 'main' is what makes 'review-verdict/h10' load-bearing (ersatztv#743) — check it before merging." ;;
|
||||
undecidable)
|
||||
decide ask "H6 merge gate: a glob branch-protection rule could govern 'main', so which rule's required contexts to compare against .gitea/required-status-contexts.json is not derivable without reimplementing Gitea's matcher (ersatztv#787). Confirm the required checks manually." ;;
|
||||
unreadable)
|
||||
decide ask "H6 merge gate: branch protection for 'main', or .gitea/required-status-contexts.json itself, came back in a shape the required-contexts checker could not consume, so whether the dropped-step guard's scope is still current is unknown (ersatztv#787). Check the rules and the snapshot manually." ;;
|
||||
readfail)
|
||||
decide ask "H6 merge gate: could not read branch protection for the guard-scope freshness check (HTTP '${ctx_code:-none}' — Gitea unreachable, or these credentials lack the repo-admin scope that endpoint needs), so whether .gitea/required-status-contexts.json is still current is unknown (ersatztv#787). Confirm the required checks on 'main' manually." ;;
|
||||
*)
|
||||
decide ask "H6 merge gate: the required-contexts checker returned '${ctx_class:-nothing}', which is not a class this hook understands, so the dropped-step guard's scope could not be confirmed against branch protection (ersatztv#787).${ctx_diag:+ It said: ${ctx_diag}}Check scripts/check-required-contexts.sh." ;;
|
||||
esac
|
||||
fi # end of the guard-scope freshness arm (opened at `if [ "$ctx_repo_fold" = ... ]` above). The
|
||||
# body is left unindented to match the rest of this file, which is flat throughout; the marker
|
||||
# is here because the block is long enough that its extent is otherwise easy to misread.
|
||||
|
||||
if [ "$class" = "positive" ]; then
|
||||
# (a) CI + (b) all Done-when ticked + (c) positive verdict @ current head -> SATISFIED. Auto-grant.
|
||||
# The reason string must not claim more than was actually checked: on the merge_when_checks_succeed
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"repo": "timothy/ersatztv",
|
||||
"branch": "main",
|
||||
"read_on": "2026-08-27",
|
||||
"source": "GET /repos/timothy/ersatztv/branch_protections -> the rule governing `main` -> status_check_contexts",
|
||||
"why": "ersatztv#787. The committed mirror of the required status checks on `main`. It exists because the guards that make a required context trustworthy run in `pr-checks.yml::script-tests`, which checks out with persist-credentials:false and holds no Gitea credential, so it cannot ask the server. scripts/tests/test_ci_dropped_step_guard.py DERIVES its marked-job scope from `contexts` rather than repeating it as a literal, and scripts/check-required-contexts.sh compares this list against the live one wherever a credential does exist. Editing `contexts` by hand without re-reading the server is the one move that defeats both. The `repo` field exists because the merge-consent hook fires for whatever owner/repo the merge tool was called with: without it, merging a PR in another repo from an ersatztv session compares that repo's live contexts against THIS repo's mirror and reports a confident, flatly false finding about it.",
|
||||
"contexts": [
|
||||
"Build ErsatzTV Image / Build & test (.NET) (pull_request)",
|
||||
"Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request)",
|
||||
"review-verdict/h10"
|
||||
]
|
||||
}
|
||||
@@ -1215,6 +1215,38 @@ and the client-side Husky guards (H6/H11/H13) remain useful friction but were ne
|
||||
they are fail-open and `--no-verify` bypasses them. Tag pushes are unaffected (separate mechanism;
|
||||
`tag_protections` is empty), so the release cut in "Cutting a release" still works unchanged.
|
||||
|
||||
**A fourth arm: guard-scope freshness (ersatztv#787).** `scripts/tests/test_ci_dropped_step_guard.py`
|
||||
decides which CI jobs must carry per-step execution markers by DERIVING that scope from
|
||||
`.gitea/required-status-contexts.json`, the committed mirror of `main`'s required status checks. It
|
||||
derives rather than repeating a literal because its own CI job, `pr-checks.yml::script-tests`, checks
|
||||
out with `persist-credentials: false` and cannot ask Gitea — which leaves the snapshot as the one
|
||||
hand-maintained input in the chain. So the merge hook reconciles it: it reads
|
||||
`branch_protections`, hands the payload to `scripts/check-required-contexts.sh`, and **asks** (never
|
||||
denies) when the live list no longer matches. The comparison is against the EFFECTIVE required set:
|
||||
a rule with `enable_status_check: false` requires nothing whatever its `status_check_contexts`
|
||||
still holds, so reading the list without the flag would certify disabled protection as current.
|
||||
|
||||
Two consequences worth knowing before you meet them. **The immediate merge path now reads a
|
||||
repo-admin-scoped endpoint**: before #787 `branch_protections` was fetched only when arming a
|
||||
scheduled auto-merge, so a credential without repo-admin scope kept its auto-grant on the common
|
||||
path and now gets a prompt there instead. That is the fail-closed direction and it is loud, but it
|
||||
is a real change in what the gate needs. **The arm runs only for the repo the snapshot names**
|
||||
(its `repo` field): the merge tool is called with whatever owner/repo is being merged, and judging
|
||||
another repo by this one's mirror would report a confident finding about a repo the mirror does not
|
||||
describe.
|
||||
|
||||
And what it does not reach: a docs/process-only PR exits at the hook's docs-only passthrough before
|
||||
this arm, so a PR editing only `.gitea/required-status-contexts.json` — the snapshot-narrowing
|
||||
direction — never triggers it. That passthrough is a human prompt rather than an auto-grant, and the
|
||||
arm could not have judged such a PR anyway, since it compares the live server against the snapshot in
|
||||
the local checkout rather than the one the PR proposes. Ask rather than deny because drift does not make that
|
||||
merge unsafe — Gitea enforces the live required set server-side — while a stale guard scope means a
|
||||
required context may have no dropped-step guard at all. The arm runs **last**, so it can only
|
||||
downgrade an otherwise-satisfied auto-grant and never masks a more serious refusal, and it runs on
|
||||
**both** merge paths: the pre-existing branch-protection read sits inside the
|
||||
`merge_when_checks_succeed` branch and never fires on an immediate merge. Rule selection is shared,
|
||||
not copied — both the hook and the checker load `scripts/lib/branch-rule-classifier.jq`.
|
||||
|
||||
**The hole it closes.** `pretooluse-merge-consent.sh` proves its three consent conditions at the
|
||||
moment the merge tool is called. Pass `merge_when_checks_succeed=true` and Gitea performs the merge
|
||||
*later*, against whatever head is green then — while the Done-when and review-verdict checks were
|
||||
|
||||
@@ -210,7 +210,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `testing.enumerating-guard-identity-not-position` | A guard that cross-checks a hand-reviewed registry against call sites discovered across the whole repo must key each entry on properties INTRINSIC to the site — file, kind, and the value source text — and never on its absolute line or column. A registry keyed on position is a function of every other file in the repo, so a branch that never touches the guard can invalidate it; and because each PR is green against its own base, that failure is structurally invisible pre-merge and lands on `main` after review and after the merge gate. Dropping the position keeps every mutation the guard exists for — a NEW site, a REMOVED site and a CHANGED value each still fail, since each changes the identity multiset — and costs exactly ONE case, which must be stated rather than implied: a SAME-IDENTITY SUBSTITUTION within one file (delete a registered site, add a different unreviewed one with the same kind and value token, net-zero count) now passes. A REPORTED failure still prints the discovered line:column, because identity and diagnostics need not share a format. Comparison stays a MULTISET count rather than set membership, so two sites in one file sharing an identity must be discovered exactly that many times and a third occurrence still fails. A SCANNER test that asserts real AST positions against FIXED inline fixtures is the opposite case and keeps its line/column identity — it has no churn, because its input does not move. | 2026-07-27 | [link](records/testing/enumerating-guard-identity-not-position.md) |
|
||||
| `testing.fix-ships-a-witnessed-red-test` | A commit claiming to fix something may carry a `Proves: <pytest selector>` trailer; when it does, `scripts/prove-fix.sh` must show that selector GREEN with the fix and RED with the code side reverted, and CI enforces it per-PR. The trailer is opt-in — an unproven commit is allowed — but a claimed proof that does not hold fails the build. | 2026-08-16 | [link](records/testing/fix-ships-a-witnessed-red-test.md) |
|
||||
| `testing.full-replace-asserts-field-list` | Any path that writes a WHOLE entity or a WHOLE child collection — a PUT-replace handler, a hand-built request object, a test comparer standing in for one — derives its field list from the authoritative type and asserts SET EQUALITY against it, rather than enumerating the fields by hand. A hand-written list is correct on the day it is written and structurally unable to report the day it stops being: the field that drifts is the one nobody wrote a line for, so no amount of care in the existing lines can reach it. The failure is silent by construction — a full replace with a field omitted returns HTTP 200 and destroys that field's value (#754 drifted from a 28-property DTO by one and cleared it; the symptom arrived hours later as missing pixels). SECOND CLAUSE, separable from the first: where a replaced child row carries state keyed to its identity — progression, ordering, an enumerator position — the handler RECONCILES BY ID rather than delete-and-reinsert, because reinsertion silently resets state a client never asked to touch (#252: a schedule PUT reset fill-group progression; #500: a dedup fix became permanent data loss because the add filter and the remove filter used different keys, so the two halves must agree on the key). Delete-and-reinsert is acceptable ONLY where no such state exists, and that emptiness is a fact about today's schema that a later feature can silently invalidate — so record it where the handler is, dated, rather than leaving it to be re-derived. The canonical worked example is `ToolCatalogTests.Every_Write_Tool_Should_Declare_Exactly_Its_OpenApi_Request_Body_Fields`, which reads the accepted fields from the generated OpenAPI document and compares both directions. ON THE SPA SIDE the same rule is enforced by the TYPE SYSTEM rather than by a test: a full-replace body is built as `Complete<T>` (`web/src/api/completeRequest.ts`), a mapped type that makes every member of a generated request type required, so a builder that omits one fails `npm run typecheck`. Annotate BOTH the API-wrapper parameter (so later callers inherit it) AND each construction site including every `.map` callback return type, because the excess-property check that catches a PHANTOM field fires only on a fresh literal in a contextually typed position and a generic `.map` callback is not one. Do not infer from "most builders already typecheck" that the gap is closed: a member is omittable exactly when it is absent from the schema `required` array in the ASP.NET-produced OpenAPI document, and two live cases (#807) sat unchecked inside a large majority of checked ones. | 2026-08-21 | [link](records/testing/full-replace-asserts-field-list.md) |
|
||||
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
|
||||
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check against that source, or the guard is complete within a scope that has silently gone stale. A DATED STALENESS MARKER does not discharge this and is no longer offered as an alternative (#787): a date records when someone last looked, so it goes stale in exactly the circumstance it exists to report, and nothing reddens when it does. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`. When a scope mirrors an authoritative source across a CREDENTIAL BOUNDARY, split the invariant at that boundary rather than dating a comment: commit the mirror verbatim, derive the scope from it offline, and reconcile the mirror against the live source wherever a credential exists — `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py` was this repo's residual gap until #787 closed it that way. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
|
||||
| `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) |
|
||||
| `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin <its-own-name> <label> <capture\|stream>` as its FIRST act, before anything reads stdin. Two records are appended per invocation — a `fire` record on entry and an `exit` record carrying the exit status and the decision — to a session-scoped JSONL log. THE DECISION IS READ FROM WHAT THE HOOK ACTUALLY EMITTED, never declared by the hook author: Claude Code hooks (`capture` mode) always exit 0 and communicate by PRINTING JSON, so their stdout is diverted and replayed, and the recorded decision is parsed from those bytes; git hooks (`stream` mode) decide by EXIT CODE and their stdout is live progress text a human is watching, so it is not diverted and the decision is the status. That split is not a tuning knob — capturing a slow pre-push hook's output would hold it back until the end and read as a hang, and inferring a git hook's decision from absent JSON would put the report back into the guessing business this record exists to end. The population is DERIVED from `.claude/hooks/*.sh` by `scripts/tests/test_hook_fire_log.py`, so a new hook is uninstrumented-and-red rather than silently unobserved, and the report lists every hook that EXISTS rather than every hook that appears in the log — a report built from the log alone can only show hooks that fired, which makes the never-fired hook, the one finding worth having, invisible. THE INSTRUMENTATION MUST BE INVISIBLE TO THE HARNESS, and this is the load-bearing half: it sits in the stdin and stdout path of the most authoritative guards in the repo, so a differential test drives EVERY hook with and without it over a payload matrix and demands byte-equal stdout and equal exit status. It fails OPEN in exactly one direction — if the log cannot be written the hook behaves exactly as before — because observability that breaks a guard is worse than the blindness it replaces. Two mechanical traps are pinned by tests rather than left to care: stdout must be replayed from the FILE, since `out=$(cat f)` strips trailing newlines and delivers a guard's JSON one byte short with no parser anywhere to complain; and stdin must never be slurped when it is a TTY, because an interactive `git commit` hands its hooks a terminal and `cat` would block forever, hanging the commit the instrumentation was added to observe. | 2026-08-14 | [link](records/testing/hook-reports-its-own-execution.md) |
|
||||
| `testing.live-e2e-prepush-timing` | Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab. | 2026-07-21 | [link](records/testing/live-e2e-prepush-timing.md) |
|
||||
|
||||
@@ -86,10 +86,13 @@ same sentence appeared in the inventory and was rewritten there first, which is
|
||||
survives a fix round. A row claiming otherwise would be exactly the overclaim this record warns
|
||||
about, and it is recorded here because the record's own deliverable made it on the first pass.
|
||||
|
||||
This is the same gap `testing.guard-derives-population-from-source` already flagged one directory
|
||||
over — `MARKED_JOBS` in `test_ci_dropped_step_guard.py` is a hand-written mirror of those same
|
||||
required contexts, annotated with a date. A dated comment is a claim about the past. Two
|
||||
independent guards now depend on that configuration; one of them checks it.
|
||||
`testing.guard-derives-population-from-source` flagged the same configuration one directory over —
|
||||
`MARKED_JOBS` in `test_ci_dropped_step_guard.py` mirrored those same required contexts behind a
|
||||
date. A dated comment is a claim about the past. #787 closed that half: the mirror is now a
|
||||
committed artifact the guard derives its scope from, and `scripts/check-required-contexts.sh`
|
||||
reconciles it against the live `status_check_contexts` from the merge-consent hook, on both merge
|
||||
paths. Two independent guards depend on that configuration and both now check it — this row's own
|
||||
residual is the one that remains, and it is stated above rather than closed.
|
||||
|
||||
**The deliverable's own population was wrong three times, and that is the most transferable part of
|
||||
this record.** The inventory is the detector, so its population *is* the guard. Round one filtered
|
||||
|
||||
@@ -5,8 +5,8 @@ status: active
|
||||
since: '2026-08-13'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down.'
|
||||
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
|
||||
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check against that source, or the guard is complete within a scope that has silently gone stale. A DATED STALENESS MARKER does not discharge this and is no longer offered as an alternative (#787): a date records when someone last looked, so it goes stale in exactly the circumstance it exists to report, and nothing reddens when it does. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`. When a scope mirrors an authoritative source across a CREDENTIAL BOUNDARY, split the invariant at that boundary rather than dating a comment: commit the mirror verbatim, derive the scope from it offline, and reconcile the mirror against the live source wherever a credential exists — `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py` was this repo''s residual gap until #787 closed it that way. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down.'
|
||||
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · scope mirror across a credential boundary · split the invariant at the boundary · synthesise contexts do not parse them · one classifier two callers · paths: `.gitea/required-status-contexts.json`, `scripts/check-required-contexts.sh`, `scripts/lib/branch-rule-classifier.jq` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
|
||||
mechanics: 'Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.'
|
||||
---
|
||||
|
||||
@@ -46,19 +46,39 @@ this record: the trigger is not the token `.Where(`, it is a completeness claim
|
||||
A rule stated as "no filters in guards" would flag the correct one and, once waved through twice,
|
||||
stop being read at all.
|
||||
|
||||
**Scope is not population, and this repo's best guard shows why the distinction matters.**
|
||||
`scripts/tests/test_ci_dropped_step_guard.py` is the model for deriving a population: `_marked(job)`
|
||||
reads the marked steps out of the parsed workflow, and
|
||||
**Scope is not population, and a scope that MIRRORS a source needs its own check.**
|
||||
`scripts/tests/test_ci_dropped_step_guard.py` models deriving a population: `_marked(job)` reads the
|
||||
marked steps out of the parsed workflow, and
|
||||
`test_the_guard_expects_EXACTLY_the_set_of_marked_keys_in_the_right_bucket` asserts set equality
|
||||
against the guard's own argv. Inside its scope it is exemplary. Its SCOPE, though —
|
||||
`MARKED_JOBS = ("test", "migrations")` — is a hand-written mirror of the required contexts on
|
||||
`main`, annotated `Read live on 2026-08-10`. Those contexts are machine-readable
|
||||
(`GET /repos/{owner}/{repo}/branch_protections` → `status_check_contexts`), and nothing in this repo
|
||||
compares the two: `grep -rn 'status_check_contexts\|branch_protection'` over `*.py`/`*.sh`/`*.yml`
|
||||
returns nothing. Verified 2026-08-13 — the live contexts still match the comment, so this is a
|
||||
latent gap, not a live defect. But add a fourth required context and it acquires no dropped-step
|
||||
guard, silently, and the file that models the rule is the one that cannot report it. A dated comment
|
||||
is a claim about the past; it is not a check.
|
||||
against the guard's own argv. Its SCOPE was long this repo's counter-example —
|
||||
`MARKED_JOBS = ("test", "migrations")`, a hand-written mirror of the required contexts on `main`
|
||||
annotated `Read live on 2026-08-10`, which nothing compared against the server. A dated comment is a
|
||||
claim about the past; it is not a check.
|
||||
|
||||
#787 closed it, and the shape generalises to any scope mirroring an authoritative source across a
|
||||
CREDENTIAL BOUNDARY. Split the invariant at that boundary and check each half where it can be
|
||||
checked: commit the mirror verbatim from the source (`.gitea/required-status-contexts.json`), derive
|
||||
the scope from that artifact OFFLINE — `pr-checks.yml::script-tests` checks out with
|
||||
`persist-credentials: false` and cannot ask Gitea — and reconcile the artifact against the live
|
||||
source wherever a credential does exist (`scripts/check-required-contexts.sh`, called from the
|
||||
merge-consent hook on both merge paths). Neither half is checkable alone; composed, they cover the
|
||||
gap — but say what the reconciliation actually is: ONE client-side hook, so a merge from the web UI,
|
||||
from a session without the hook, or without credentials never reconciles, and the residual is
|
||||
bounded staleness rather than none. This is still why "a snapshot only moves the hand-maintenance"
|
||||
is wrong here: the maintenance becomes a verbatim copy with an equality check on each side, and
|
||||
updating it to ADD a context makes CI red until the newly required job is marked or its disposition
|
||||
recorded. The direction is asymmetric and the asymmetry is the residual — hand-NARROWING the
|
||||
snapshot narrows the derived scope and stays green, exactly as the literal it replaced did.
|
||||
|
||||
Three details carry it. The derivation SYNTHESISES each candidate context string from the workflow
|
||||
(`<workflow name> / <job name> (<event>)`) and matches by equality rather than parsing, because the
|
||||
delimiters occur inside the names they delimit — `Build & test (.NET)` ends in a parenthesis of its
|
||||
own. A required context that maps to neither a job nor an explicit disposition is a HARD failure at
|
||||
import, which is what makes a fourth required check impossible to add silently. And the
|
||||
reconciliation refuses rather than guesses when a glob rule could govern the branch, delegating that
|
||||
question to the ONE classifier (`scripts/lib/branch-rule-classifier.jq`) the merge hook also loads —
|
||||
extracted, not copied, because two copies of a security predicate drift and the copy without
|
||||
authority is historically the one that goes stale.
|
||||
|
||||
**A weak detector is the symptom-keyed mistake, demonstrated rather than argued.** #774 wrote this
|
||||
rule and then, in the same change, built a guard that violated it: a test asserting set equality
|
||||
|
||||
@@ -137,7 +137,7 @@ The audit #806 asked for, recorded whichever way it came out, because "we looked
|
||||
| `test_workflow_persist_credentials.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — every `actions/checkout` in every workflow must drop the persisted credential | derived from the index from the start (#835). Shipped with NO exemption list, which is why it waited for #744: the one non-compliant checkout would otherwise have bought a permanent entry that keeps passing after its reason expires |
|
||||
| `web/src/api/pageSizeCallSites.guard.test.ts` | `web/src/**/*.{ts,tsx,mts,cts}` via `import.meta.glob` | **yes** — an unregistered discovered site fails | DEFERRED to #819, assessed not skipped. The glob is a documented workaround: `@types/node` is deliberately out of `tsconfig.app.json`, and wiring it in was tried and reverted (it leaked Node's `setTimeout` into the app project and broke three unrelated tests), so there is no `node:child_process` to reach the index from. Over-enumerates, so it fails loudly rather than going blind |
|
||||
| `test_ci_release_path_scan_job.py` | `.gitea/workflows/*.y*ml` + `scripts/**` | **no** — a fixture assembling a tmp harness, asserted about behaviour not membership | takes its file LIST from the index anyway, for hermeticity not completeness: `shutil.copytree` copied whatever was on disk, including untracked files and `scripts/__pycache__`, into a tree whose behaviour the probes then measure. Content still comes from the working tree. The copy is not a git repo, so the two files this step RUNS may not use the helper — see the fixture docstring |
|
||||
| `test_ci_dropped_step_guard.py` | the parsed workflow document | **no filesystem population at all** | unchanged; its residual is `MARKED_JOBS`, a SCOPE mirror of the required contexts on `main`, which #806 does not close |
|
||||
| `test_ci_dropped_step_guard.py` | the parsed workflow document (its OWN subject only — the release-path `scan` job runs it in a non-git copy, so it may not derive from the index) | **no filesystem population at all** | unchanged by #806; the SCOPE residual it left open — `MARKED_JOBS` as a hand-written mirror of the required contexts on `main` — was closed by #787, which derives it from `.gitea/required-status-contexts.json` and reconciles that snapshot against the server in `scripts/check-required-contexts.sh`. The cross-workflow half of that claim lives in `test_ci_status_context_uniqueness.py`, which CAN use the index |
|
||||
|
||||
`test_remote_state_inventory.py` fixed its own population under #778 and kept a private copy of the
|
||||
derivation; #806 folded it onto the shared one, so that module is covered by the proof above like
|
||||
@@ -176,6 +176,7 @@ recorded as unexamined rather than as cleared.
|
||||
| `scripts/build_decisions_catalog.py` | the `decisions-guard` job, on a stale catalog | GUARD | MUTATION | `test_build_catalog_check_path.py::test_MUTATION_disarming_the_stale_comparison_stops_detection` |
|
||||
| `scripts/check-doc-narrative.py` | nothing, by design (advisory `::warning::` only, exits 0 on every path (the error/degradation arms are defensive and unproven — see the record) — `docs.no-session-narrative` says a string predicate over prose may not be load-bearing) | TOOLING | NONE | — |
|
||||
| `scripts/check-kickoff-guard.sh` | the `decisions-guard` job, on a revived #237 reference | GUARD | NONE | — |
|
||||
| `scripts/check-required-contexts.sh` | the merge-consent hook, when `main`'s live required status checks no longer match `.gitea/required-status-contexts.json` (the snapshot `test_ci_dropped_step_guard.py` scopes itself to) | GUARD | MUTATION | `test_check_required_contexts.py::test_MUTATION_disarming_the_set_comparison_stops_every_drift_report` |
|
||||
| `scripts/check-review-verdict.sh` | the merge-consent hook's verdict classification | GUARD | BEHAVIOUR-ONLY | `test_check_review_verdict.py::test_falseopen_token_must_be_a_whole_word` |
|
||||
| `scripts/ci-detect-already-validated.sh` | nothing directly (feeds the skip gate) | GUARD | NONE | — |
|
||||
| `scripts/ci-detect-docs-only.sh` | nothing directly (feeds the skip gate) | GUARD | NONE | — |
|
||||
@@ -196,11 +197,13 @@ recorded as unexamined rather than as cleared.
|
||||
| `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_build_catalog_check_path.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_check_required_contexts.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_check_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_check_doc_narrative.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_ci_dropped_step_guard.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_ci_image_pin_population.py` | the `script-tests` job, when a container job loses its pin | GUARD | MUTATION | `test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED` |
|
||||
| `scripts/tests/test_ci_release_path_scan_job.py` | the `script-tests` job, on a weakened release-path scan job | GUARD | NONE | — |
|
||||
| `scripts/tests/test_ci_status_context_uniqueness.py` | the `script-tests` job, when two CI jobs synthesize the same status-check context (which branch protection cannot tell apart) | GUARD | MUTATION | `test_ci_status_context_uniqueness.py::test_no_two_jobs_synthesize_the_SAME_status_context` |
|
||||
| `scripts/tests/test_ci_toolchain_image_resolves.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_decisions_lib.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_decisions_validate.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
@@ -244,7 +247,7 @@ clause-provable and the entry is regraded.
|
||||
|
||||
## What the numbers say
|
||||
|
||||
40 guards, 6 tooling scripts, 20 proof files. **18 guards carry a mutation proof; 6 are
|
||||
42 guards, 6 tooling scripts, 21 proof files. **20 guards carry a mutation proof; 6 are
|
||||
behaviour-only; 16 have none.** These figures are asserted against the table by
|
||||
`test_the_summary_counts_match_the_table` — they were wrong in the first draft (28/4/6/3/19 against
|
||||
a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it,
|
||||
|
||||
@@ -5,7 +5,7 @@ that read, and whether the read is bound to something that cannot change underne
|
||||
|
||||
| Directory | Files |
|
||||
|---|---|
|
||||
| `scripts/` (recursive, **excluding `scripts/tests/`**) | `*.sh`, `*.py` |
|
||||
| `scripts/` (recursive, **excluding `scripts/tests/`**) | `*.sh`, `*.py`, `*.jq` |
|
||||
| `.claude/hooks/` | `*.sh` |
|
||||
| `.husky/` | all tracked files |
|
||||
| `.gitea/workflows/` | `*.yml`, `*.yaml` |
|
||||
@@ -69,6 +69,7 @@ classifications differ; otherwise the strictest applies and the Note names the e
|
||||
| `.claude/hooks/pretooluse-merge-consent.sh` — scheduled auto-merge (`merge_when_checks_succeed`) | `UNSAFE-KNOWN` | **Preflight, not a pin** — graded down by cold review, which was right: the hook's own comment concedes the branch-protection read pins nothing, so calling it `PINNED` contradicted this file's definition. What the hook proves is a snapshot; Gitea merges later; and since #778 it also verifies that `review-verdict/h10` is a required check, reading the repo's **full rule list** (never the by-name endpoint, which does no matching and knows nothing about precedence) — nothing can govern the base → deny, unreadable → ask, and a **glob rule that could govern it → ask**, because the hook deliberately does not reimplement Gitea's glob dialect. It also asks when two rule names fold equal, or when either name is non-ASCII, since Gitea's `EqualFold` is Unicode-aware and its rule precedence is not derivable here. That converts an unobserved assumption into an observed precondition and detects drift, but an admin can still weaken the protection *after* the read. Accepted, and the earlier wording here was **circular** — it said the residual was "closed one layer down" by the very branch protection an admin may have removed. It is not closed; it is BOUNDED, and the bound is a trust assumption that should be stated rather than dressed as a mechanism: everything on this path assumes repo-admin branch-protection config is not hostile. If protection is present at preflight and removed afterwards, an unreviewed head can merge, and nothing in this repo would detect it. What the check does buy is that the far commoner case — protection already weakened when the merge is attempted — stops being silent. A PreToolUse hook cannot add `head_commit_id` to the merge call, so it can never convert its own grant into a CAS — it can only refuse, which is what it now does. |
|
||||
| `.claude/hooks/pretooluse-merge-consent.sh` — base-retarget detection | `UNSAFE-KNOWN` | `recorded_base` parsed from the H10 status description is compared against the PR's live `.base.ref` (#632), and it deliberately compares the base **ref** rather than `base.sha` — the tip moves on every unrelated merge, so comparing it would deadlock every open PR. The residual is the same ABA the enumerator has: a name can be retargeted away and back, and the comparison cannot see it. Accepted because the alternative that CAN see it is the monotonic event count, which lives in the workflow that writes the enforced status rather than in this advisory hook. |
|
||||
| `.claude/hooks/pretooluse-merge-consent.sh` — `## Done-when` issue-body read | `UNSAFE-KNOWN` | The issue body carries an `updated_at` that is not used, so a box unticked between the read and the merge is invisible. Accepted: the only actor who can edit the issue is the one requesting the merge, so this is a self-inflicted race with no adversary and no silent-failure mode. |
|
||||
| `.claude/hooks/pretooluse-merge-consent.sh` — guard-scope freshness `branch_protections` read (ersatztv#787) | `UNSAFE-KNOWN` | A SECOND, separate read of the same endpoint, and the first one on the IMMEDIATE merge path: the pre-existing row above fires only when arming a scheduled auto-merge. Read-then-act with nothing pinning it — branch protection can change between this read and the merge Gitea performs. Accepted because of what the read AUTHORIZES, which is nothing: every outcome is `ask` or silence, never a grant and never a deny, so a stale or raced read can only cost a prompt. It is also skipped entirely unless the merge target is the repo `.gitea/required-status-contexts.json` names. The reconciliation it performs is bounded to merges made through this hook — a web-UI merge never runs it. |
|
||||
| `.claude/hooks/prepush-donewhen.sh` — `## Done-when` issue-body read | `UNSAFE-KNOWN` | No pin, but the hook's exit code gates the push synchronously — git blocks on this process. Blast radius is near zero regardless: `main` carries `enable_push: false` and `block_admin_merge_override: true`, so the direct push this hook exists to block is refused server-side for every account (#743). This is belt-and-braces over a path the server already refuses. |
|
||||
| `.claude/hooks/pretooluse-nav-guard.sh` | `N/A` | Reads only the proposed tool call's own parameters and decides synchronously; the `curl` mentions in the file are prose, not executed lines. |
|
||||
|
||||
@@ -111,6 +112,8 @@ classifications differ; otherwise the strictest applies and the Note names the e
|
||||
| `scripts/refresh-shared-checkout.sh` — `git fetch origin main`, then `git merge --ff-only origin/main` and a conditional `npm ci` | `UNSAFE-KNOWN` | Fetch-then-act with no re-check between the fetch and the merge. Accepted because every action is self-refusing or reversible: `--ff-only` fails harmlessly rather than diverging if the ref moved on, the script refuses outright when the tree is not clean `main` or is ahead or mid-rebase, and this is a developer-convenience checkout rather than a release or merge-authorization path — a stale read costs one extra fetch next run, never lost work. |
|
||||
| `scripts/lib/review-verdict-vocabulary.sh` | `N/A` | Reads nothing at all — a sourced declaration of the H10 verdict word list plus pure functions over it; it makes no call, local or remote. Listed because it is in this inventory's scope (`scripts/**/*.sh`), not because it has a remote-state shape. |
|
||||
| `scripts/check-review-verdict.sh` | `N/A` | Reads no live remote state itself — classifies a comments JSON payload supplied on stdin; the fetch belongs to the caller's row. |
|
||||
| `scripts/check-required-contexts.sh` | `N/A` | Reads no live remote state itself — classifies a `branch_protections` payload supplied on stdin; the fetch belongs to the caller's row (`.claude/hooks/pretooluse-merge-consent.sh` — guard-scope freshness). |
|
||||
| `scripts/lib/branch-rule-classifier.jq` | `N/A` | Reads nothing at all — a jq program over a `branch_protections` payload its caller supplies, resolving which protection rule governs a branch. Loaded by both `pretooluse-merge-consent.sh` and `check-required-contexts.sh`; every fetch belongs to those callers' rows. |
|
||||
| `scripts/decisions_validate.py` | `N/A` | Reads no live remote state — its `git log`/`show`/`ls-tree`/`merge-base` calls operate on refs the caller already checked out or passed via `--base`/`--head`, never a fetch. |
|
||||
| `scripts/prove-fix.sh` | `N/A` | Reads no live remote state — `git worktree add`/`rev-parse`/`diff-tree` operate on the local repository's own objects. |
|
||||
| `scripts/add-migration.sh` | `N/A` | Reads no live remote state — runs `dotnet ef migrations add` against local project files; implicit NuGet resolution is dependency supply-chain, out of this class per limit 1. |
|
||||
@@ -171,7 +174,7 @@ classifications differ; otherwise the strictest applies and the Note names the e
|
||||
reads rather than a field a script checks.
|
||||
|
||||
3. **Scope is hand-written; the population inside it is derived, with no content filter at all.**
|
||||
Scope is every file under `scripts/` (`*.sh`, `*.py`), `.claude/hooks/`, `.husky/` and
|
||||
Scope is every file under `scripts/` (`*.sh`, `*.py`, `*.jq`), `.claude/hooks/`, `.husky/` and
|
||||
`.gitea/workflows/` — a reviewed policy choice. Inside it, *every* file gets a row, and a file
|
||||
that reads no remote state earns an explicit `N/A` rather than silently staying out.
|
||||
|
||||
|
||||
Executable
+127
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# Compare the LIVE required status checks on a branch against the committed snapshot
|
||||
# (`.gitea/required-status-contexts.json`). ersatztv#787.
|
||||
#
|
||||
# WHY A SCRIPT AND NOT SIX LINES IN THE HOOK. `scripts/check-review-verdict.sh` was extracted from
|
||||
# `pretooluse-merge-consent.sh` in #629 for exactly this reason: while the classification lived in
|
||||
# the hook it had no tests, and three false-opens survived in it. This file is the same species —
|
||||
# a pure classifier over a JSON payload its CALLER fetched — so it is testable in isolation and
|
||||
# reads no remote state itself (`docs/remote-state-inventory.md` grades it `N/A` for that reason).
|
||||
#
|
||||
# WHAT IT IS FOR. `scripts/tests/test_ci_dropped_step_guard.py` derives its marked-job SCOPE from
|
||||
# that snapshot, because `pr-checks.yml::script-tests` checks out with `persist-credentials: false`
|
||||
# and cannot ask the server. That makes the snapshot the one hand-maintained input in the chain, so
|
||||
# it needs a reconciliation wherever a credential DOES exist. This is that reconciliation.
|
||||
#
|
||||
# Input : the `GET /repos/{owner}/{repo}/branch_protections` array, on stdin.
|
||||
# Output: exactly one class word on stdout.
|
||||
# match — the live contexts for the branch equal the snapshot, as SETS
|
||||
# drift — they were both read and they differ (the finding)
|
||||
# nomatch — no rule governs the branch at all (protection removed)
|
||||
# undecidable — a glob rule could govern the branch, so which rule applies is not derivable here
|
||||
# unreadable — the payload, or a field this decision consumes, is not the shape it must be
|
||||
# Exit : 0 having printed a class; 2 on a USAGE error (never a class, so a caller cannot mistake
|
||||
# a broken invocation for a finding).
|
||||
set -euo pipefail
|
||||
|
||||
branch=main
|
||||
snapshot=""
|
||||
snapshot_given=no
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
# `[ $# -ge 2 ]` before the shift, or a trailing `--branch` makes `shift 2` fail and `set -e`
|
||||
# exits 1 with no diagnostic — a status this script's own contract reserves for nothing, from a
|
||||
# path its usage arm was written to cover.
|
||||
--branch) [ $# -ge 2 ] || { echo "--branch needs a value" >&2; exit 2; }; branch=$2; shift 2 ;;
|
||||
--snapshot) [ $# -ge 2 ] || { echo "--snapshot needs a value" >&2; exit 2; }; snapshot=$2; snapshot_given=yes; shift 2 ;;
|
||||
*) echo "usage: $0 [--branch NAME] [--snapshot PATH] < branch_protections.json" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$branch" ] || { echo "--branch may not be empty" >&2; exit 2; }
|
||||
# An EXPLICIT empty --snapshot is a usage error, not a request for the default: falling back would
|
||||
# silently read a different file than the caller named.
|
||||
if [ -z "$snapshot" ] && [ "$snapshot_given" = yes ]; then
|
||||
echo "--snapshot may not be empty" >&2; exit 2
|
||||
fi
|
||||
if [ -z "$snapshot" ]; then
|
||||
snapshot="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/.gitea/required-status-contexts.json"
|
||||
fi
|
||||
[ -r "$snapshot" ] || { echo "snapshot not readable: $snapshot" >&2; exit 2; }
|
||||
command -v jq >/dev/null 2>&1 || { echo "jq is required" >&2; exit 2; }
|
||||
|
||||
payload=$(cat)
|
||||
|
||||
# The snapshot is validated to the SAME depth it is consumed at. A `contexts` that is present but
|
||||
# holds a non-string would otherwise compare unequal against a well-formed live list and be reported
|
||||
# as `drift` — a confident finding derived from a payload that was never understood, which is the
|
||||
# shape this repo has fixed twice in the merge hook (a `// []` default that fired on `false`, and an
|
||||
# object-typed validation that never checked its members).
|
||||
snap=$(jq -c 'if (.contexts | type) == "array" and (all(.contexts[]; type == "string"))
|
||||
then (.contexts | sort | unique) else "BAD" end' "$snapshot" 2>/dev/null || true)
|
||||
if [ -z "$snap" ] || [ "$snap" = '"BAD"' ] || [ "$snap" = "null" ]; then
|
||||
echo unreadable; exit 0
|
||||
fi
|
||||
|
||||
# THE SNAPSHOT NAMES ITS OWN BRANCH, and a mismatch against `--branch` is a USAGE error rather than a
|
||||
# class: comparing one branch's live contexts against another branch's mirror would produce a
|
||||
# confident `match` or `drift` about a pair that was never meant to be compared. Exiting 2 is the
|
||||
# only report that cannot be mistaken for a finding.
|
||||
# REQUIRED, not merely cross-checked when present. An absent, empty or non-string `branch` would
|
||||
# otherwise let a contexts list be compared against ANY `--branch`, which is the same defect the
|
||||
# mismatch check exists to prevent, reached by omission instead of by disagreement.
|
||||
snap_branch=$(jq -r 'if (.branch | type) == "string" then .branch else "" end' "$snapshot" 2>/dev/null || true)
|
||||
if [ -z "$snap_branch" ]; then
|
||||
echo "snapshot names no branch (a non-empty string \`branch\` is required): $snapshot" >&2; exit 2
|
||||
fi
|
||||
if [ "$snap_branch" != "$branch" ]; then
|
||||
echo "snapshot describes branch '$snap_branch' but --branch is '$branch'" >&2; exit 2
|
||||
fi
|
||||
|
||||
# WHICH RULE GOVERNS THE BRANCH is not decided here. That question — Gitea's Priority ordering, its
|
||||
# gobwas/glob dialect, case folding, non-ASCII names — is answered by the ONE classifier both this
|
||||
# script and `pretooluse-merge-consent.sh` load. Reimplementing it here would be the second copy of a
|
||||
# security predicate that the merge hook's own history argues against, and it would be the copy
|
||||
# without authority, which is historically the one that goes stale.
|
||||
classifier="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/branch-rule-classifier.jq"
|
||||
[ -r "$classifier" ] || { echo "classifier not readable: $classifier" >&2; exit 2; }
|
||||
|
||||
verdict=$(printf '%s' "$payload" | jq --arg b "$branch" -c -f "$classifier" 2>/dev/null || true)
|
||||
case $(printf '%s' "$verdict" | jq -r '.verdict // ""' 2>/dev/null || true) in
|
||||
none) printf 'nomatch\n'; exit 0 ;;
|
||||
undecidable) printf 'undecidable\n'; exit 0 ;;
|
||||
exact) : ;;
|
||||
# jq threw, or the payload was not an array the classifier could read. The comparison did NOT
|
||||
# happen, and `unreadable` is the class that says exactly that — never a silent success.
|
||||
*) printf 'unreadable\n'; exit 0 ;;
|
||||
esac
|
||||
|
||||
# The rule's own shape is validated to the depth it is CONSUMED at, not merely as an object. An
|
||||
# `enable_status_check` that arrived as the STRING "true", or a contexts list holding a non-string,
|
||||
# would otherwise compare unequal against a well-formed snapshot and be reported as `drift` — a
|
||||
# confident finding derived from a payload that was never understood, which is the one-level-down
|
||||
# swallow the merge hook has had to fix twice.
|
||||
#
|
||||
# `null`/absent contexts is NOT a read failure: it legitimately means "none required", so against a
|
||||
# non-empty snapshot it is the FINDING that every required context was removed.
|
||||
class=$(printf '%s' "$verdict" | jq -r --argjson snap "$snap" '
|
||||
.rule as $rule
|
||||
# The EFFECTIVE required set, not the raw list. `enable_status_check: false` means Gitea requires
|
||||
# nothing on this branch whatever `status_check_contexts` still holds, so type-checking that flag
|
||||
# and then ignoring its VALUE certifies disabled protection as a current mirror — reporting `match`
|
||||
# on a branch that is not gating anything at all.
|
||||
| if ($rule.enable_status_check | type) != "boolean" then "unreadable"
|
||||
elif ($rule.enable_status_check | not) then (if ($snap | length) == 0 then "match" else "drift" end)
|
||||
elif (($rule | has("status_check_contexts")) | not) or ($rule.status_check_contexts == null)
|
||||
then (if ($snap | length) == 0 then "match" else "drift" end)
|
||||
elif ($rule.status_check_contexts | type) != "array"
|
||||
or any($rule.status_check_contexts[]; type != "string") then "unreadable"
|
||||
else (if ($rule.status_check_contexts | sort | unique) == $snap then "match" else "drift" end)
|
||||
end' 2>/dev/null || true)
|
||||
|
||||
case "$class" in
|
||||
match|drift|nomatch|undecidable|unreadable) printf '%s\n' "$class" ;;
|
||||
# jq threw, or produced a word this contract does not define. Either way the comparison did NOT
|
||||
# happen, and saying so is the only safe report — `unreadable` is the class that means exactly
|
||||
# "no comparison was made", never a silent success.
|
||||
*) printf 'unreadable\n' ;;
|
||||
esac
|
||||
@@ -0,0 +1,36 @@
|
||||
# THE governing-branch-protection-rule classifier, as ONE program with two callers (ersatztv#787).
|
||||
#
|
||||
# Input : the `GET /repos/{owner}/{repo}/branch_protections` array. Arg `$b`: the branch to resolve.
|
||||
# Output: {verdict:"exact", rule:<the rule>} | {verdict:"undecidable"} | {verdict:"none"}
|
||||
#
|
||||
# It lived inline in `.claude/hooks/pretooluse-merge-consent.sh` until #787 needed the same question
|
||||
# answered for `main` from a second caller. It was EXTRACTED rather than copied: this is a security
|
||||
# predicate, and that hook's own history is the argument — the docs-only file enumeration was written
|
||||
# twice, four rounds of hardening landed on one copy and never reached the other, and the copy with
|
||||
# real authority ended up strictly weaker than the copy without. Two copies of a security predicate
|
||||
# drift; one cannot.
|
||||
#
|
||||
# DO NOT "simplify" the undecidable-first ordering. Gitea picks the governing rule with
|
||||
# `GetFirstMatched` over a list sorted by Priority, then by plain-name-ness, so a GLOB rule with a
|
||||
# better Priority outranks an exactly-named one. Preferring the exact rule would inspect a rule Gitea
|
||||
# might not be applying. `superset` deliberately over-approximates a glob (literal prefix + `.*` +
|
||||
# literal suffix) instead of reimplementing gobwas/glob: refusing whenever a glob COULD match is
|
||||
# sound without knowing the dialect, which is the only claim either caller makes.
|
||||
def esc: gsub("(?<c>[.+?^${}()|\\[\\]\\\\])"; "\\" + .c);
|
||||
def offs: [match("[*?\\[\\]{}\\\\]"; "g").offset];
|
||||
def superset: . as $n | (offs) as $o
|
||||
| ($n[0:$o[0]] | esc) + ".*" + ($n[($o[-1]+1):] | esc);
|
||||
def nonascii: explode | any(. > 127);
|
||||
. as $rules | $b as $base |
|
||||
($rules | map(select((.branch_name // .rule_name // "") as $n
|
||||
| (($n|offs|length) == 0)
|
||||
and (($n|ascii_downcase) == ($base|ascii_downcase))))) as $exacts |
|
||||
(($base|nonascii) or ($rules | any((.branch_name // .rule_name // "") as $n
|
||||
| ($n|offs|length) == 0 and ($n|nonascii)))) as $unfoldable |
|
||||
if ($rules | any((.branch_name // .rule_name // "") as $n
|
||||
| (($n|offs|length) > 0)
|
||||
and ($base | test("^" + ($n|superset) + "$")))) then {verdict:"undecidable"}
|
||||
elif $unfoldable then {verdict:"undecidable"}
|
||||
elif ($exacts | length) > 1 then {verdict:"undecidable"}
|
||||
elif ($exacts | length) == 1 then {verdict:"exact", rule:($exacts | first)}
|
||||
else {verdict:"none"} end
|
||||
@@ -92,6 +92,38 @@ MUTATIONS: tuple[Mutation, ...] = (
|
||||
why="The per-key membership test is what turns a dropped step into a red job; disarmed, every "
|
||||
"expected key reads as present and the guard passes a run in which nothing executed.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/tests/test_ci_status_context_uniqueness.py",
|
||||
target=".gitea/workflows/docker-build.yml",
|
||||
clause=" name: API docs in sync (OpenAPI + endpoint index)",
|
||||
replacement=" name: Build & test (.NET)",
|
||||
proof="test_ci_status_context_uniqueness.py::test_no_two_jobs_synthesize_the_SAME_status_context",
|
||||
granularity=CLAUSE,
|
||||
expect="these status contexts can be produced by more than one job",
|
||||
why="THE CHECKER-GUARD SHAPE: this guard IS a test, so disarming its own assertion makes it "
|
||||
"absent rather than red. Per `testing.guard-ships-with-mutation-proof` the mutation goes into "
|
||||
"the guarded ARTIFACT instead — here a workflow job renamed to collide with the REQUIRED "
|
||||
"`Build & test (.NET)` job, which is exactly the evasion the guard exists to catch: two jobs "
|
||||
"synthesizing one status context that branch protection cannot tell apart, so the required "
|
||||
"check could be satisfied by the producer whose steps carry no execution markers.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/check-required-contexts.sh",
|
||||
target="scripts/check-required-contexts.sh",
|
||||
clause="($rule.status_check_contexts | sort | unique) == $snap",
|
||||
replacement="true",
|
||||
proof="test_check_required_contexts.py::test_MUTATION_disarming_the_set_comparison_stops_every_drift_report",
|
||||
expect="the set-comparison clause has moved or been reworded",
|
||||
granularity=CLAUSE,
|
||||
why="The set comparison IS the finding: it is the only thing that turns a live required-check "
|
||||
"list differing from the committed snapshot into `drift`. Witnessed: unmutated reports "
|
||||
"`drift` on an added context, the mutant reports `match` — a permanent no-op that would "
|
||||
"confirm the snapshot fresh forever. The proof asserts the mutant's EXACT verdict "
|
||||
"`(0, 'match')` rather than merely 'not drift', and ships a positive control for its own "
|
||||
"tmp layout: an earlier draft copied the script without the classifier it loads beside "
|
||||
"itself, so the mutant exited 2 with empty stdout and 'not drift' was satisfied by a copy "
|
||||
"that never ran. The clause string is the one the proof test asserts on before mutating.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/decisions_validate.py",
|
||||
target="scripts/decisions_validate.py",
|
||||
|
||||
@@ -0,0 +1,378 @@
|
||||
"""Proof for `scripts/check-required-contexts.sh` (ersatztv#787).
|
||||
|
||||
The script reconciles the LIVE required status checks on `main` against the committed snapshot
|
||||
`.gitea/required-status-contexts.json`, from which `test_ci_dropped_step_guard.py` derives its
|
||||
marked-job scope. It is a pure classifier over a payload its caller fetched, which is the whole
|
||||
reason it is a script at all rather than a few lines inside `pretooluse-merge-consent.sh`: the
|
||||
merge hook's own comments record that while its verdict classification lived inline it had no tests
|
||||
and three false-opens survived in it (#629).
|
||||
|
||||
Every test here execs the REAL script. Nothing reimplements its logic — a second copy of the
|
||||
classifier in the test would agree with the first by construction and prove nothing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
SCRIPT = REPO_ROOT / "scripts" / "check-required-contexts.sh"
|
||||
SNAPSHOT = REPO_ROOT / ".gitea" / "required-status-contexts.json"
|
||||
|
||||
# THE CONTRACT, pinned. `pretooluse-merge-consent.sh` maps each of these onto a hook decision and
|
||||
# treats anything else as "could not tell" -> ask. Pinning the set here means a sixth class cannot be
|
||||
# added to the script without this test demanding the hook learn about it: an unmapped class would
|
||||
# otherwise reach the hook's catch-all and read as a malfunction rather than the finding it is.
|
||||
CLASSES = {"match", "drift", "nomatch", "undecidable", "unreadable"}
|
||||
|
||||
SNAP_CONTEXTS = json.loads(SNAPSHOT.read_text())["contexts"]
|
||||
|
||||
|
||||
def run(payload: str, snapshot: Path | None = None, branch: str = "main") -> tuple[int, str, str]:
|
||||
proc = subprocess.run(
|
||||
[str(SCRIPT), "--branch", branch, "--snapshot", str(snapshot or SNAPSHOT)],
|
||||
input=payload,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
return proc.returncode, proc.stdout.strip(), proc.stderr
|
||||
|
||||
|
||||
def rule(contexts, name="main", enabled=True):
|
||||
return {"branch_name": name, "enable_status_check": enabled, "status_check_contexts": contexts}
|
||||
|
||||
|
||||
def test_the_script_is_executable():
|
||||
assert SCRIPT.is_file() and SCRIPT.stat().st_mode & 0o111, f"{SCRIPT} must exist and be executable"
|
||||
|
||||
|
||||
def test_the_committed_snapshot_is_well_formed_and_not_empty():
|
||||
"""ANTI-VACUITY. An empty snapshot would make `match` reachable against an empty live list while
|
||||
`test_ci_dropped_step_guard.py` derived an empty scope and collected zero parametrized cases."""
|
||||
data = json.loads(SNAPSHOT.read_text())
|
||||
assert isinstance(data.get("contexts"), list) and data["contexts"]
|
||||
assert all(isinstance(c, str) and c.strip() for c in data["contexts"])
|
||||
assert data.get("branch") == "main"
|
||||
assert data.get("read_on"), "the snapshot must date itself — it is a claim about a moment"
|
||||
|
||||
|
||||
def test_live_equal_to_the_snapshot_is_match():
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS))]))[1] == "match"
|
||||
|
||||
|
||||
def test_the_comparison_is_by_SET_not_by_order():
|
||||
"""Gitea does not promise an order, so an order-sensitive compare would report drift forever —
|
||||
and a guard that cries wolf on every merge is one somebody switches off."""
|
||||
assert run(json.dumps([rule(list(reversed(SNAP_CONTEXTS)))]))[1] == "match"
|
||||
|
||||
|
||||
def test_an_ADDED_required_context_is_drift():
|
||||
"""THE FINDING #787 EXISTS FOR: a fourth required check whose job carries no execution marker."""
|
||||
code, out, _ = run(json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])]))
|
||||
assert (code, out) == (0, "drift")
|
||||
|
||||
|
||||
def test_a_REMOVED_required_context_is_drift():
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS[:-1]))]))[1] == "drift"
|
||||
|
||||
|
||||
def test_null_contexts_against_a_non_empty_snapshot_is_drift():
|
||||
"""`null` is a legitimate payload meaning "none required", not a read failure — so it is the
|
||||
FINDING that every required context was removed, never `unreadable`."""
|
||||
payload = json.dumps([{"branch_name": "main", "enable_status_check": True, "status_check_contexts": None}])
|
||||
assert run(payload)[1] == "drift"
|
||||
|
||||
|
||||
def test_a_rule_with_status_checks_DISABLED_is_drift_against_a_nonempty_snapshot():
|
||||
"""The EFFECTIVE required set, not the raw list.
|
||||
|
||||
`enable_status_check: false` means Gitea requires nothing on the branch whatever
|
||||
`status_check_contexts` still holds. Type-checking the flag and ignoring its VALUE certifies
|
||||
disabled protection as a current mirror — measured: with that clause removed the same payload
|
||||
returns `match`, and all 71 tests across this file and the merge-consent file stayed green,
|
||||
because `rule()`'s `enabled=` parameter had no call site that ever passed False.
|
||||
"""
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS), enabled=False)]))[1] == "drift"
|
||||
|
||||
|
||||
def test_a_rule_with_status_checks_DISABLED_matches_an_EMPTY_snapshot(tmp_path):
|
||||
"""The other side of the same clause: disabled protection IS an accurate mirror of a snapshot
|
||||
that requires nothing, so it must not be reported as drift."""
|
||||
empty = tmp_path / "snap.json"
|
||||
empty.write_text(json.dumps({"repo": "timothy/ersatztv", "branch": "main", "contexts": []}))
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS), enabled=False)]), snapshot=empty)[1] == "match"
|
||||
|
||||
|
||||
def test_no_rule_governing_the_branch_is_nomatch():
|
||||
assert run(json.dumps([rule([], name="develop")]))[1] == "nomatch"
|
||||
|
||||
|
||||
def test_a_glob_rule_makes_it_undecidable_even_when_an_exact_rule_matches():
|
||||
"""ORDER MATTERS, and this is the test that pins it. Gitea picks the governing rule by Priority
|
||||
with gobwas/glob semantics, so a glob rule can outrank an exactly-named one. Preferring the exact
|
||||
rule would compare against a rule Gitea may not be applying and could report `match` on a branch
|
||||
whose real required set has drifted — a false-open in the direction that costs something."""
|
||||
payload = json.dumps([rule([], name="m*"), rule(list(SNAP_CONTEXTS))])
|
||||
assert run(payload)[1] == "undecidable"
|
||||
|
||||
|
||||
def test_two_rules_folding_to_the_same_name_are_undecidable():
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS)), rule(list(SNAP_CONTEXTS), name="MAIN")]))[1] == "undecidable"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("label", "payload"),
|
||||
[
|
||||
("not json", "not json at all"),
|
||||
("top level object", '{"branch_name": "main"}'),
|
||||
(
|
||||
"enable_status_check as a string",
|
||||
json.dumps([{"branch_name": "main", "enable_status_check": "true", "status_check_contexts": []}]),
|
||||
),
|
||||
("a non-string context member", json.dumps([rule(["ok", 123])])),
|
||||
("contexts as an object", json.dumps([rule({"a": 1})])),
|
||||
],
|
||||
)
|
||||
def test_shapes_this_decision_cannot_consume_are_unreadable(label, payload):
|
||||
"""Validated to the depth it is CONSUMED at, not just the top-level type. An object-typed check
|
||||
that never inspects its members is the one-level-down swallow the merge hook fixed twice."""
|
||||
assert run(payload)[1] == "unreadable", label
|
||||
|
||||
|
||||
def test_every_class_the_script_emits_is_one_the_contract_declares():
|
||||
"""Every class the contract DECLARES is reachable, and these five payloads produce no other.
|
||||
|
||||
Deliberately not claimed: that a sixth class cannot ship. This drives five canned inputs, so a
|
||||
class emitted only on a sixth input shape would leave `seen` unchanged and pass — the hook's
|
||||
catch-all arm is what keeps that fail-closed, not this test. What it does catch is a declared
|
||||
class going unreachable (dead contract) or one of these five silently changing.
|
||||
"""
|
||||
seen = {
|
||||
run(json.dumps([rule(list(SNAP_CONTEXTS))]))[1],
|
||||
run(json.dumps([rule([*SNAP_CONTEXTS, "x (pull_request)"])]))[1],
|
||||
run(json.dumps([rule([], name="develop")]))[1],
|
||||
run(json.dumps([rule([], name="m*")]))[1],
|
||||
run("not json")[1],
|
||||
}
|
||||
assert seen == CLASSES, f"classes reached {sorted(seen)} != contract {sorted(CLASSES)}"
|
||||
|
||||
|
||||
def test_a_usage_error_exits_2_and_prints_NO_class():
|
||||
"""A broken invocation must not be mistakable for a finding. If it printed a class AND exited
|
||||
non-zero, a caller reading only stdout would act on a comparison that never happened."""
|
||||
proc = subprocess.run([str(SCRIPT), "--bogus"], input="[]", capture_output=True, text=True, check=False)
|
||||
assert proc.returncode == 2
|
||||
assert proc.stdout.strip() == ""
|
||||
assert not (CLASSES & set(proc.stdout.split()))
|
||||
|
||||
|
||||
def test_an_unreadable_snapshot_is_a_usage_error_not_a_class():
|
||||
proc = subprocess.run(
|
||||
[str(SCRIPT), "--snapshot", str(REPO_ROOT / "does" / "not" / "exist.json")],
|
||||
input="[]",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert proc.returncode == 2 and proc.stdout.strip() == ""
|
||||
|
||||
|
||||
def test_a_snapshot_naming_NO_branch_is_a_usage_error(tmp_path):
|
||||
"""The snapshot must name the branch it mirrors. Treating an absent `branch` as "compare against
|
||||
whatever was asked for" reaches the same defect as a mismatched one — one branch's live contexts
|
||||
weighed against another's mirror — by omission rather than by disagreement."""
|
||||
bad = tmp_path / "snap.json"
|
||||
bad.write_text(json.dumps({"contexts": list(SNAP_CONTEXTS)}))
|
||||
code, out, err = run(json.dumps([rule(list(SNAP_CONTEXTS))]), snapshot=bad)
|
||||
assert (code, out) == (2, ""), "an unbranded snapshot must be a usage error, never a class"
|
||||
# THE ARM-UNIQUE WORDING. Exit 2 alone cannot see this clause: with `.branch` absent
|
||||
# `snap_branch` is empty, and `--branch` can never be empty, so the MISMATCH guard on the next
|
||||
# line fires anyway with the same exit code and the same empty stdout. Two guards masking each
|
||||
# other (#685) — measured: deleting the required-branch clause left this file at 27 passed.
|
||||
assert "names no branch" in err, (
|
||||
f"the required-branch clause did not fire; the mismatch guard answered instead: {err.strip()[:160]}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_snapshot_for_a_DIFFERENT_branch_is_a_usage_error():
|
||||
code, out, _ = run(json.dumps([rule(list(SNAP_CONTEXTS))]), branch="develop")
|
||||
assert (code, out) == (2, ""), "a branch mismatch must be a usage error, never a class"
|
||||
|
||||
|
||||
def test_a_malformed_snapshot_is_unreadable_not_a_confident_drift(tmp_path):
|
||||
"""The snapshot is validated to the same depth as the live payload. A `contexts` holding a
|
||||
non-string would otherwise compare unequal against a well-formed live list and be reported as
|
||||
`drift` — a confident finding derived from data that was never understood."""
|
||||
bad = tmp_path / "snap.json"
|
||||
bad.write_text(json.dumps({"contexts": ["ok", 7]}))
|
||||
assert run(json.dumps([rule(list(SNAP_CONTEXTS))]), snapshot=bad)[1] == "unreadable"
|
||||
|
||||
|
||||
# The clause this guard hangs on: the set comparison that turns a differing live list into `drift`.
|
||||
COMPARE_CLAUSE = "($rule.status_check_contexts | sort | unique) == $snap"
|
||||
|
||||
|
||||
def test_a_glob_that_could_NOT_govern_the_branch_does_not_block_the_comparison():
|
||||
"""Rule selection is delegated to `scripts/lib/branch-rule-classifier.jq`, the SAME program
|
||||
`pretooluse-merge-consent.sh` loads. A rule named `a[b` carries a metacharacter but cannot govern
|
||||
`main`, so it must not make the comparison undecidable — a blanket "any metacharacter anywhere"
|
||||
test would ask forever in a repo that has one, which is how a correct guard gets switched off."""
|
||||
payload = json.dumps([rule([], name="a[b"), rule(list(SNAP_CONTEXTS))])
|
||||
assert run(payload)[1] == "match"
|
||||
|
||||
|
||||
CLASSIFIER = REPO_ROOT / "scripts" / "lib" / "branch-rule-classifier.jq"
|
||||
MERGE_HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
||||
|
||||
|
||||
SENTINEL_CLASSIFIER = (
|
||||
"# sentinel: a classifier that answers only one way, so a caller that loads THIS file changes\n"
|
||||
'{verdict: "undecidable"}\n'
|
||||
)
|
||||
|
||||
|
||||
def test_ONE_classifier_TWO_callers_BEHAVIOURALLY():
|
||||
"""The structural property #787 relies on, PROVEN by swapping the file — not by reading source.
|
||||
|
||||
A string-presence check was the seventh test in this change to pass for the wrong reason: cold
|
||||
review re-inlined a BYTE-IDENTICAL copy of the classifier into the merge hook, left a comment
|
||||
naming `scripts/lib/branch-rule-classifier.jq` above it, and the whole suite stayed green at
|
||||
exactly 1062 passed. A comment satisfies a substring assertion. And the byte-identical inline is
|
||||
precisely the drift-creating refactor that matters, because it AGREES on day one — a weakened
|
||||
copy would have been caught behaviourally.
|
||||
|
||||
So the proof replaces the shared file with a sentinel that can only answer `undecidable`, and
|
||||
requires the caller's behaviour to change. Nothing but actually loading that path can do that.
|
||||
"""
|
||||
assert CLASSIFIER.is_file(), f"the shared classifier is missing at {CLASSIFIER}"
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
victim = root / "check-required-contexts.sh"
|
||||
victim.write_text(SCRIPT.read_text(encoding="utf-8"))
|
||||
victim.chmod(0o755)
|
||||
(root / "lib").mkdir()
|
||||
(root / "lib" / "branch-rule-classifier.jq").write_text(SENTINEL_CLASSIFIER)
|
||||
|
||||
payload = json.dumps([rule(list(SNAP_CONTEXTS))])
|
||||
proc = subprocess.run(
|
||||
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
||||
input=payload,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert (proc.returncode, proc.stdout.strip()) == (0, "undecidable"), (
|
||||
"swapping scripts/lib/branch-rule-classifier.jq did not change this script's verdict "
|
||||
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r}) — it is not loading that file, so "
|
||||
"the one-copy property is a claim rather than a fact"
|
||||
)
|
||||
|
||||
|
||||
def _mutant_tree(tmp_path, source_text):
|
||||
"""The script in a RUNNABLE layout: it resolves `lib/branch-rule-classifier.jq` relative to its
|
||||
own directory, so a bare copy exits 2 before reaching any of its logic.
|
||||
|
||||
This is not incidental plumbing. The first version of the mutation proof below copied only the
|
||||
script, so the mutant died on `classifier not readable` with EMPTY stdout — and an assertion that
|
||||
stdout merely differed from `drift` was satisfied by that empty string. The proof passed for the
|
||||
wrong reason and the `MUTATION` grade it justified was unsupported.
|
||||
"""
|
||||
victim = tmp_path / "mutant.sh"
|
||||
victim.write_text(source_text)
|
||||
victim.chmod(0o755)
|
||||
shutil.copytree(REPO_ROOT / "scripts" / "lib", tmp_path / "lib")
|
||||
return victim
|
||||
|
||||
|
||||
def test_the_mutation_harness_layout_ITSELF_reproduces_the_real_verdict(tmp_path):
|
||||
"""POSITIVE CONTROL for the harness below, without which it can only prove a broken copy.
|
||||
|
||||
The UNMUTATED script, in the same tmp layout, must reach the same verdict it reaches in the
|
||||
repo. If this fails, the mutation test's red says nothing about the clause.
|
||||
"""
|
||||
victim = _mutant_tree(tmp_path, SCRIPT.read_text(encoding="utf-8"))
|
||||
drifted = json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])])
|
||||
proc = subprocess.run(
|
||||
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
||||
input=drifted,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert (proc.returncode, proc.stdout.strip()) == (0, "drift"), (
|
||||
f"the unmutated script did not report drift in the harness layout "
|
||||
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r} stderr={proc.stderr.strip()[:160]!r}) — "
|
||||
"the mutation proof below would then be measuring a broken copy, not a disarmed clause"
|
||||
)
|
||||
|
||||
|
||||
def test_a_MISSING_shared_classifier_is_a_usage_error_not_an_unreadable_CLASS():
|
||||
"""The twin of a fix the hook made loudly, on the second caller.
|
||||
|
||||
Without this guard `jq -f` on a missing program fails, `verdict` is empty, and the script prints
|
||||
the CLASS `unreadable` with exit 0 — reporting "the payload came back in a shape I could not
|
||||
consume" about a missing LOCAL FILE. That is the states-a-cause-that-did-not-happen defect the
|
||||
hook added its own `[ -r "$classifier" ]` check to avoid; measured, removing this one left the
|
||||
file at 27 passed.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
victim = Path(tmp) / "check-required-contexts.sh"
|
||||
victim.write_text(SCRIPT.read_text(encoding="utf-8"))
|
||||
victim.chmod(0o755)
|
||||
# deliberately NO lib/ beside it
|
||||
proc = subprocess.run(
|
||||
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
||||
input=json.dumps([rule(list(SNAP_CONTEXTS))]),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
assert proc.returncode == 2, f"a missing classifier must be a usage error, got exit {proc.returncode}"
|
||||
assert proc.stdout.strip() == "", (
|
||||
f"a missing local file was reported as the class {proc.stdout.strip()!r} — a cause that did not happen"
|
||||
)
|
||||
assert "classifier not readable" in proc.stderr
|
||||
|
||||
|
||||
def test_MUTATION_disarming_the_set_comparison_stops_every_drift_report(tmp_path):
|
||||
"""`testing.guard-ships-with-mutation-proof` (#775) — the DECLARED clause, re-run every suite.
|
||||
|
||||
The set comparison is the guard's entire finding. Disarm that clause alone — make it constantly
|
||||
true — and a live list that genuinely differs from the snapshot must stop reporting `drift`. If
|
||||
it still did, the finding comes from somewhere other than the clause this guard hangs on and
|
||||
every passing test above proves nothing about it.
|
||||
"""
|
||||
text = SCRIPT.read_text(encoding="utf-8")
|
||||
assert COMPARE_CLAUSE in text, (
|
||||
"the set-comparison clause has moved or been reworded; RETARGET this mutation at its new "
|
||||
"location rather than loosening the string match — a mutation that silently stops mutating "
|
||||
"is the exact failure this file exists to catch"
|
||||
)
|
||||
mutated = text.replace(COMPARE_CLAUSE, "true", 1)
|
||||
assert mutated != text and COMPARE_CLAUSE not in mutated, "the replacement did not change the source"
|
||||
|
||||
victim = _mutant_tree(tmp_path, mutated)
|
||||
drifted = json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])])
|
||||
proc = subprocess.run(
|
||||
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
||||
input=drifted,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
# EXACT, not `!= "drift"`. The mutant must RUN and reach the disarmed clause, so the only
|
||||
# acceptable evidence is the specific wrong answer it now gives. Any exit code but 0, or an
|
||||
# empty stdout, means it died on the way and proves nothing.
|
||||
assert (proc.returncode, proc.stdout.strip()) == (0, "match"), (
|
||||
f"the mutant did not reach the disarmed clause and report `match` "
|
||||
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r} stderr={proc.stderr.strip()[:160]!r}). "
|
||||
"With the set comparison constantly true it must certify a drifted list as current; "
|
||||
"anything else means the proof measured a broken copy rather than a disarmed guard"
|
||||
)
|
||||
@@ -21,6 +21,7 @@ THE TESTS COME IN THREE KINDS AND NONE SUBSTITUTES FOR ANOTHER, which is the les
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
@@ -32,22 +33,190 @@ import yaml
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
# ASSESSED FOR ersatztv#806: this file has NO filesystem-derived population. Its members come from
|
||||
# the PARSED workflow (`_marked(job)` reads the marked steps out of `_DOC`), which is already an
|
||||
# authoritative machine-readable source, so the index changes nothing here. Its known residual is at
|
||||
# the other altitude — `MARKED_JOBS` is a hand-written mirror of the required contexts on `main`,
|
||||
# a SCOPE rather than a population, and `testing.guard-derives-population-from-source` already
|
||||
# carries it as this repo's canonical residual gap. #806 does not close it.
|
||||
# authoritative machine-readable source, so the index changes nothing here. The residual #806 left
|
||||
# open was at the other altitude — `MARKED_JOBS` as a hand-written mirror of the required contexts
|
||||
# on `main`, a SCOPE rather than a population — and ersatztv#787 closes it: the scope is DERIVED
|
||||
# below from `.gitea/required-status-contexts.json`, and that snapshot is reconciled against the
|
||||
# live server by `scripts/check-required-contexts.sh`.
|
||||
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
|
||||
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
|
||||
REQUIRED_CONTEXTS = REPO_ROOT / ".gitea" / "required-status-contexts.json"
|
||||
|
||||
# The jobs whose contexts branch protection REQUIRES on `main`. Read live on 2026-08-10:
|
||||
# Build ErsatzTV Image / Build & test (.NET) (pull_request)
|
||||
# Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request)
|
||||
# review-verdict/h10
|
||||
# The third is guarded by test_pr_changed_files.py; these two are this file's subject. `build`,
|
||||
# `api-docs` and `format` are deliberately NOT here — they are not required, and all three
|
||||
# Required contexts that are NOT a job of `WORKFLOW`, each mapped to the guard that DOES cover it.
|
||||
# A required context reaching neither this map nor a job of `WORKFLOW` is a hard error at import,
|
||||
# and that is the entire point of ersatztv#787: a fourth required check must not be able to arrive
|
||||
# and acquire no dropped-step guard in silence. Both directions are asserted below, so an entry that
|
||||
# stops being required is reported too rather than lingering as a permanent excuse.
|
||||
ACCOUNTED_ELSEWHERE = {
|
||||
"review-verdict/h10": (
|
||||
"not an Actions job context at all — posted onto the head sha by "
|
||||
"scripts/post-review-verdict.sh and enforced by .gitea/workflows/review-verdict.yml, whose "
|
||||
"changed-file derivation is guarded by test_pr_changed_files.py"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _workflow_events(doc) -> list[str]:
|
||||
"""The events a workflow declares.
|
||||
|
||||
YAML 1.1 parses a bare `on:` key as the BOOLEAN `True`, so `doc["on"]` is absent in every
|
||||
workflow in this repo and a lookup that only tried the string would silently synthesize nothing.
|
||||
"""
|
||||
on = doc.get(True, doc.get("on"))
|
||||
if isinstance(on, str):
|
||||
return [on]
|
||||
if isinstance(on, dict):
|
||||
return sorted(on)
|
||||
return sorted(on or [])
|
||||
|
||||
|
||||
def _workflow_job_contexts() -> dict[str, str]:
|
||||
"""Every status context THIS workflow can produce -> job key.
|
||||
|
||||
SCOPE, and deliberately just `WORKFLOW` — this file's subject is `docker-build.yml`, and its
|
||||
every other assertion already reads `_DOC`. Note the failure DIRECTION, which is what makes a
|
||||
narrow index safe rather than a hidden gap: this is a lookup table, not a completeness
|
||||
population. The completeness claim is over the SNAPSHOT's contexts, every one of which must
|
||||
resolve here or be dispositioned, so an index missing a workflow makes a context UNRESOLVABLE
|
||||
and the guard strictly LOUDER. It may not be derived from the git index: the release-path `scan`
|
||||
job runs `test_ci_release_path_scan_job.py`, whose harness executes THIS file inside a poisoned
|
||||
COPY of the tree that is not a git repository, where `git ls-files` exits 128. (The scan job
|
||||
itself runs in a real checkout — the non-git tree is the harness's, which is why the constraint
|
||||
is real but the shorthand "the scan job runs this in a non-git copy" is imprecise.) An earlier
|
||||
draft of this change walked straight into it.
|
||||
|
||||
SYNTHESIZED and matched by equality, never parsed. Gitea names an Actions context
|
||||
`<workflow name> / <job name> (<event>)`, and both names are already in the YAML — so building
|
||||
every candidate string avoids writing a parser for a format whose delimiters also occur INSIDE
|
||||
the names they delimit: `Build & test (.NET)` ends in a parenthesis of its own, and a regex
|
||||
anchored on the last `(...)` group is one rename away from splitting in the wrong place.
|
||||
"""
|
||||
doc = yaml.safe_load(WORKFLOW.read_text())
|
||||
workflow_name = doc.get("name")
|
||||
assert workflow_name, f"{WORKFLOW.name} declares no `name:`, so it produces no derivable context"
|
||||
# A MULTIMAP, then a uniqueness assertion — never a plain dict assignment. Job KEYS are unique in
|
||||
# YAML but job `name:` VALUES are not, and the context string is built from the name. Two jobs
|
||||
# sharing a name collide on one key, the later silently wins, and the required context then
|
||||
# resolves to whichever job happened to sort last while the other's steps go unguarded. Branch
|
||||
# protection sees only the shared string, so it cannot tell them apart either.
|
||||
index: dict[str, list[str]] = {}
|
||||
for job_key, job in (doc.get("jobs") or {}).items():
|
||||
if not isinstance(job, dict):
|
||||
continue
|
||||
job_name = job.get("name") or job_key
|
||||
for event in _workflow_events(doc):
|
||||
index.setdefault(f"{workflow_name} / {job_name} ({event})", []).append(job_key)
|
||||
ambiguous = {ctx: keys for ctx, keys in index.items() if len(keys) > 1}
|
||||
assert not ambiguous, (
|
||||
f"{WORKFLOW.name} declares jobs that synthesize the SAME status context: "
|
||||
+ "; ".join(f"{ctx!r} <- {sorted(keys)}" for ctx, keys in sorted(ambiguous.items()))
|
||||
+ ". Branch protection identifies a required check by that string alone, so it cannot "
|
||||
"distinguish them and this guard could scope itself to the wrong job while the other's "
|
||||
"steps go unmarked. Give the jobs distinct `name:` values."
|
||||
)
|
||||
return {ctx: keys[0] for ctx, keys in index.items()}
|
||||
|
||||
|
||||
def required_contexts() -> list[str]:
|
||||
"""The committed mirror of `status_check_contexts` on `main`.
|
||||
|
||||
Shape-checked rather than trusted: a snapshot that arrived empty or as the wrong type would
|
||||
derive an empty `MARKED_JOBS`, and an empty `parametrize` list collects ZERO tests and reports
|
||||
green — the vacuity this file exists to prevent, one altitude up.
|
||||
"""
|
||||
data = json.loads(REQUIRED_CONTEXTS.read_text())
|
||||
contexts = data.get("contexts")
|
||||
assert isinstance(contexts, list) and contexts and all(isinstance(c, str) for c in contexts), (
|
||||
f"{REQUIRED_CONTEXTS.name} must hold a non-empty `contexts` list of strings; got "
|
||||
f"{contexts!r}. Every marked-job scope in this file derives from it."
|
||||
)
|
||||
return contexts
|
||||
|
||||
|
||||
def _derive_marked_jobs(contexts: list[str] | None = None) -> tuple[str, ...]:
|
||||
"""`MARKED_JOBS`, derived. Raises at import rather than narrowing silently.
|
||||
|
||||
`contexts` is injectable ONLY so the two assertions below can be proven. They are the change
|
||||
ersatztv#787 turns on — the unaccounted-context hard failure is what makes a fourth required
|
||||
check impossible to add in silence — and while this read the committed snapshot unconditionally
|
||||
they were unprovable: disarming `assert not unaccounted` and planting a fourth context left the
|
||||
scope silently behind with 51 tests still green. Production calls it with no argument.
|
||||
"""
|
||||
index = _workflow_job_contexts()
|
||||
jobs: list[str] = []
|
||||
unaccounted: list[str] = []
|
||||
for context in required_contexts() if contexts is None else contexts:
|
||||
job_key = index.get(context)
|
||||
if job_key is not None:
|
||||
jobs.append(job_key)
|
||||
continue
|
||||
if context in ACCOUNTED_ELSEWHERE:
|
||||
continue
|
||||
unaccounted.append(f"{context!r} (no job of {WORKFLOW.name} produces it)")
|
||||
assert not unaccounted, (
|
||||
"branch protection requires status context(s) this guard cannot account for: "
|
||||
+ "; ".join(sorted(unaccounted))
|
||||
+ f". A required context whose steps carry no execution marker is fail-OPEN — a step the "
|
||||
"runner drops concludes success and takes the whole check green having done no work "
|
||||
f"(ersatztv#756). If a job of {WORKFLOW.name} produces it, mark that job's steps and it will "
|
||||
"derive here. If ANOTHER workflow produces it, this file cannot cover it — its index is "
|
||||
f"{WORKFLOW.name} alone, because the release-path `scan` job runs this file in a non-git "
|
||||
"copy — so give it an ACCOUNTED_ELSEWHERE entry naming the guard that does, and add that "
|
||||
"guard if none exists."
|
||||
)
|
||||
assert jobs, (
|
||||
"no required status context maps to a job of "
|
||||
f"{WORKFLOW.name} — every parametrized test below would collect zero cases and report "
|
||||
"green. Check .gitea/required-status-contexts.json against the live server with "
|
||||
"scripts/check-required-contexts.sh."
|
||||
)
|
||||
return tuple(sorted(set(jobs)))
|
||||
|
||||
|
||||
def test_a_required_context_this_guard_CANNOT_ACCOUNT_FOR_is_a_hard_failure():
|
||||
"""THE CLAUSE #787 TURNS ON, proven rather than described.
|
||||
|
||||
A fourth required context that maps to no job of this workflow and to no stated disposition must
|
||||
stop the derivation dead. Without this the guard's scope silently stays behind — which is the
|
||||
original defect, reproduced one layer up: measured, with `assert not unaccounted` disarmed and a
|
||||
fourth context planted, the suite reported 51 passed and nothing anywhere went red.
|
||||
"""
|
||||
planted = [*required_contexts(), "Some Other Workflow / Fourth (pull_request)"]
|
||||
with pytest.raises(AssertionError, match="cannot account for"):
|
||||
_derive_marked_jobs(planted)
|
||||
|
||||
|
||||
def test_a_context_set_that_maps_to_NO_job_here_is_a_hard_failure():
|
||||
"""The anti-vacuity clause, proven. An empty derived scope makes every `parametrize` below
|
||||
collect ZERO cases and report green — a guard that asserts nothing while looking healthy."""
|
||||
with pytest.raises(AssertionError, match="no required status context maps"):
|
||||
_derive_marked_jobs(list(ACCOUNTED_ELSEWHERE))
|
||||
|
||||
|
||||
def test_every_ACCOUNTED_ELSEWHERE_entry_is_STILL_a_required_context():
|
||||
"""The other direction, without which the map is a one-way excuse list.
|
||||
|
||||
`_derive_marked_jobs` reads ACCOUNTED_ELSEWHERE to EXCUSE a required context from needing a
|
||||
marked job here. Nothing in that direction notices an entry that has stopped being required —
|
||||
it would sit there permanently, naming a guard for a check nobody runs, and the next reader
|
||||
would take it as a statement about the current required set. Same shape as the stale-exemption
|
||||
rule `test_guard_populations_derive_from_git.py` applies to POPULATION_EXEMPT.
|
||||
"""
|
||||
stale = sorted(set(ACCOUNTED_ELSEWHERE) - set(required_contexts()))
|
||||
assert not stale, (
|
||||
f"ACCOUNTED_ELSEWHERE excuses {stale}, which branch protection no longer requires per "
|
||||
".gitea/required-status-contexts.json. Drop the entry — a stale excuse is worse than none, "
|
||||
"because it reads as a checked decision about the current required set."
|
||||
)
|
||||
thin = sorted(k for k, why in ACCOUNTED_ELSEWHERE.items() if not str(why).strip())
|
||||
assert not thin, f"ACCOUNTED_ELSEWHERE entries with no stated guardian: {thin}"
|
||||
|
||||
|
||||
# The jobs whose contexts branch protection REQUIRES on `main`, DERIVED from the committed snapshot.
|
||||
# `build`, `api-docs` and `format` are absent because they are not required, and all three
|
||||
# legitimately interpolate into a `run:` body, so extending the absolute rule to them would be false.
|
||||
# Per-step markers apply to the two REQUIRED contexts, where a dropped step is fail-OPEN.
|
||||
MARKED_JOBS = ("test", "migrations")
|
||||
# Per-step markers apply to the REQUIRED contexts, where a dropped step is fail-OPEN.
|
||||
MARKED_JOBS = _derive_marked_jobs()
|
||||
|
||||
# The delimiter ban is WIDER than the marker set, and the extra job is not an afterthought.
|
||||
# `build`'s "Smoke + IPTV E2E" step runs AFTER `Build and push`, so on a `v*` tag the image is
|
||||
@@ -62,7 +231,21 @@ MARKED_JOBS = ("test", "migrations")
|
||||
# is consequential" rather than "ban wherever it happens to be free right now".
|
||||
# `api-docs` and `format` keep one delimiter each, both `github.base_ref` in a detect step, and gate
|
||||
# nothing that ships.
|
||||
DELIMITER_BAN_JOBS = ("test", "migrations", "build")
|
||||
DELIMITER_BAN_JOBS = tuple(sorted({*MARKED_JOBS, "build"}))
|
||||
|
||||
# THE STATED INVARIANT, NOW ENFORCED. "Wider than the marker set" was prose while both sides were
|
||||
# literals edited together. #787 made `MARKED_JOBS` DERIVE from branch protection, so the two can now
|
||||
# move independently — and the direction that matters is silent: on the day a fourth required context
|
||||
# arrives, the marker half goes red and demands markers, the author adds them, CI greens, and the new
|
||||
# required job's `run:` bodies were never checked for the `${{` delimiter that CAUSES the drop the
|
||||
# markers detect. Deriving the ban set from the marker set closes it by construction; the assertion
|
||||
# below states the property anyway, so a future edit back to a literal cannot quietly re-open it.
|
||||
assert set(MARKED_JOBS) <= set(DELIMITER_BAN_JOBS), (
|
||||
f"the delimiter ban {sorted(DELIMITER_BAN_JOBS)} is NARROWER than the derived marker set "
|
||||
f"{sorted(MARKED_JOBS)}. Every job whose steps must record that they ran must also be barred "
|
||||
"from interpolating into a `run:` body, or the guard demands a marker for a job whose bodies can "
|
||||
"still carry the delimiter that makes the runner drop the step in the first place (ersatztv#756)."
|
||||
)
|
||||
|
||||
# THE RAW OPENER, not a closed `${{ … }}` pair — found by cold review. The runner's rewrite is
|
||||
# triggered by the OPENER; a closed-pair regex therefore misses `# ${{` with no closer, which would
|
||||
@@ -141,13 +324,13 @@ def _is_gated(step) -> bool:
|
||||
|
||||
@pytest.mark.parametrize("job", DELIMITER_BAN_JOBS)
|
||||
def test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body(job):
|
||||
"""The absolute rule from `review-verdict.yml`, extended to the two required build jobs.
|
||||
"""The absolute rule from `review-verdict.yml`, extended to the required build jobs.
|
||||
|
||||
This is the cheaper and more general half of #756: the drop mechanism REQUIRES an opener in the
|
||||
scalar, so a job with none is immune by construction and the runtime markers are a backstop
|
||||
rather than the only line of defence.
|
||||
|
||||
The scope is the three jobs in `DELIMITER_BAN_JOBS` — see the comment there for why `build` is in
|
||||
The scope is `DELIMITER_BAN_JOBS` — see the comment there for why `build` is in
|
||||
and `functional-e2e` is not. Do NOT restate this docstring as "scoped to the required pair":
|
||||
round 2 moved `build`'s two payloads into `env:` and brought it into the ban, and this docstring
|
||||
sits directly above the decorator that parametrises over the wider set.
|
||||
|
||||
@@ -287,7 +287,9 @@ def test_docker_build_is_the_ONLY_workflow_pinning_the_toolchain_image():
|
||||
text claims "All container jobs must run the same toolchain image".
|
||||
|
||||
Found by cold review, which correctly noted this file criticises `MARKED_JOBS` for exactly this
|
||||
and then shipped the same shape without even the dated comment `MARKED_JOBS` carries.
|
||||
and then shipped the same shape without even the dated comment `MARKED_JOBS` then carried.
|
||||
`MARKED_JOBS` has since been derived (#787); `TOOLCHAIN_JOBS` below is the remaining literal and
|
||||
is tracked in #789.
|
||||
|
||||
The population comes from the GIT INDEX (ersatztv#806). A `Path.glob` here answered a question
|
||||
about the machine rather than about the repo: an untracked scratch workflow left in
|
||||
|
||||
@@ -300,14 +300,27 @@ def _repo_copy(tmp_path: Path) -> Path:
|
||||
that step runs. Neither may derive a population through `scripts/tests/tracked_files.py`:
|
||||
`git ls-files` inside the copy fails, and the release-path scan step fails with it. The copying
|
||||
happens HERE, in the real repo, which is why this fixture may use the index while its subjects
|
||||
may not. `test_ci_dropped_step_guard.py`'s `MARKED_JOBS` is explicitly left open as a residual
|
||||
gap, so a future session is invited to edit exactly that file — this paragraph is what stands
|
||||
between that edit and a broken release gate.
|
||||
may not. #787 rewrote `test_ci_dropped_step_guard.py`'s scope to derive from
|
||||
`.gitea/required-status-contexts.json` and reached for `tracked_paths` on the way, which broke
|
||||
all three tests below until the index derivation came back out — so this paragraph earned its
|
||||
keep, and it is what stands between the next such edit and a broken release gate.
|
||||
"""
|
||||
dst = tmp_path / "repo"
|
||||
(dst / ".gitea" / "workflows").mkdir(parents=True)
|
||||
for wf in tracked_files.tracked_paths(*WORKFLOWS):
|
||||
shutil.copy2(wf, dst / ".gitea" / "workflows" / wf.name)
|
||||
# `.gitea/` DATA the copied tests read, which is not a workflow and so is not in the loop above.
|
||||
# `test_ci_dropped_step_guard.py` derives its marked-job scope from this snapshot (#787), and
|
||||
# without it the scan step dies on a FileNotFoundError that says nothing about the delimiter ban
|
||||
# this harness exists to prove. Named explicitly rather than copied by glob: a new `.gitea/` file
|
||||
# should have to be considered, not silently swept into a hermetic fixture.
|
||||
for rel in (".gitea/required-status-contexts.json",):
|
||||
source = REPO_ROOT / rel
|
||||
assert source.is_file(), (
|
||||
f"{rel} is missing, so the harness copy would be incomplete and the probes below would "
|
||||
"measure a tree that is missing part of the thing under test."
|
||||
)
|
||||
shutil.copy2(source, dst / rel)
|
||||
for rel in tracked_files._git_ls_files():
|
||||
if not rel.startswith("scripts/"):
|
||||
continue
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""No two CI jobs may synthesize the SAME status-check context string (ersatztv#787).
|
||||
|
||||
Gitea names an Actions status context `<workflow name> / <job name> (<event>)`, and branch protection
|
||||
identifies a required check by that string ALONE. So two jobs that synthesize the same string are
|
||||
indistinguishable to it: a required context can be satisfied by whichever of them reports, and a
|
||||
guard that scopes itself to one of them leaves the other's steps unguarded while every test stays
|
||||
green. `test_ci_dropped_step_guard.py` resolves the required contexts on `main` to the jobs that must
|
||||
carry per-step execution markers, and that resolution is only sound if the mapping is injective.
|
||||
|
||||
WHY THIS IS ITS OWN FILE rather than an assertion inside the dropped-step guard. That guard is one of
|
||||
the two files `docker-build.yml`'s release-path `scan` job runs, and it runs them inside a poisoned
|
||||
COPY of the tree that is NOT a git repository — `git ls-files` exits 128 there. A cross-workflow
|
||||
claim needs the whole workflow population, the authoritative source for which is the git index
|
||||
(`testing.guard-derives-population-from-source`, #806), so the two requirements are incompatible in
|
||||
one file. Splitting them lets each have the right source: the dropped-step guard reads only its own
|
||||
subject workflow and therefore survives the copy, while this file derives the full population and
|
||||
runs in `pr-checks.yml::script-tests`, which is a real checkout.
|
||||
TWO LIMITS, stated so they are not rediscovered as bugs. A workflow declaring no `name:` is skipped
|
||||
entirely, so a context Gitea would label from the FILENAME is unmodeled. And job names are compared
|
||||
as raw text, so a name carrying an expression that renders to an existing one at runtime does not
|
||||
collide here — on the dropped-step side that stays loud (the context resolves to no job and is a hard
|
||||
failure), only the uniqueness side is quiet about it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from scripts.tests.tracked_files import REPO_ROOT, tracked_paths
|
||||
|
||||
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
|
||||
|
||||
|
||||
def workflow_files():
|
||||
"""THE WORKFLOW POPULATION, from the git index. Named so the shared proof in
|
||||
`test_guard_populations_derive_from_git.py` can assert it never admits an untracked file."""
|
||||
return tracked_paths(*WORKFLOWS)
|
||||
|
||||
|
||||
def _events(doc) -> list[str]:
|
||||
"""YAML 1.1 parses a bare `on:` key as the BOOLEAN `True`, so a lookup that only tried the
|
||||
string would silently synthesize nothing and make this guard vacuous."""
|
||||
on = doc.get(True, doc.get("on"))
|
||||
if isinstance(on, str):
|
||||
return [on]
|
||||
if isinstance(on, dict):
|
||||
return sorted(on)
|
||||
return sorted(on or [])
|
||||
|
||||
|
||||
def synthesized_contexts() -> dict[str, list[str]]:
|
||||
"""context string -> every `<workflow file>::<job key>` that can produce it."""
|
||||
index: dict[str, list[str]] = {}
|
||||
for path in workflow_files():
|
||||
doc = yaml.safe_load(path.read_text())
|
||||
if not isinstance(doc, dict):
|
||||
continue
|
||||
workflow_name = doc.get("name")
|
||||
if not workflow_name:
|
||||
continue
|
||||
# The label is for the failure message only. Computed defensively because the detector's own
|
||||
# proof feeds it a planted workflow from a tmp dir, and a `relative_to` that throws there
|
||||
# would make the proof fail on plumbing instead of on the collision it plants.
|
||||
try:
|
||||
rel = path.relative_to(REPO_ROOT).as_posix()
|
||||
except ValueError:
|
||||
rel = str(path)
|
||||
for job_key, job in (doc.get("jobs") or {}).items():
|
||||
if not isinstance(job, dict):
|
||||
continue
|
||||
job_name = job.get("name") or job_key
|
||||
for event in _events(doc):
|
||||
index.setdefault(f"{workflow_name} / {job_name} ({event})", []).append(f"{rel}::{job_key}")
|
||||
return index
|
||||
|
||||
|
||||
def test_anti_vacuity_the_synthesis_found_a_plausible_number_of_contexts():
|
||||
"""Every assertion here is over `synthesized_contexts()`, and a synthesis that collapsed to
|
||||
nothing would satisfy all of them while proving nothing."""
|
||||
contexts = synthesized_contexts()
|
||||
assert len(contexts) >= 20, (
|
||||
f"synthesized only {len(contexts)} contexts from {len(workflow_files())} workflow file(s) — "
|
||||
"the synthesis is broken, not the repo, and the uniqueness assertion below is vacuous."
|
||||
)
|
||||
|
||||
|
||||
def test_no_two_jobs_synthesize_the_SAME_status_context():
|
||||
"""The injectivity `test_ci_dropped_step_guard.py`'s context->job resolution depends on."""
|
||||
collisions = {ctx: producers for ctx, producers in synthesized_contexts().items() if len(producers) > 1}
|
||||
assert not collisions, (
|
||||
"these status contexts can be produced by more than one job: "
|
||||
+ "; ".join(f"{ctx!r} <- {sorted(producers)}" for ctx, producers in sorted(collisions.items()))
|
||||
+ ". Branch protection identifies a required check by that string alone, so it cannot tell "
|
||||
"them apart: a required context could be satisfied by the producer whose steps carry no "
|
||||
"execution markers, while the guard scoped itself to the other one. Give the workflows or "
|
||||
"the jobs distinct `name:` values."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"planted",
|
||||
[
|
||||
pytest.param(("Build ErsatzTV Image", "Build & test (.NET)"), id="same-workflow-and-job-name"),
|
||||
pytest.param(("Build ErsatzTV Image", "EF migration integrity (SQLite + MySql)"), id="other-required-job"),
|
||||
],
|
||||
)
|
||||
def test_the_detector_FIRES_on_a_planted_cross_workflow_twin(planted, tmp_path, monkeypatch):
|
||||
"""The detector's own proof. A second workflow FILE declaring the same workflow name, job name
|
||||
and event as a REQUIRED job is the exact evasion this file exists to catch, and it is invisible
|
||||
to any check that reads only `docker-build.yml`.
|
||||
"""
|
||||
workflow_name, job_name = planted
|
||||
twin = tmp_path / "twin.yml"
|
||||
twin.write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"name": workflow_name,
|
||||
"on": {"pull_request": None},
|
||||
"jobs": {"innocuous": {"name": job_name, "steps": []}},
|
||||
}
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"scripts.tests.test_ci_status_context_uniqueness.workflow_files",
|
||||
lambda: [*tracked_paths(*WORKFLOWS), twin],
|
||||
)
|
||||
with pytest.raises(AssertionError, match="more than one job"):
|
||||
test_no_two_jobs_synthesize_the_SAME_status_context()
|
||||
@@ -40,6 +40,7 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from scripts.tests import test_ci_image_pin_population as image_pin
|
||||
from scripts.tests import test_ci_status_context_uniqueness as ctx_unique
|
||||
from scripts.tests import test_guard_inventory as guard_inventory
|
||||
from scripts.tests import test_hook_fire_log as hook_fire
|
||||
from scripts.tests import test_pr_changed_files as pr_changed
|
||||
@@ -107,6 +108,7 @@ DERIVATIONS = (
|
||||
("test_guard_inventory.derived_guard_files", guard_inventory.derived_guard_files, 25),
|
||||
("test_hook_fire_log.hook_scripts", hook_fire.hook_scripts, 10),
|
||||
("test_ci_image_pin_population.workflow_files", image_pin.workflow_files, 5),
|
||||
("test_ci_status_context_uniqueness.workflow_files", ctx_unique.workflow_files, 5),
|
||||
("test_remote_state_inventory.derived_population", remote_state.derived_population, 40),
|
||||
("test_pr_changed_files._workflow_files", pr_changed._workflow_files, 5),
|
||||
("test_workflow_persist_credentials.workflow_files", persist_creds.workflow_files, 5),
|
||||
@@ -543,7 +545,7 @@ def test_every_index_derived_module_is_registered():
|
||||
Without this, adding another index-derived guard and forgetting to register it leaves that guard
|
||||
unproven while this file reads as covering them all — a completeness claim standing behind a
|
||||
hand-maintained list, which is the defect one altitude up (#773 Family C, and the shape that put
|
||||
`MARKED_JOBS` in the record as a residual gap).
|
||||
`MARKED_JOBS` in the record as a residual gap until #787 closed it).
|
||||
|
||||
THE SCOPE THIS CANNOT SEE, stated because a check described as complete stops being re-examined:
|
||||
it detects modules that IMPORT the shared helper. A module deriving a file population some other
|
||||
|
||||
@@ -31,6 +31,7 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
@@ -49,6 +50,12 @@ CURL_SHIM = r"""#!/usr/bin/env python3
|
||||
import json, os, sys, pathlib, urllib.parse
|
||||
|
||||
state = pathlib.Path(os.environ["STUB_DIR"])
|
||||
# The repo's OWN committed snapshot of `main`'s required checks (ersatztv#787). The arms that model
|
||||
# a healthy, grantable repo serve exactly this list, so the hook's guard-scope freshness check sees
|
||||
# `match` and the grant path stays reachable. Read from the file rather than restated here: a second
|
||||
# copy would drift the day the real required checks change, and these tests would then be asserting
|
||||
# that a grant happens in a repo state that no longer exists.
|
||||
REQUIRED = json.loads(pathlib.Path(os.environ["REQUIRED_CONTEXTS_JSON"]).read_text())["contexts"]
|
||||
args = sys.argv[1:]
|
||||
url = [a for a in args if a.startswith("http")][-1]
|
||||
|
||||
@@ -118,13 +125,19 @@ if url.rstrip("/").endswith("/branch_protections"):
|
||||
respond(json.dumps([{"branch_name": "a[b", "enable_status_check": True,
|
||||
"status_check_contexts": ["review-verdict/h10"]},
|
||||
{"branch_name": "main", "enable_status_check": True,
|
||||
"status_check_contexts": ["review-verdict/h10"]}]))
|
||||
"status_check_contexts": REQUIRED}]))
|
||||
if mode == "ESCAPED-META-RULE":
|
||||
respond(json.dumps([{"branch_name": "a\\{b", "enable_status_check": True,
|
||||
"status_check_contexts": ["review-verdict/h10"]}]))
|
||||
if mode == "DRIFTED-CONTEXTS":
|
||||
# Still requires review-verdict/h10, so every check ABOVE the freshness one is satisfied and
|
||||
# the hook reaches it — but the live list carries a context the committed snapshot does not.
|
||||
respond(json.dumps([{"branch_name": "main", "enable_status_check": True,
|
||||
"status_check_contexts":
|
||||
[*REQUIRED, "Build ErsatzTV Image / Fourth (pull_request)"]}]))
|
||||
if mode == "CASEFOLD-RULE":
|
||||
respond(json.dumps([{"branch_name": "MAIN", "enable_status_check": True,
|
||||
"status_check_contexts": ["review-verdict/h10"]}]))
|
||||
"status_check_contexts": REQUIRED}]))
|
||||
if mode == "TWO-CASE-VARIANT-RULES":
|
||||
respond(json.dumps([{"branch_name": "MAIN", "enable_status_check": True,
|
||||
"status_check_contexts": ["review-verdict/h10"]},
|
||||
@@ -171,9 +184,7 @@ if url.rstrip("/").endswith("/branch_protections"):
|
||||
["Build ErsatzTV Image / Build & test (.NET) (pull_request)"]}]))
|
||||
# GUARDED
|
||||
respond(json.dumps([{"branch_name": "main", "enable_status_check": True,
|
||||
"status_check_contexts":
|
||||
["Build ErsatzTV Image / Build & test (.NET) (pull_request)",
|
||||
"review-verdict/h10"]}]))
|
||||
"status_check_contexts": REQUIRED}]))
|
||||
|
||||
if "/pulls/" in url and "/files" in url:
|
||||
q = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
|
||||
@@ -259,6 +270,7 @@ def hook(tmp_path):
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["REQUIRED_CONTEXTS_JSON"] = str(REPO_ROOT / ".gitea" / "required-status-contexts.json")
|
||||
env["STUB_SHA"] = SHA
|
||||
env["STUB_SHORT"] = SHORT
|
||||
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
@@ -286,17 +298,24 @@ def hook(tmp_path):
|
||||
f = state / "bp_urls"
|
||||
return f.read_text().splitlines() if f.exists() else []
|
||||
|
||||
def decision(self, scheduled=True):
|
||||
def decision(self, scheduled=True, project_dir=None, hook_path=None, owner="timothy", repo="ersatztv"):
|
||||
payload = {
|
||||
"tool_input": {
|
||||
"method": "merge",
|
||||
"owner": "timothy",
|
||||
"repo": "ersatztv",
|
||||
"owner": owner,
|
||||
"repo": repo,
|
||||
"pull_number": 42,
|
||||
"merge_when_checks_succeed": scheduled,
|
||||
}
|
||||
}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
|
||||
run_env = env if project_dir is None else {**env, "CLAUDE_PROJECT_DIR": project_dir}
|
||||
r = subprocess.run(
|
||||
["bash", str(hook_path or HOOK)],
|
||||
input=json.dumps(payload),
|
||||
env=run_env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert r.returncode == 0, r.stderr
|
||||
if not r.stdout.strip():
|
||||
return None
|
||||
@@ -841,3 +860,256 @@ def test_a_200_the_classifier_cannot_PARSE_asks_without_blaming_the_transport(ho
|
||||
assert "ask" in reason, "an unparseable rule list produced a decision instead of a question"
|
||||
assert "could not parse" in reason, "the ask blamed the transport for a 200 the classifier simply could not read"
|
||||
assert "unreachable" not in reason
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# Guard-scope freshness (ersatztv#787). `test_ci_dropped_step_guard.py` derives which jobs must
|
||||
# carry per-step execution markers from `.gitea/required-status-contexts.json`; these arms are what
|
||||
# stop that snapshot going quietly stale.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scheduled", [True, False], ids=["SCHEDULED", "IMMEDIATE"])
|
||||
def test_a_DRIFTED_contexts_snapshot_ASKS_on_BOTH_merge_paths(hook, scheduled):
|
||||
"""BOTH paths, and that parametrization is the point rather than thoroughness for its own sake.
|
||||
|
||||
The branch-protection read this repo already had lives inside the `merge_when_checks_succeed`
|
||||
branch and never runs on an immediate merge — the common case. Hanging the freshness check off
|
||||
it would have fired it only when an auto-merge was armed. This file already records that exact
|
||||
shape once: the base re-read "first landed inside the scheduled-auto-merge branch only", and
|
||||
cold review found scheduled+retarget denied while immediate+retarget auto-GRANTED.
|
||||
"""
|
||||
hook.set_branch_protection("DRIFTED-CONTEXTS")
|
||||
decision = hook.decision(scheduled=scheduled)
|
||||
assert decision is not None, "the hook passed through instead of deciding"
|
||||
out = decision["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "ask", f"a stale guard scope must ask, not {out['permissionDecision']!r}"
|
||||
assert "required-status-contexts.json" in out["permissionDecisionReason"], (
|
||||
"the ask must name the snapshot to reconcile, not merely report unease"
|
||||
)
|
||||
# ASK, never DENY — folded in here rather than kept as its own test. As a separate assertion on
|
||||
# `!= "deny"` it was VACUOUS: deleting the whole freshness arm leaves the hook GRANTING, which
|
||||
# also satisfies it, and so would any unrelated earlier ask. It only means something alongside
|
||||
# the exact-`ask` and reason assertions above, which pin WHICH arm answered.
|
||||
assert out["permissionDecision"] != "deny", (
|
||||
"drift must not deny: Gitea enforces the live required set server-side, so this merge is "
|
||||
"not unsafe — what is stale is a guard's scope, and denying would state something false "
|
||||
"about the change in front of the reader"
|
||||
)
|
||||
|
||||
|
||||
def test_the_freshness_check_does_not_PREEMPT_a_more_serious_refusal(hook):
|
||||
"""ORDERING, asserted rather than described. The freshness check is last because it can only
|
||||
downgrade a grant, and a reader whose merge is blocked for a more serious reason must be told
|
||||
THAT reason. Here branch protection does not require review-verdict/h10 at all: the deny for
|
||||
that must win over the freshness ask, even though the contexts also differ from the snapshot."""
|
||||
hook.set_branch_protection("MISSING-CONTEXT")
|
||||
out = hook.decision()["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "deny"
|
||||
assert "review-verdict/h10' is NOT a required status check" in out["permissionDecisionReason"]
|
||||
assert "required-status-contexts.json" not in out["permissionDecisionReason"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scheduled", [True, False], ids=["SCHEDULED", "IMMEDIATE"])
|
||||
def test_a_healthy_repo_still_auto_grants_with_the_freshness_check_in_place(hook, scheduled):
|
||||
"""NEGATIVE CONTROL. Every test above passes if the freshness check simply asks on everything.
|
||||
|
||||
Parametrized over BOTH paths deliberately: the IMMEDIATE path is where the branch-protection
|
||||
fetch is NEW, so it is the one where an over-eager arm would silently cost the ersatztv#314
|
||||
auto-grant — and it is the path a default `scheduled=True` control would never have covered.
|
||||
"""
|
||||
out = hook.decision(scheduled=scheduled)["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "allow", (
|
||||
f"the freshness check broke the grant path (got {out['permissionDecision']!r}: "
|
||||
f"{out['permissionDecisionReason'][:200]})"
|
||||
)
|
||||
|
||||
|
||||
def test_a_MISSING_checker_asks_rather_than_skipping_the_freshness_check(hook, tmp_path):
|
||||
"""A guard whose implementation has gone missing must ASK, never quietly not run.
|
||||
|
||||
The `unknown must FAIL, not warn` arm: a checker that cannot be executed produces no comparison,
|
||||
and no comparison is not a passing one. The hook resolves the checker from its OWN location
|
||||
(`$repo_root`) rather than `$CLAUDE_PROJECT_DIR`, because an env var is not a sound input to a
|
||||
security decision — so this exercises it by running a COPY of the hook from a tree that is
|
||||
complete except for the checker, which is the only way to move `$repo_root`.
|
||||
"""
|
||||
project = tmp_path / "repo"
|
||||
(project / ".claude" / "hooks").mkdir(parents=True)
|
||||
(project / "scripts").mkdir()
|
||||
(project / ".gitea").mkdir()
|
||||
# Everything the arms BEFORE this one resolve, so the decision isolates the missing checker
|
||||
# rather than tripping the changed-files or verdict-classifier arms.
|
||||
for rel in ("scripts/pr-changed-files.sh", "scripts/check-review-verdict.sh"):
|
||||
shutil.copy2(REPO_ROOT / rel, project / rel)
|
||||
shutil.copytree(REPO_ROOT / "scripts" / "lib", project / "scripts" / "lib")
|
||||
shutil.copy2(REPO_ROOT / ".gitea" / "required-status-contexts.json", project / ".gitea")
|
||||
copied_hook = project / ".claude" / "hooks" / HOOK.name
|
||||
shutil.copy2(HOOK, copied_hook)
|
||||
assert not (project / "scripts" / "check-required-contexts.sh").exists()
|
||||
|
||||
# IMMEDIATE path, so the scheduled branch's own branch-protection read cannot muddy the
|
||||
# no-fetch assertion below.
|
||||
out = hook.decision(scheduled=False, project_dir=str(project), hook_path=copied_hook)["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "ask", (
|
||||
f"a missing checker must ask, not {out['permissionDecision']!r} — no comparison ran"
|
||||
)
|
||||
# WORDING UNIQUE TO THIS ARM, and then the behaviour only this arm produces. Asserting the
|
||||
# substring `check-required-contexts.sh` was NOT enough: delete the `[ ! -x ]` clause and the
|
||||
# exec simply fails, `ctx_class` is empty, and the catch-all arm asks with a reason ending
|
||||
# "Check scripts/check-required-contexts.sh." — satisfying both of the original assertions while
|
||||
# proving nothing about the clause this test names. The two guards then mask each other (#685).
|
||||
assert "missing or not executable" in out["permissionDecisionReason"], (
|
||||
f"the ask came from a different arm: {out['permissionDecisionReason'][:200]}"
|
||||
)
|
||||
assert not hook.branch_protection_urls(), (
|
||||
"the hook fetched branch protection before noticing its checker was missing — only the "
|
||||
"early `[ ! -x ]` clause prevents that fetch, so this proves the clause ran"
|
||||
)
|
||||
|
||||
|
||||
def test_an_UNREADABLE_class_gets_its_OWN_arm_not_the_catch_all(hook):
|
||||
"""`unreadable` is a class the checker's contract DECLARES. Letting it fall through to the
|
||||
catch-all asks for the right reason with the wrong explanation — it tells the operator the
|
||||
checker returned a word the hook does not understand and sends them to debug the script, when
|
||||
what actually happened is that branch protection or the snapshot came back malformed."""
|
||||
hook.set_branch_protection("NON-STRING-MEMBER")
|
||||
# THE IMMEDIATE PATH, deliberately. On the scheduled path the h10 required-check arm sees the
|
||||
# same malformed payload and asks first, so asserting there proves nothing about this arm — the
|
||||
# first draft of this test did exactly that and passed without ever reaching the code it names.
|
||||
reason = hook.decision(scheduled=False)["hookSpecificOutput"]["permissionDecisionReason"]
|
||||
assert "could not consume" in reason, f"the unreadable arm did not fire; some other arm answered: {reason[:200]}"
|
||||
assert "not a class this hook understands" not in reason, (
|
||||
f"a declared class reached the catch-all arm: {reason[:200]}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_repo_gate_is_CASE_INSENSITIVE_like_Gitea_itself(hook):
|
||||
"""Gitea resolves owner/repo case-insensitively — verified live, `/repos/TIMOTHY/ErsatzTV` and
|
||||
`/repos/timothy/ersatztv` both answer 200. A byte-exact gate would let a case variant skip the
|
||||
freshness arm entirely while every other arm still resolved, so drift would go unreported with
|
||||
no ask: the gate failing open on a spelling."""
|
||||
hook.set_branch_protection("DRIFTED-CONTEXTS")
|
||||
out = hook.decision(owner="TIMOTHY", repo="ErsatzTV")["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "ask", (
|
||||
f"a case variant of this repo skipped the freshness arm (got {out['permissionDecision']!r})"
|
||||
)
|
||||
assert "required-status-contexts.json" in out["permissionDecisionReason"]
|
||||
|
||||
|
||||
def test_the_freshness_arm_does_NOT_fire_for_a_DIFFERENT_repo(hook):
|
||||
"""The arm compares a hardcoded `main` against a snapshot committed in THIS checkout, while the
|
||||
merge tool is called with whatever owner/repo the user is merging. Running it for another repo
|
||||
weighs that repo's live contexts against this repo's mirror: measured, `server-management`
|
||||
returns `[]`, which classifies as `nomatch` and would have asked with a reason naming
|
||||
`.gitea/required-status-contexts.json` — a confident statement about a repo it does not describe.
|
||||
"""
|
||||
# The foreign repo's protection must actually DIFFER from this repo's snapshot, or the arm would
|
||||
# report `match` and stay silent whether or not it ran — which is how the first draft of this
|
||||
# test passed with the repo gate disabled. Asserting the absence of a string only means something
|
||||
# when the string WOULD be there without the guard.
|
||||
hook.set_branch_protection("DRIFTED-CONTEXTS")
|
||||
out = hook.decision(repo="server-management")["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "allow", (
|
||||
"a repo this snapshot does not describe must not be judged by it; the gate should be "
|
||||
f"skipped and the merge auto-granted, got {out['permissionDecision']!r}: "
|
||||
f"{out['permissionDecisionReason'][:200]}"
|
||||
)
|
||||
assert "required-status-contexts.json" not in out["permissionDecisionReason"], (
|
||||
"the guard-scope freshness arm fired for a repo its snapshot does not describe: "
|
||||
f"{out['permissionDecisionReason'][:200]}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_HOOK_loads_the_SHARED_classifier_behaviourally(hook, tmp_path):
|
||||
"""The other half of the one-copy property, proven by swapping the file rather than reading source.
|
||||
|
||||
Cold review re-inlined a BYTE-IDENTICAL copy of the classifier into this hook, left a comment
|
||||
naming the shared path above it, and the whole suite stayed green — a substring assertion cannot
|
||||
tell a loaded file from a mentioned one, and the byte-identical inline is exactly the refactor
|
||||
that drifts later because it agrees today. Replacing the shared program with a sentinel that can
|
||||
only answer `undecidable` must therefore change what the hook DECIDES.
|
||||
"""
|
||||
project = tmp_path / "repo"
|
||||
(project / ".claude" / "hooks").mkdir(parents=True)
|
||||
(project / "scripts").mkdir()
|
||||
(project / ".gitea").mkdir()
|
||||
for rel in ("scripts/pr-changed-files.sh", "scripts/check-review-verdict.sh", "scripts/check-required-contexts.sh"):
|
||||
shutil.copy2(REPO_ROOT / rel, project / rel)
|
||||
(project / "scripts" / "lib").mkdir()
|
||||
(project / "scripts" / "lib" / "branch-rule-classifier.jq").write_text(
|
||||
'# sentinel: answers only one way, so a caller that loads THIS file changes\n{verdict: "undecidable"}\n'
|
||||
)
|
||||
shutil.copy2(REPO_ROOT / ".gitea" / "required-status-contexts.json", project / ".gitea")
|
||||
copied_hook = project / ".claude" / "hooks" / HOOK.name
|
||||
shutil.copy2(HOOK, copied_hook)
|
||||
|
||||
# SCHEDULED, because that is the path whose rule selection consumes the classifier.
|
||||
out = hook.decision(scheduled=True, project_dir=str(project), hook_path=copied_hook)["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "ask", (
|
||||
f"swapping the shared classifier did not change the hook's decision (got "
|
||||
f"{out['permissionDecision']!r}) — it is not loading that file"
|
||||
)
|
||||
assert "decidably" in out["permissionDecisionReason"], (
|
||||
"the hook decided for some other reason than the sentinel classifier's `undecidable`: "
|
||||
f"{out['permissionDecisionReason'][:200]}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("mode", "phrase"),
|
||||
[
|
||||
pytest.param("EMPTY-LIST", "no branch-protection rule governs 'main' at all", id="nomatch"),
|
||||
pytest.param("GLOB-RULE", "a glob branch-protection rule could govern 'main'", id="undecidable"),
|
||||
pytest.param("LIST-UNREADABLE", "could not read branch protection for the guard-scope", id="readfail"),
|
||||
],
|
||||
)
|
||||
def test_each_remaining_freshness_class_gets_its_OWN_arm(hook, mode, phrase):
|
||||
"""One arm of four had a proof; the other three were carried by the catch-all.
|
||||
|
||||
Fail-safe is not the same as proven: this file's own argument in the `unreadable` test — an arm
|
||||
that asks "for the right reason with the wrong explanation" — applies verbatim to `nomatch`,
|
||||
`undecidable` and `readfail`. Measured before this test existed: deleting all three arms left the
|
||||
file green at 47 passed, because the catch-all answered for them and blamed the checker.
|
||||
|
||||
The IMMEDIATE path throughout: on the scheduled path the h10 required-check arm consumes these
|
||||
same payloads and answers first, which is how an earlier test in this file passed without
|
||||
reaching the code it named.
|
||||
"""
|
||||
hook.set_branch_protection(mode)
|
||||
reason = hook.decision(scheduled=False)["hookSpecificOutput"]["permissionDecisionReason"]
|
||||
assert phrase in reason, f"the {mode} payload did not reach its own arm: {reason[:220]}"
|
||||
assert "not a class this hook understands" not in reason, (
|
||||
f"a declared class fell through to the catch-all: {reason[:220]}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_checker_USAGE_error_reaches_the_operator_WITH_its_cause(hook, tmp_path):
|
||||
"""The catch-all must name what actually happened, not just that something did.
|
||||
|
||||
The checker exits 2 with a diagnostic on a usage error — an unreadable snapshot, a branch
|
||||
mismatch, a missing shared classifier. While its stderr went to /dev/null every one of those
|
||||
arrived as "returned 'nothing'", which names no cause: the same states-a-cause-that-did-not-happen
|
||||
shape this arm avoids elsewhere. Here the classifier is absent, so the checker exits 2 saying so.
|
||||
"""
|
||||
project = tmp_path / "repo"
|
||||
(project / ".claude" / "hooks").mkdir(parents=True)
|
||||
(project / "scripts").mkdir()
|
||||
(project / ".gitea").mkdir()
|
||||
for rel in ("scripts/pr-changed-files.sh", "scripts/check-review-verdict.sh", "scripts/check-required-contexts.sh"):
|
||||
shutil.copy2(REPO_ROOT / rel, project / rel)
|
||||
# `scripts/lib/` IS copied, minus the one file under test: omitting the whole directory also
|
||||
# removes `review-verdict-vocabulary.sh`, which `check-review-verdict.sh` sources — and that arm
|
||||
# answers first, so the test would assert against a completely different failure.
|
||||
shutil.copytree(REPO_ROOT / "scripts" / "lib", project / "scripts" / "lib")
|
||||
(project / "scripts" / "lib" / "branch-rule-classifier.jq").unlink()
|
||||
shutil.copy2(REPO_ROOT / ".gitea" / "required-status-contexts.json", project / ".gitea")
|
||||
copied_hook = project / ".claude" / "hooks" / HOOK.name
|
||||
shutil.copy2(HOOK, copied_hook)
|
||||
|
||||
out = hook.decision(scheduled=False, project_dir=str(project), hook_path=copied_hook)["hookSpecificOutput"]
|
||||
assert out["permissionDecision"] == "ask"
|
||||
assert "classifier not readable" in out["permissionDecisionReason"], (
|
||||
"the checker's own diagnostic was discarded, so the operator is told only that it returned "
|
||||
f"nothing: {out['permissionDecisionReason'][:220]}"
|
||||
)
|
||||
|
||||
@@ -407,8 +407,10 @@ def test_the_walks_ASSUMPTIONS_about_the_document_still_hold() -> None:
|
||||
"""The derivation's scope mirrors two properties of the OpenAPI document. Check them.
|
||||
|
||||
`testing.guard-derives-population-from-source`: "When the scope itself MIRRORS an authoritative
|
||||
source, the mirror needs its own equality check or a dated staleness marker, or the guard is
|
||||
complete within a scope that has silently gone stale." Two such assumptions are baked into
|
||||
source, the mirror needs its own equality check against that source, or the guard is complete
|
||||
within a scope that has silently gone stale." (That rule used to offer "or a dated staleness
|
||||
marker" as an alternative; #787 retired it — a date records when someone last looked, so it goes
|
||||
stale in exactly the circumstance it exists to report.) Two such assumptions are baked into
|
||||
`_request_reachable`, and both are true of the document today (2026-08-22) rather than
|
||||
guaranteed by anything:
|
||||
|
||||
|
||||
@@ -60,7 +60,12 @@ CLASSES = {"PINNED", "CAS", "UNSAFE-KNOWN", "N/A"}
|
||||
# safe direction" — was answered by the filter ALSO under-including. Enumerating the directories
|
||||
# costs more rows and has no blind spot; that is the trade the rule already made.
|
||||
SCOPE = (
|
||||
("scripts", ("*.sh", "*.py")),
|
||||
# `*.jq` joined in ersatztv#787, when `scripts/lib/branch-rule-classifier.jq` became the first
|
||||
# non-shell program under `scripts/` — a jq file the merge-consent hook loads to make a security
|
||||
# decision. Extensions are the SCOPE and are stated deliberately (see the doc), but a scope that
|
||||
# silently omits a new executable artifact type is the frozen-filter defect this guard is about:
|
||||
# the file would have shipped outside a population whose whole claim is that it has no filter.
|
||||
("scripts", ("*.sh", "*.py", "*.jq")),
|
||||
(".claude/hooks", ("*.sh",)),
|
||||
(".husky", ("*",)),
|
||||
(".gitea/workflows", ("*.yml", "*.yaml")),
|
||||
|
||||
Reference in New Issue
Block a user