Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / CI toolchain image resolves (push) Successful in 10s
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 27s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 10m49s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 6m41s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 7m18s
Build ErsatzTV Image / Build & push image (amd64) (push) Failing after 1m26s
#859 was filed as a wrong STATED CAUSE. It was masking a live false-open in the merge gate. Gitea reports a GLOB branch-protection rule with an EMPTY `branch_name` — the canonical name lives only in `rule_name`. Measured 2026-08-30 on a scratch repo against 1.27.1. jq's `//` fires on null and false but NOT on `""`, so `(.branch_name // .rule_name // "")` resolved every glob rule to the empty string — a name with no metacharacters — and the glob test, the entire basis of the classifier's undecidable-first ordering, never saw it. Measured on the predecessor: glob `m*` (not requiring review-verdict/h10) beside plain `main` (requiring it) resolved to `exact` on `main` and AUTO-GRANTED a scheduled merge, while Gitea — ordering by Priority then plain-name-ness — may be applying `m*`. That is #622's hole, reached through the ordering written to close it. Mirror case: a glob alone resolved to `none` and DENIED about a rule that provably governs the base. A name is now a non-empty string. Each field resolves to a NAME, a SKIP (absent/null/ empty — fall through), or POISON (present, wrong type — poisons whichever field carries it). A rule with no usable name is a distinct `unreadable` verdict with its own operator cause, instead of feeding `none`, whose whole authority is "the full rule list was read and none matches". The short-circuit is STRUCTURAL: jq binds `as` eagerly, so the flat form still evaluated `offs`/`nonascii` on the bad name and died before reaching the arm meant to prevent that. Also #859: `branch_protections` is fetched ONCE per run, not twice. The round trip is the smaller half — it is mutable config, so two reads can disagree and the two arms then decide about different repo states with neither able to notice. #858: `verdict_script` resolves from `$repo_root`, not `$CLAUDE_PROJECT_DIR`. And the finding that mattered more — `ETV_HOOK_FIRE_LIB` is `. `-SOURCED, so it is CODE running before stdin is read and before `decide` exists. A first draft exempted it as "telemetry, not a predicate"; cold review refuted that by execution: a decoy hook-fire-log.sh in an env-var-named tree printing an allow and exiting 0 GRANTS THE MERGE, bypassing every check. Classify a path by how it is CONSUMED, never by what it is called. This hook's copy is self-located; the other twelve are #891 (high/security), which records the reachable case — husky launches the prepush hooks by RELATIVE path, so the two roots diverge there. check-required-contexts.sh gains an array-type gate (a JSON object previously printed `nomatch`, a positive claim about server config from a body it cannot consume). Verification: 1377 passed / 2 skipped; 11 declared mutants, 11 detected, disjoint reddened sets; classifier executed across jq 1.8.2 and 1.6 with identical results; both env-var tests ship a negative control, because the passing outcome is also what an inert decoy produces. Four cold review rounds plus a bounded prose check. Every round found defects the previous round's fixes introduced — a type conflation that re-opened the auto-grant, a comment asserting the opposite of the line its own commit changed, and a corrected sentence whose identical twin survived in the same diff. Docs: new record `process.hook-resolves-inputs-from-repo-root`; both inline sites cite it rather than arguing it twice. docs/remote-state-inventory.md's row for the second read updated. Follow-ups filed: #891 (the other 12 hooks), #895 ("all N tests green" claims). fixes #858 fixes #859 Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
556 lines
28 KiB
Python
556 lines
28 KiB
Python
"""Proof for `scripts/check-required-contexts.sh` (ersatztv#787).
|
|
|
|
The script reconciles the LIVE required status checks on `main` against the committed snapshot
|
|
`.gitea/required-status-contexts.json`, from which `test_ci_dropped_step_guard.py` derives its
|
|
marked-job scope. It is a pure classifier over a payload its caller fetched, which is the whole
|
|
reason it is a script at all rather than a few lines inside `pretooluse-merge-consent.sh`: the
|
|
merge hook's own comments record that while its verdict classification lived inline it had no tests
|
|
and three false-opens survived in it (#629).
|
|
|
|
Every test here execs the REAL script. Nothing reimplements its logic — a second copy of the
|
|
classifier in the test would agree with the first by construction and prove nothing.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import shutil
|
|
import subprocess
|
|
import tempfile
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
SCRIPT = REPO_ROOT / "scripts" / "check-required-contexts.sh"
|
|
SNAPSHOT = REPO_ROOT / ".gitea" / "required-status-contexts.json"
|
|
|
|
# THE CONTRACT, pinned. `pretooluse-merge-consent.sh` maps each of these onto a hook decision and
|
|
# treats anything else as "could not tell" -> ask. Pinning the set here means a sixth class cannot be
|
|
# added to the script without this test demanding the hook learn about it: an unmapped class would
|
|
# otherwise reach the hook's catch-all and read as a malfunction rather than the finding it is.
|
|
CLASSES = {"match", "drift", "nomatch", "undecidable", "unreadable"}
|
|
|
|
SNAP_CONTEXTS = json.loads(SNAPSHOT.read_text())["contexts"]
|
|
|
|
|
|
def run(payload: str, snapshot: Path | None = None, branch: str = "main") -> tuple[int, str, str]:
|
|
proc = subprocess.run(
|
|
[str(SCRIPT), "--branch", branch, "--snapshot", str(snapshot or SNAPSHOT)],
|
|
input=payload,
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
return proc.returncode, proc.stdout.strip(), proc.stderr
|
|
|
|
|
|
def rule(contexts, name="main", enabled=True):
|
|
return {"branch_name": name, "enable_status_check": enabled, "status_check_contexts": contexts}
|
|
|
|
|
|
def test_the_script_is_executable():
|
|
assert SCRIPT.is_file() and SCRIPT.stat().st_mode & 0o111, f"{SCRIPT} must exist and be executable"
|
|
|
|
|
|
def test_the_committed_snapshot_is_well_formed_and_not_empty():
|
|
"""ANTI-VACUITY. An empty snapshot would make `match` reachable against an empty live list while
|
|
`test_ci_dropped_step_guard.py` derived an empty scope and collected zero parametrized cases."""
|
|
data = json.loads(SNAPSHOT.read_text())
|
|
assert isinstance(data.get("contexts"), list) and data["contexts"]
|
|
assert all(isinstance(c, str) and c.strip() for c in data["contexts"])
|
|
assert data.get("branch") == "main"
|
|
assert data.get("read_on"), "the snapshot must date itself — it is a claim about a moment"
|
|
|
|
|
|
def test_live_equal_to_the_snapshot_is_match():
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS))]))[1] == "match"
|
|
|
|
|
|
def test_the_comparison_is_by_SET_not_by_order():
|
|
"""Gitea does not promise an order, so an order-sensitive compare would report drift forever —
|
|
and a guard that cries wolf on every merge is one somebody switches off."""
|
|
assert run(json.dumps([rule(list(reversed(SNAP_CONTEXTS)))]))[1] == "match"
|
|
|
|
|
|
def test_an_ADDED_required_context_is_drift():
|
|
"""THE FINDING #787 EXISTS FOR: a fourth required check whose job carries no execution marker."""
|
|
code, out, _ = run(json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])]))
|
|
assert (code, out) == (0, "drift")
|
|
|
|
|
|
def test_a_REMOVED_required_context_is_drift():
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS[:-1]))]))[1] == "drift"
|
|
|
|
|
|
def test_null_contexts_against_a_non_empty_snapshot_is_drift():
|
|
"""`null` is a legitimate payload meaning "none required", not a read failure — so it is the
|
|
FINDING that every required context was removed, never `unreadable`."""
|
|
payload = json.dumps([{"branch_name": "main", "enable_status_check": True, "status_check_contexts": None}])
|
|
assert run(payload)[1] == "drift"
|
|
|
|
|
|
def test_a_rule_with_status_checks_DISABLED_is_drift_against_a_nonempty_snapshot():
|
|
"""The EFFECTIVE required set, not the raw list.
|
|
|
|
`enable_status_check: false` means Gitea requires nothing on the branch whatever
|
|
`status_check_contexts` still holds. Type-checking the flag and ignoring its VALUE certifies
|
|
disabled protection as a current mirror — measured: with that clause removed the same payload
|
|
returns `match`, and the ENTIRE suite across this file and the merge-consent file stayed green,
|
|
because `rule()`'s `enabled=` parameter had no call site that ever passed False. (Stated as an
|
|
invariant: this sentence carried a test COUNT until ersatztv#859, and a count is falsified by
|
|
the next person to add a test — which is exactly what that PR did to it.)
|
|
"""
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS), enabled=False)]))[1] == "drift"
|
|
|
|
|
|
def test_a_rule_with_status_checks_DISABLED_matches_an_EMPTY_snapshot(tmp_path):
|
|
"""The other side of the same clause: disabled protection IS an accurate mirror of a snapshot
|
|
that requires nothing, so it must not be reported as drift."""
|
|
empty = tmp_path / "snap.json"
|
|
empty.write_text(json.dumps({"repo": "timothy/ersatztv", "branch": "main", "contexts": []}))
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS), enabled=False)]), snapshot=empty)[1] == "match"
|
|
|
|
|
|
def test_no_rule_governing_the_branch_is_nomatch():
|
|
assert run(json.dumps([rule([], name="develop")]))[1] == "nomatch"
|
|
|
|
|
|
def test_a_glob_rule_makes_it_undecidable_even_when_an_exact_rule_matches():
|
|
"""ORDER MATTERS, and this is the test that pins it. Gitea picks the governing rule by Priority
|
|
with gobwas/glob semantics, so a glob rule can outrank an exactly-named one. Preferring the exact
|
|
rule would compare against a rule Gitea may not be applying and could report `match` on a branch
|
|
whose real required set has drifted — a false-open in the direction that costs something."""
|
|
payload = json.dumps([rule([], name="m*"), rule(list(SNAP_CONTEXTS))])
|
|
assert run(payload)[1] == "undecidable"
|
|
|
|
|
|
def test_two_rules_folding_to_the_same_name_are_undecidable():
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS)), rule(list(SNAP_CONTEXTS), name="MAIN")]))[1] == "undecidable"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("label", "payload"),
|
|
[
|
|
("not json", "not json at all"),
|
|
("top level object", '{"branch_name": "main"}'),
|
|
(
|
|
"enable_status_check as a string",
|
|
json.dumps([{"branch_name": "main", "enable_status_check": "true", "status_check_contexts": []}]),
|
|
),
|
|
("a non-string context member", json.dumps([rule(["ok", 123])])),
|
|
("contexts as an object", json.dumps([rule({"a": 1})])),
|
|
],
|
|
)
|
|
def test_shapes_this_decision_cannot_consume_are_unreadable(label, payload):
|
|
"""Validated to the depth it is CONSUMED at, not just the top-level type. An object-typed check
|
|
that never inspects its members is the one-level-down swallow the merge hook fixed twice."""
|
|
assert run(payload)[1] == "unreadable", label
|
|
|
|
|
|
def test_every_class_the_script_emits_is_one_the_contract_declares():
|
|
"""Every class the contract DECLARES is reachable, and these five payloads produce no other.
|
|
|
|
Deliberately not claimed: that a sixth class cannot ship. This drives five canned inputs, so a
|
|
class emitted only on a sixth input shape would leave `seen` unchanged and pass — the hook's
|
|
catch-all arm is what keeps that fail-closed, not this test. What it does catch is a declared
|
|
class going unreachable (dead contract) or one of these five silently changing.
|
|
"""
|
|
seen = {
|
|
run(json.dumps([rule(list(SNAP_CONTEXTS))]))[1],
|
|
run(json.dumps([rule([*SNAP_CONTEXTS, "x (pull_request)"])]))[1],
|
|
run(json.dumps([rule([], name="develop")]))[1],
|
|
run(json.dumps([rule([], name="m*")]))[1],
|
|
run("not json")[1],
|
|
}
|
|
assert seen == CLASSES, f"classes reached {sorted(seen)} != contract {sorted(CLASSES)}"
|
|
|
|
|
|
def test_a_usage_error_exits_2_and_prints_NO_class():
|
|
"""A broken invocation must not be mistakable for a finding. If it printed a class AND exited
|
|
non-zero, a caller reading only stdout would act on a comparison that never happened."""
|
|
proc = subprocess.run([str(SCRIPT), "--bogus"], input="[]", capture_output=True, text=True, check=False)
|
|
assert proc.returncode == 2
|
|
assert proc.stdout.strip() == ""
|
|
assert not (CLASSES & set(proc.stdout.split()))
|
|
|
|
|
|
def test_an_unreadable_snapshot_is_a_usage_error_not_a_class():
|
|
proc = subprocess.run(
|
|
[str(SCRIPT), "--snapshot", str(REPO_ROOT / "does" / "not" / "exist.json")],
|
|
input="[]",
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
assert proc.returncode == 2 and proc.stdout.strip() == ""
|
|
|
|
|
|
def test_a_snapshot_naming_NO_branch_is_a_usage_error(tmp_path):
|
|
"""The snapshot must name the branch it mirrors. Treating an absent `branch` as "compare against
|
|
whatever was asked for" reaches the same defect as a mismatched one — one branch's live contexts
|
|
weighed against another's mirror — by omission rather than by disagreement."""
|
|
bad = tmp_path / "snap.json"
|
|
bad.write_text(json.dumps({"contexts": list(SNAP_CONTEXTS)}))
|
|
code, out, err = run(json.dumps([rule(list(SNAP_CONTEXTS))]), snapshot=bad)
|
|
assert (code, out) == (2, ""), "an unbranded snapshot must be a usage error, never a class"
|
|
# THE ARM-UNIQUE WORDING. Exit 2 alone cannot see this clause: with `.branch` absent
|
|
# `snap_branch` is empty, and `--branch` can never be empty, so the MISMATCH guard on the next
|
|
# line fires anyway with the same exit code and the same empty stdout. Two guards masking each
|
|
# other (#685) — measured: deleting the required-branch clause left this file at 27 passed.
|
|
assert "names no branch" in err, (
|
|
f"the required-branch clause did not fire; the mismatch guard answered instead: {err.strip()[:160]}"
|
|
)
|
|
|
|
|
|
def test_a_snapshot_for_a_DIFFERENT_branch_is_a_usage_error():
|
|
code, out, _ = run(json.dumps([rule(list(SNAP_CONTEXTS))]), branch="develop")
|
|
assert (code, out) == (2, ""), "a branch mismatch must be a usage error, never a class"
|
|
|
|
|
|
def test_a_malformed_snapshot_is_unreadable_not_a_confident_drift(tmp_path):
|
|
"""The snapshot is validated to the same depth as the live payload. A `contexts` holding a
|
|
non-string would otherwise compare unequal against a well-formed live list and be reported as
|
|
`drift` — a confident finding derived from data that was never understood."""
|
|
bad = tmp_path / "snap.json"
|
|
bad.write_text(json.dumps({"contexts": ["ok", 7]}))
|
|
assert run(json.dumps([rule(list(SNAP_CONTEXTS))]), snapshot=bad)[1] == "unreadable"
|
|
|
|
|
|
# The clause this guard hangs on: the set comparison that turns a differing live list into `drift`.
|
|
COMPARE_CLAUSE = "($rule.status_check_contexts | sort | unique) == $snap"
|
|
|
|
|
|
def test_a_glob_that_could_NOT_govern_the_branch_does_not_block_the_comparison():
|
|
"""Rule selection is delegated to `scripts/lib/branch-rule-classifier.jq`, the SAME program
|
|
`pretooluse-merge-consent.sh` loads. A rule named `a[b` carries a metacharacter but cannot govern
|
|
`main`, so it must not make the comparison undecidable — a blanket "any metacharacter anywhere"
|
|
test would ask forever in a repo that has one, which is how a correct guard gets switched off."""
|
|
payload = json.dumps([rule([], name="a[b"), rule(list(SNAP_CONTEXTS))])
|
|
assert run(payload)[1] == "match"
|
|
|
|
|
|
CLASSIFIER = REPO_ROOT / "scripts" / "lib" / "branch-rule-classifier.jq"
|
|
MERGE_HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
|
|
|
|
|
SENTINEL_CLASSIFIER = (
|
|
"# sentinel: a classifier that answers only one way, so a caller that loads THIS file changes\n"
|
|
'{verdict: "undecidable"}\n'
|
|
)
|
|
|
|
|
|
def test_ONE_classifier_TWO_callers_BEHAVIOURALLY():
|
|
"""The structural property #787 relies on, PROVEN by swapping the file — not by reading source.
|
|
|
|
A string-presence check was the seventh test in this change to pass for the wrong reason: cold
|
|
review re-inlined a BYTE-IDENTICAL copy of the classifier into the merge hook, left a comment
|
|
naming `scripts/lib/branch-rule-classifier.jq` above it, and the whole suite stayed green at
|
|
exactly 1062 passed. A comment satisfies a substring assertion. And the byte-identical inline is
|
|
precisely the drift-creating refactor that matters, because it AGREES on day one — a weakened
|
|
copy would have been caught behaviourally.
|
|
|
|
So the proof replaces the shared file with a sentinel that can only answer `undecidable`, and
|
|
requires the caller's behaviour to change. Nothing but actually loading that path can do that.
|
|
"""
|
|
assert CLASSIFIER.is_file(), f"the shared classifier is missing at {CLASSIFIER}"
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
root = Path(tmp)
|
|
victim = root / "check-required-contexts.sh"
|
|
victim.write_text(SCRIPT.read_text(encoding="utf-8"))
|
|
victim.chmod(0o755)
|
|
(root / "lib").mkdir()
|
|
(root / "lib" / "branch-rule-classifier.jq").write_text(SENTINEL_CLASSIFIER)
|
|
|
|
payload = json.dumps([rule(list(SNAP_CONTEXTS))])
|
|
proc = subprocess.run(
|
|
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
|
input=payload,
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
assert (proc.returncode, proc.stdout.strip()) == (0, "undecidable"), (
|
|
"swapping scripts/lib/branch-rule-classifier.jq did not change this script's verdict "
|
|
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r}) — it is not loading that file, so "
|
|
"the one-copy property is a claim rather than a fact"
|
|
)
|
|
|
|
|
|
def _mutant_tree(tmp_path, source_text):
|
|
"""The script in a RUNNABLE layout: it resolves `lib/branch-rule-classifier.jq` relative to its
|
|
own directory, so a bare copy exits 2 before reaching any of its logic.
|
|
|
|
This is not incidental plumbing. The first version of the mutation proof below copied only the
|
|
script, so the mutant died on `classifier not readable` with EMPTY stdout — and an assertion that
|
|
stdout merely differed from `drift` was satisfied by that empty string. The proof passed for the
|
|
wrong reason and the `MUTATION` grade it justified was unsupported.
|
|
"""
|
|
victim = tmp_path / "mutant.sh"
|
|
victim.write_text(source_text)
|
|
victim.chmod(0o755)
|
|
shutil.copytree(REPO_ROOT / "scripts" / "lib", tmp_path / "lib")
|
|
return victim
|
|
|
|
|
|
def test_the_mutation_harness_layout_ITSELF_reproduces_the_real_verdict(tmp_path):
|
|
"""POSITIVE CONTROL for the harness below, without which it can only prove a broken copy.
|
|
|
|
The UNMUTATED script, in the same tmp layout, must reach the same verdict it reaches in the
|
|
repo. If this fails, the mutation test's red says nothing about the clause.
|
|
"""
|
|
victim = _mutant_tree(tmp_path, SCRIPT.read_text(encoding="utf-8"))
|
|
drifted = json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])])
|
|
proc = subprocess.run(
|
|
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
|
input=drifted,
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
assert (proc.returncode, proc.stdout.strip()) == (0, "drift"), (
|
|
f"the unmutated script did not report drift in the harness layout "
|
|
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r} stderr={proc.stderr.strip()[:160]!r}) — "
|
|
"the mutation proof below would then be measuring a broken copy, not a disarmed clause"
|
|
)
|
|
|
|
|
|
def test_a_MISSING_shared_classifier_is_a_usage_error_not_an_unreadable_CLASS():
|
|
"""The twin of a fix the hook made loudly, on the second caller.
|
|
|
|
Without this guard `jq -f` on a missing program fails, `verdict` is empty, and the script prints
|
|
the CLASS `unreadable` with exit 0 — reporting "the payload came back in a shape I could not
|
|
consume" about a missing LOCAL FILE. That is the states-a-cause-that-did-not-happen defect the
|
|
hook added its own `[ -r "$classifier" ]` check to avoid; measured, removing this one left the
|
|
file at 27 passed.
|
|
"""
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
victim = Path(tmp) / "check-required-contexts.sh"
|
|
victim.write_text(SCRIPT.read_text(encoding="utf-8"))
|
|
victim.chmod(0o755)
|
|
# deliberately NO lib/ beside it
|
|
proc = subprocess.run(
|
|
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
|
input=json.dumps([rule(list(SNAP_CONTEXTS))]),
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
assert proc.returncode == 2, f"a missing classifier must be a usage error, got exit {proc.returncode}"
|
|
assert proc.stdout.strip() == "", (
|
|
f"a missing local file was reported as the class {proc.stdout.strip()!r} — a cause that did not happen"
|
|
)
|
|
assert "classifier not readable" in proc.stderr
|
|
|
|
|
|
def test_MUTATION_disarming_the_set_comparison_stops_every_drift_report(tmp_path):
|
|
"""`testing.guard-ships-with-mutation-proof` (#775) — the DECLARED clause, re-run every suite.
|
|
|
|
The set comparison is the guard's entire finding. Disarm that clause alone — make it constantly
|
|
true — and a live list that genuinely differs from the snapshot must stop reporting `drift`. If
|
|
it still did, the finding comes from somewhere other than the clause this guard hangs on and
|
|
every passing test above proves nothing about it.
|
|
"""
|
|
text = SCRIPT.read_text(encoding="utf-8")
|
|
assert COMPARE_CLAUSE in text, (
|
|
"the set-comparison clause has moved or been reworded; RETARGET this mutation at its new "
|
|
"location rather than loosening the string match — a mutation that silently stops mutating "
|
|
"is the exact failure this file exists to catch"
|
|
)
|
|
mutated = text.replace(COMPARE_CLAUSE, "true", 1)
|
|
assert mutated != text and COMPARE_CLAUSE not in mutated, "the replacement did not change the source"
|
|
|
|
victim = _mutant_tree(tmp_path, mutated)
|
|
drifted = json.dumps([rule([*SNAP_CONTEXTS, "Build ErsatzTV Image / Fourth (pull_request)"])])
|
|
proc = subprocess.run(
|
|
[str(victim), "--branch", "main", "--snapshot", str(SNAPSHOT)],
|
|
input=drifted,
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
# EXACT, not `!= "drift"`. The mutant must RUN and reach the disarmed clause, so the only
|
|
# acceptable evidence is the specific wrong answer it now gives. Any exit code but 0, or an
|
|
# empty stdout, means it died on the way and proves nothing.
|
|
assert (proc.returncode, proc.stdout.strip()) == (0, "match"), (
|
|
f"the mutant did not reach the disarmed clause and report `match` "
|
|
f"(exit={proc.returncode} stdout={proc.stdout.strip()!r} stderr={proc.stderr.strip()[:160]!r}). "
|
|
"With the set comparison constantly true it must certify a drifted list as current; "
|
|
"anything else means the proof measured a broken copy rather than a disarmed guard"
|
|
)
|
|
|
|
|
|
# --- ersatztv#859: a rule with no usable NAME is `unreadable`, not `nomatch` ---------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"bad_rule",
|
|
[
|
|
{"enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS},
|
|
{"branch_name": None, "enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS},
|
|
{"branch_name": False, "enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS},
|
|
{"branch_name": 7, "enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS},
|
|
{"branch_name": "", "rule_name": "", "enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS},
|
|
],
|
|
ids=["absent", "null", "false", "numeric", "both-empty"],
|
|
)
|
|
def test_a_rule_with_an_UNUSABLE_NAME_is_unreadable_not_nomatch(bad_rule):
|
|
"""`(.branch_name // .rule_name // "")` collapsed all of these to the empty string — a VALID
|
|
name that matches nothing — so the shared classifier answered `none` and this script reported
|
|
`nomatch`: "no rule governs the branch at all (protection removed)".
|
|
|
|
That is a finding about a list the program could not read. `nomatch` is this script's only class
|
|
that asserts something POSITIVE about the server's configuration, so it is the one that must not
|
|
be reachable from an unread rule. On this caller the wrong answer was merely safe — the hook asks
|
|
on `nomatch` — while the sibling caller DENIED on it; the classifier is shared, so both are fixed
|
|
by the same change and both are pinned by their own suite.
|
|
|
|
`numeric` is here for a second reason: it used to make jq THROW, so this script reached
|
|
`unreadable` through its catch-all rather than by classifying. Same output, no longer resting on
|
|
a crash.
|
|
"""
|
|
code, out, err = run(json.dumps([bad_rule]))
|
|
assert (code, out) == (0, "unreadable"), (
|
|
f"a rule with no usable name classified as {out!r} (exit {code}, stderr {err.strip()[:160]!r}) "
|
|
"— `nomatch` asserts protection was REMOVED, which is a claim about a list that was never read"
|
|
)
|
|
|
|
|
|
def test_an_EMPTY_branch_name_falls_through_to_rule_name():
|
|
"""jq's `//` fires on null and false but NOT on `""`, so an empty `branch_name` SHADOWED a
|
|
perfectly good `rule_name` and the fully-protected branch reported `nomatch`.
|
|
|
|
REACHABLE, and measured: Gitea 1.27.1 reports a GLOB rule as `{"branch_name":"","rule_name":
|
|
"release/*"}` — verified 2026-08-30 by creating one rule of each kind on a scratch repo and
|
|
reading the list back. An earlier probe against this repo saw only its single plain rule and
|
|
wrongly concluded both fields are always populated; the positive case had to be constructed.
|
|
So every glob rule reached this classifier as a name with no metacharacters, and `nomatch` — a
|
|
positive claim that protection was REMOVED — was returned for branches a glob provably governs.
|
|
"""
|
|
payload = json.dumps(
|
|
[{"branch_name": "", "rule_name": "main", "enable_status_check": True, "status_check_contexts": SNAP_CONTEXTS}]
|
|
)
|
|
code, out, err = run(payload)
|
|
assert (code, out) == (0, "match"), (
|
|
f"an empty `branch_name` still hid the real rule name: {out!r} (exit {code}, stderr {err.strip()[:160]!r})"
|
|
)
|
|
|
|
|
|
def test_a_readable_rule_beside_an_unreadable_one_still_poisons_the_batch():
|
|
"""The whole LIST is the input, so one unreadable member is enough. A rule the program cannot
|
|
read might be the rule Gitea applies — Priority ordering is the server's, not this script's — so
|
|
classifying the readable one and ignoring the other would answer confidently about a list that
|
|
was only partly understood. Conservative direction: the caller asks instead of comparing."""
|
|
payload = json.dumps([rule(SNAP_CONTEXTS), {"branch_name": 7, "enable_status_check": True}])
|
|
code, out, _ = run(payload)
|
|
assert (code, out) == (0, "unreadable"), (
|
|
f"a readable rule beside an unreadable one produced {out!r} — the batch was classified on partial information"
|
|
)
|
|
|
|
|
|
CLASSIFIER = REPO_ROOT / "scripts" / "lib" / "branch-rule-classifier.jq"
|
|
|
|
|
|
def classify(payload: str, branch: str = "main") -> tuple[int, str]:
|
|
"""The classifier ALONE, with no script wrapping it. `check-required-contexts.sh` maps both a
|
|
declared `unreadable` verdict and a jq CRASH onto the same output word, so a test that only reads
|
|
that word cannot tell the two apart — cold review demonstrated exactly that by mutating the
|
|
classifier's structural guard to `if false`, making jq exit 5, and watching the wrapper still
|
|
print `unreadable`. Asserting the raw verdict is what pins the mechanism."""
|
|
proc = subprocess.run(
|
|
["jq", "--arg", "b", branch, "-c", "-f", str(CLASSIFIER)],
|
|
input=payload,
|
|
capture_output=True,
|
|
text=True,
|
|
check=False,
|
|
)
|
|
return proc.returncode, proc.stdout.strip()
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"bad_rule",
|
|
[
|
|
{"enable_status_check": True},
|
|
{"branch_name": None},
|
|
{"branch_name": False},
|
|
{"branch_name": 7},
|
|
{"branch_name": "", "rule_name": ""},
|
|
],
|
|
ids=["absent", "null", "false", "numeric", "both-empty"],
|
|
)
|
|
def test_the_CLASSIFIER_ITSELF_returns_the_declared_unreadable_verdict(bad_rule):
|
|
"""Exit 0 AND the verdict word, not merely the wrapper's output. A crash exits 5 with empty
|
|
stdout; a classification exits 0 with `{"verdict":"unreadable"}`. Only the second is the
|
|
contract the two callers were taught to consume."""
|
|
code, out = classify(json.dumps([bad_rule]))
|
|
assert (code, out) == (0, '{"verdict":"unreadable"}'), (
|
|
f"the classifier did not CLASSIFY an unusable name — it produced exit={code} out={out!r}. "
|
|
"An exit 5 with empty stdout is a crash the callers absorb, not the declared verdict."
|
|
)
|
|
|
|
|
|
def test_the_classifier_still_reaches_its_OTHER_verdicts():
|
|
"""Anti-vacuity. Every assertion above is satisfied by a classifier that answers `unreadable` to
|
|
everything, which would be a catastrophic permanent `ask`. These pin that the other three arms
|
|
are still reachable, so the new one is an addition rather than a swallow."""
|
|
assert classify(json.dumps([rule(SNAP_CONTEXTS)]))[1].startswith('{"verdict":"exact"'), "exact is unreachable"
|
|
assert classify(json.dumps([rule(SNAP_CONTEXTS, name="m*")])) == (0, '{"verdict":"undecidable"}'), (
|
|
"undecidable is unreachable"
|
|
)
|
|
assert classify(json.dumps([rule(SNAP_CONTEXTS, name="develop")])) == (0, '{"verdict":"none"}'), (
|
|
"none is unreachable"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"payload",
|
|
["{}", '{"a":{"branch_name":"main","enable_status_check":true,"status_check_contexts":[]}}', '"main"', "42"],
|
|
ids=["empty-object", "object-of-rules", "string", "number"],
|
|
)
|
|
def test_a_NON_ARRAY_payload_is_unreadable_not_a_finding(payload):
|
|
"""`nomatch` is this script's only class that asserts something POSITIVE about the server —
|
|
"no rule governs the branch at all (protection removed)". It must not be reachable from a body
|
|
that is not the shape this script consumes.
|
|
|
|
Measured before the gate: `{}` reached the classifier, matched no rule, and printed `nomatch`.
|
|
The merge hook has had this gate since #778; this script did not, so the same payload produced a
|
|
confident finding here and a correct refusal there. Pre-existing rather than introduced by
|
|
ersatztv#859, and fixed with it because the whole issue is arms asserting findings they never
|
|
established.
|
|
"""
|
|
code, out, _ = run(payload)
|
|
assert (code, out) == (0, "unreadable"), (
|
|
f"a non-array payload classified as {out!r} — anything but `unreadable` is a claim about "
|
|
"branch protection derived from a body that was never understood"
|
|
)
|
|
|
|
|
|
def test_an_EMPTY_array_is_still_a_real_finding():
|
|
"""NEGATIVE CONTROL for the gate above. `[]` IS the shape this script consumes and it genuinely
|
|
establishes absence — the list was read and holds no rule. A gate that swallowed it too would
|
|
turn the one payload that legitimately proves "protection removed" into a shrug, and every test
|
|
above would still pass."""
|
|
code, out, _ = run("[]")
|
|
assert (code, out) == (0, "nomatch"), (
|
|
f"the array gate swallowed a legitimately empty rule list: {out!r} — `[]` is the only "
|
|
"payload that establishes absence, and it must stay a finding"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_rule_name_poisons_even_beside_a_GOOD_branch_name():
|
|
"""Poison is field-agnostic, which the first draft of the fix got wrong in one direction.
|
|
|
|
It short-circuited on `branch_name`, so `{"branch_name":42,"rule_name":"main"}` poisoned while
|
|
`{"branch_name":"main","rule_name":42}` answered `exact` off the good field and never looked at
|
|
the malformed one — an invariant the comment claimed and the code did not hold. Gitea 1.27.1 does
|
|
not send that second shape, so this pins a property rather than a bug: "a malformed field poisons
|
|
the list" has to be true of BOTH fields or it is not the rule that is written down.
|
|
"""
|
|
for bad in (42, [], {}, True):
|
|
payload = json.dumps([{"branch_name": "main", "rule_name": bad, "enable_status_check": True}])
|
|
code, out = classify(payload)
|
|
assert (code, out) == (0, '{"verdict":"unreadable"}'), (
|
|
f"a malformed `rule_name` ({bad!r}) beside a usable `branch_name` produced {out!r} "
|
|
"— poison must not depend on which field carries it"
|
|
)
|