Files
ersatztv/scripts/tests/test_pr_changed_files.py
T
timothyandClaude Opus 5 03955675a8
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 12s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 27s
review-verdict/h10 Awaiting review verdict for 0395567
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m51s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m21s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 5m58s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 5m47s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
fix(806): assess test_pr_changed_files.py too, and correct three counts this change invalidated
Two findings from the cold review of the pushed head, one of them the more
interesting kind: a defect in a file the issue did not list.

`test_pr_changed_files.py` DERIVED A WORKFLOW POPULATION FROM `Path.glob`, and it
carries two completeness claims over it — "no OTHER workflow writes the
review-verdict status", and the delimiter ban over every workflow. Cold review
demonstrated it in a disposable clone: one untracked `.yaml` dropped in
`.gitea/workflows/` reddens both guards while absent from `git ls-files`. That is
the developer-red/CI-green shape #806 exists to remove, in a guard that was never
assessed.

It was never assessed because #806's body names four files to check and I took that
list as the population — which is this issue's own defect, one altitude up, in the
work implementing it. My first sweep printed this file among the filesystem-derived
populations and I narrowed to the issue's list without saying so. Both call sites
now come from the index, `_workflow_files` is registered in `DERIVATIONS` so both
proofs cover it, and the audit table in `docs/guard-inventory.md` says plainly that
the issue's list was a starting point rather than the population.

THREE COUNTS IN `mutation-claims-are-executed.md` WERE INVALIDATED BY THIS CHANGE
and not updated: it said thirteen manifest entries, ten behavioural reds, and twelve
of thirteen clause-provable. Adding a fourteenth entry makes those fourteen, eleven,
and thirteen of fourteen. The record is where `test_mutation_harness.py`'s own
docstring sends a reader for that measurement, and the same commit corrected the
identical sentence in `docs/guard-inventory.md` — so the tree shipped two
same-dated documents contradicting each other on one number. Swept by subject this
time rather than by the phrase I had been shown; no other record is stale.

Also:
  * `docs/guard-inventory.md` still said "what falls outside that window" one
    sentence after arguing the boundary is not a window.
  * the same file's "one implementation rather than five" read as contradicting the
    commit message's "rather than two"; both are true at different scopes and the
    sentence now says which it means.

refs #806

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 16:19:54 +02:00

3091 lines
164 KiB
Python

"""Tests for `scripts/pr-changed-files.sh` — the SHARED PR file enumeration (ersatztv#649).
Why this file exists. The enumeration used to be written twice: once in
`.claude/hooks/pretooluse-merge-consent.sh` (ADVISORY — a failure produces a human prompt) and once
inline in `.gitea/workflows/review-verdict.yml` (ENFORCED — it writes the branch-protection-required
`review-verdict/h10` status). They drifted, and in the dangerous direction: four rounds of
ersatztv#643 hardening landed on the advisory copy and never reached the enforced one, so the copy
with real authority ended up strictly weaker than the copy without.
The specific thing this suite pins is the point of ersatztv#649's second Done-when box. A round-4
review traced that the enforced copy's fail-closed behaviour on a garbage response was INCIDENTAL,
not designed: `n` came back empty, `[ "$n" -lt 50 ]` errored to false, the loop ran to MAX_PAGES and
left complete=no. The right answer, reached through a bash arithmetic error that any refactor of the
loop could have silently flipped. Every failure-path test below therefore asserts a NON-ZERO exit
explicitly, so the behaviour is a contract rather than a coincidence.
Observable contract of the script:
exit 0 -> enumeration complete and bound to the expected head; stdout is the authoritative path set
exit 1 -> could not enumerate/verify; stdout meaningless, caller MUST withhold any exemption
exit 2 -> usage error
"""
from __future__ import annotations
import json
import os
import re
import subprocess
from pathlib import Path
import pytest
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT = REPO_ROOT / "scripts" / "pr-changed-files.sh"
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "review-verdict.yml"
SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
OTHER_SHA = "b71c0d4e2f8a91b3c5d7e9f1a3b5c7d9e1f3a5b7"
# Serves paged `pulls/N/files`, plus the PR object the script re-reads to bind the enumeration.
CURL_SHIM = r"""#!/usr/bin/env python3
import json, os, sys, pathlib, urllib.parse
state = pathlib.Path(os.environ["STUB_DIR"])
args = sys.argv[1:]
url = [a for a in args if a.startswith("http")][-1]
if "/pulls/" in url and "/files" in url:
q = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
page = int(q.get("page", ["1"])[0])
pages = json.loads((state / "pages.json").read_text())
if page > len(pages):
print("[]"); sys.exit(0)
entry = pages[page - 1]
if entry == "ERROR": # transport failure on this page
sys.exit(22)
if entry == "GARBAGE": # 200 with a non-array body (proxy/error page)
print('{"message":"internal error"}'); sys.exit(0)
print(json.dumps(entry)); sys.exit(0)
if "/pulls/" in url:
# The PR object is read TWICE now: once before paging to bind the base, once after to bind both
# the head and the base (ersatztv#698). A counter distinguishes them, so a test can move either
# field at either end and the two guards can be isolated from each other.
counter = state / "pr_reads.txt"
n = int(counter.read_text()) if counter.exists() else 0
counter.write_text(str(n + 1))
sha = os.environ["STUB_SHA"]
alt = state / "pr_sha_after.txt"
if alt.exists() and n >= 1: # force-push landing between pagination round-trips
sha = alt.read_text().strip()
base = os.environ.get("STUB_BASE", "main")
base_before = state / "pr_base_before.txt"
base_after = state / "pr_base_after.txt"
if base_before.exists() and n == 0: # already retargeted when the job started
base = base_before.read_text().strip()
if base_after.exists() and n >= 1: # retargeted mid-enumeration
base = base_after.read_text().strip()
# The base's TIP sha (ersatztv#707) — independent of the branch NAME above. Defaults to a fixed
# value so an unmoved base is the default for every existing test that never touches this.
base_sha = os.environ.get("STUB_BASE_SHA", "aaaa000000000000000000000000000000000a")
base_sha_before = state / "pr_base_sha_before.txt"
base_sha_after = state / "pr_base_sha_after.txt"
if base_sha_before.exists() and n == 0:
base_sha = base_sha_before.read_text().strip()
if base_sha_after.exists() and n >= 1:
base_sha = base_sha_after.read_text().strip()
base_obj = {"ref": base}
if base_sha != "MISSING":
base_obj["sha"] = base_sha
print(json.dumps({"head": {"sha": sha}, "base": base_obj}))
sys.exit(0)
print("{}")
"""
def _rows(paths, status="modified"):
return [{"filename": p, "status": status} for p in paths]
@pytest.fixture
def enumerate_files(tmp_path):
bindir = tmp_path / "bin"
bindir.mkdir()
shim = bindir / "curl"
shim.write_text(CURL_SHIM)
shim.chmod(0o755)
state = tmp_path / "state"
state.mkdir()
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["STUB_SHA"] = SHA
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env.pop("ETV_GITEA_BASICAUTH", None)
env.pop("GITEA_TOKEN", None)
class Handle:
def __init__(self, env_, state_):
self.env = env_
self.state = state_
def set_pages(self, *pages):
(state / "pages.json").write_text(json.dumps(list(pages)))
def head_moves_to(self, sha):
(state / "pr_sha_after.txt").write_text(sha)
def base_is_already(self, ref):
"""The PR targets `ref` before the first page is requested (retargeted pre-run)."""
(state / "pr_base_before.txt").write_text(ref)
def base_moves_to(self, ref):
"""The PR is retargeted to `ref` between the files pages and the binding re-read."""
(state / "pr_base_after.txt").write_text(ref)
def base_sha_is_already(self, sha):
"""The base tip sits at `sha` for the WHOLE run — i.e. it advanced before this
enumeration started, which is outside the window the #707 guard polices."""
(state / "pr_base_sha_before.txt").write_text(sha)
(state / "pr_base_sha_after.txt").write_text(sha)
def base_sha_moves_to(self, sha):
"""The base branch's TIP advances to `sha` between the files pages and the binding
re-read, with the branch NAME unchanged (ersatztv#707)."""
(state / "pr_base_sha_after.txt").write_text(sha)
def base_sha_missing_after(self):
"""The base tip sha is absent/unparseable on the post-paging re-read."""
(state / "pr_base_sha_after.txt").write_text("MISSING")
def run(self, expected_sha=SHA, args=("timothy", "ersatztv", "42"), expected_base="main"):
argv = ["bash", str(SCRIPT), *args, expected_sha]
if expected_base is not None:
argv.append(expected_base)
return subprocess.run(argv, env=self.env, capture_output=True, text=True)
def paths(self):
"""Assert success and return the enumerated path set."""
r = self.run()
assert r.returncode == 0, f"expected success, got {r.returncode}: {r.stderr}"
return [ln for ln in r.stdout.splitlines() if ln.strip()]
def fails_closed(self):
"""The whole point: a NON-ZERO exit, asserted, not inferred."""
r = self.run()
return r.returncode != 0
return Handle(env, state)
# --- The happy path, so the failure-path assertions below cannot pass vacuously ----------------
def test_complete_enumeration_returns_every_path(enumerate_files):
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs", "README.md"]))
assert enumerate_files.paths() == ["docs/a.md", "ErsatzTV/Program.cs", "README.md"]
def test_a_rename_contributes_BOTH_sides(enumerate_files):
"""One row, two paths — the `git mv` hole. Reading `.filename` alone hides the source."""
enumerate_files.set_pages(
[
{
"filename": "docs/innocuous-note.md",
"status": "renamed",
"previous_filename": ".gitea/workflows/renovate.yml",
}
]
)
assert sorted(enumerate_files.paths()) == [".gitea/workflows/renovate.yml", "docs/innocuous-note.md"]
def test_paths_on_a_later_page_are_included(enumerate_files):
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["scripts/decisions_lib.py"]))
assert "scripts/decisions_lib.py" in enumerate_files.paths()
# --- Fail-closed contract: each of these MUST be non-zero, by design ---------------------------
def test_transport_failure_mid_pagination_fails_closed(enumerate_files):
"""The defect that started all of this: an errored page counted as zero rows and read as
'end of list', completing the enumeration over a PARTIAL list."""
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), "ERROR", _rows(["docs/tail.md"]))
assert enumerate_files.fails_closed()
def test_non_array_body_fails_closed(enumerate_files):
enumerate_files.set_pages("GARBAGE")
assert enumerate_files.fails_closed()
def test_an_OBJECT_OF_VALID_ROWS_isolates_the_top_level_array_check(enumerate_files):
"""Cold review found `test_non_array_body_fails_closed` passing for the wrong reason, and the
first attempt to fix it failed for a THIRD reason — worth recording, because both near-misses
look like coverage.
`jq`'s `all(.[]; …)` iterates an object's VALUES, so the top-level `type == "array"` check is
only load-bearing when those values would themselves validate:
* `{"message":"internal error"}` — values are strings, `.filename` on a string errors. Rejected
with the clause deleted, so it never isolated it.
* `{"filename":"docs/a.md","status":"modified"}` — a single row, but its values are still
strings. Same non-isolation, one level less obvious.
* `{"0": {"filename":"docs/a.md","status":"modified"}}` — values ARE valid rows. With the clause
deleted this validates, `length` is 1, the path is collected, and the next page ends the
enumeration cleanly: a non-array body enumerated as a complete docs-only list. That is the
fail-open, and only this shape exposes it.
"""
enumerate_files.set_pages({"0": {"filename": "docs/a.md", "status": "modified"}})
assert enumerate_files.fails_closed()
def test_rows_without_filename_fail_closed(enumerate_files):
"""`[{}]` is a well-formed array that yields no paths — a partial list wearing a valid shape."""
enumerate_files.set_pages([{}, {}])
assert enumerate_files.fails_closed()
def test_a_row_with_a_VALID_status_but_no_filename_isolates_the_filename_check(enumerate_files):
"""Same wrong-reason problem: `[{}]` is also rejected by the closed `.status` allow-list, since
an absent status is not in it. A row carrying a legitimate `status` and no `filename` removes
that second reason, leaving only the guard under test."""
enumerate_files.set_pages([{"status": "modified"}])
assert enumerate_files.fails_closed()
def test_array_of_scalars_fails_closed(enumerate_files):
enumerate_files.set_pages(["docs/a.md", "docs/b.md"])
assert enumerate_files.fails_closed()
@pytest.mark.parametrize("evil", ["safe.md\ndocs/Program.cs", "safe.md\rdocs/Program.cs"])
def test_CRLF_in_filename_fails_closed(enumerate_files, evil):
"""A newline splits one path into two lines that are each allow-list-matched separately, so
`safe.md\\ndocs/Program.cs` reads as two exempt paths while the real path ends in .cs."""
enumerate_files.set_pages(_rows([evil]))
assert enumerate_files.fails_closed()
def test_CRLF_in_previous_filename_on_a_NON_renamed_row_fails_closed(enumerate_files):
"""The hole one predicate wide: `previous_filename` is CONSUMED on every row, so it must be
VALIDATED on every row — not only where `.status == "renamed"` makes it semantically expected."""
enumerate_files.set_pages(
[{"filename": "docs/a.md", "status": "modified", "previous_filename": "safe.md\ndocs/Program.cs"}]
)
assert enumerate_files.fails_closed()
def test_dotdot_path_component_fails_closed(enumerate_files):
"""The callers' allow-lists anchor `^docs/`, which `docs/../ErsatzTV/Program.cs` matches."""
enumerate_files.set_pages(_rows(["docs/../ErsatzTV/Program.cs"]))
assert enumerate_files.fails_closed()
@pytest.mark.parametrize("status", ["Renamed", "moved", ""])
def test_status_outside_the_closed_allow_list_fails_closed(enumerate_files, status):
"""Without a closed set, the `renamed => previous_filename REQUIRED` clause is dodgeable by any
other value, letting a `git mv` drop its source path and read as docs-only."""
enumerate_files.set_pages([{"filename": "docs/a.md", "status": status, "previous_filename": "ErsatzTV/Program.cs"}])
assert enumerate_files.fails_closed()
def test_renamed_row_without_previous_filename_fails_closed(enumerate_files):
enumerate_files.set_pages([{"filename": "docs/a.md", "status": "renamed"}])
assert enumerate_files.fails_closed()
def test_head_differing_from_the_expected_sha_fails_closed(enumerate_files):
"""Narrowed to what this actually proves, per cold review.
The stub serves the alternate sha from the BINDING read (the one after paging; since
ersatztv#698 the script also reads the PR object BEFORE paging, to bind the base). So this
exercises "the final head does not equal the expected sha", not movement *during* enumeration.
The distinction matters: an A->B->A force-push round trip would restore the expected sha and
pass this check while the pages came from two different states. That race is inherent to
enumerating a mutable list over several round-trips against an API with no commit-pinned files
endpoint, and is tracked separately rather than papered over with a test name that implies it is
covered.
"""
enumerate_files.set_pages(_rows(["docs/a.md"]))
enumerate_files.head_moves_to(OTHER_SHA)
assert enumerate_files.fails_closed()
def test_missing_credentials_fails_closed(enumerate_files):
enumerate_files.set_pages(_rows(["docs/a.md"]))
enumerate_files.env.pop("ETV_GITEA_TOKEN", None)
assert enumerate_files.fails_closed()
@pytest.mark.parametrize("args", [("timothy", "ersatztv"), ("timothy", "ersatztv", "")])
def test_usage_errors_exit_2(enumerate_files, args):
enumerate_files.set_pages(_rows(["docs/a.md"]))
r = (
enumerate_files.run(args=args)
if len(args) == 3
else subprocess.run(["bash", str(SCRIPT), *args], env=enumerate_files.env, capture_output=True, text=True)
)
assert r.returncode == 2, r.stderr
# --- The base-ref binding (ersatztv#698 route 1) ------------------------------------------------
#
# `/pulls/{n}/files` diffs against the PR's LIVE base, so retargeting changes the answer without
# moving the head. Reproduced on the real instance as probe PR #703: opened into `main`, retargeted
# mid-run, enumerated docs-only, granted `review-verdict/h10=success` while its diff against `main`
# carried a C# file.
def test_a_FOUR_argument_call_is_a_usage_error_not_an_unbound_enumeration(enumerate_files):
"""The binding is REQUIRED, not optional.
This is the test that matters most for the shape of the fix. Had the base ref been added as an
OPTIONAL 5th argument, every existing caller would have kept compiling and kept running
unbound — an opt-out that is invisible at the call site, and the caller most likely to omit it
is the one that most needed it. Dropping the argument must be LOUD.
"""
enumerate_files.set_pages(_rows(["docs/a.md"]))
r = enumerate_files.run(expected_base=None)
assert r.returncode == 2, (
"a 4-argument call was accepted, so the base binding is effectively optional and any caller "
f"that forgets it silently enumerates against a mutable base (stdout={r.stdout!r})"
)
def test_an_EMPTY_base_ref_argument_fails_closed(enumerate_files):
"""The hook passes `.base.ref` straight from PR JSON; unparseable JSON yields an empty string."""
enumerate_files.set_pages(_rows(["docs/a.md"]))
r = enumerate_files.run(expected_base="")
assert r.returncode == 2, r.stderr
def test_a_PR_already_retargeted_before_the_enumeration_fails_closed(enumerate_files):
"""The pre-paging read. Without it, a PR retargeted before the job started would enumerate
against the scratch base with every page agreeing with every other page — internally consistent
and entirely wrong."""
enumerate_files.set_pages(_rows(["docs/a.md"]))
enumerate_files.base_is_already("probe/scratch-base")
assert enumerate_files.fails_closed()
def test_a_RETARGET_DURING_the_enumeration_fails_closed(enumerate_files):
"""The post-paging read. The head never moves here, which is the entire point: the existing
head-sha binding cannot see a retarget, so removing the base half of the binding leaves this
case exempted."""
enumerate_files.set_pages(_rows(["docs/a.md"]))
enumerate_files.base_moves_to("probe/scratch-base")
assert enumerate_files.fails_closed()
def test_positive_control_an_unmoved_base_still_enumerates(enumerate_files):
"""Without this, the three tests above could pass because the 5-argument call is broken outright
rather than because the binding works."""
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs"]))
assert enumerate_files.paths() == ["docs/a.md", "ErsatzTV/Program.cs"]
def test_a_BASE_ADVANCE_DURING_enumeration_fails_closed(enumerate_files):
"""ersatztv#707. `/pulls/{n}/files` diffs EACH PAGE against the base's LIVE tip; if `main`
advances between the pre-paging read and the post-paging re-read, later pages can be diffed
against a base earlier pages never saw. Rows can drop out of the result entirely (a file `main`
no longer differs on) while later rows shift into offset ranges already consumed against the old
tip. `.base.ref` never changes here — that is the entire point of the defect and of this being an
ADDITIONAL guard layered on top of the existing retarget check, not a reclassification of it."""
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs"]))
enumerate_files.base_sha_moves_to("cccc000000000000000000000000000000000c")
r = enumerate_files.run()
assert r.returncode != 0, f"expected fail-closed, got {r.returncode}: stdout={r.stdout!r}"
assert not r.stdout.strip(), f"stdout must be meaningless on a failed enumeration: {r.stdout!r}"
assert "base" in r.stderr and "advanced" in r.stderr, (
f"stderr should name the base-advance failure distinctly, got: {r.stderr!r}"
)
assert re.search(r"[0-9a-f]{7}", r.stderr), f"stderr should name both short shas involved, got: {r.stderr!r}"
def test_positive_control_an_UNMOVED_base_sha_still_enumerates_across_pages(enumerate_files):
"""Without this, the fail-closed tests around it could be passing only because the new guard
broke the ordinary path outright rather than because it correctly distinguishes movement from
no movement. Deliberately multi-page, to prove the guard survives several round trips."""
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["ErsatzTV/Program.cs"]))
paths = enumerate_files.paths()
assert len(paths) == 51
assert "ErsatzTV/Program.cs" in paths
def test_the_707_WINDOW_base_ref_unchanged_but_TIP_advanced_still_fails(enumerate_files):
"""The exact #707 window, and the centerpiece of this suite. The base BRANCH NAME is unchanged
throughout (`main` -> `main`), so the pre-existing `.base.ref` retarget guard sees nothing to
object to — by construction, since a mere advance is not a retarget. Only the branch's TIP moved.
A test that passes here proves the NEW sha-based guard is what fired, not the old ref guard,
which cannot see this case at all."""
enumerate_files.set_pages(_rows(["docs/a.md"]), _rows(["ErsatzTV/Program.cs"]))
enumerate_files.base_sha_moves_to("dddd000000000000000000000000000000000d")
r = enumerate_files.run() # expected_base="main" throughout; ref never moves
assert r.returncode != 0, (
"base ref stayed 'main' but its tip advanced mid-enumeration; must still fail closed "
f"(got {r.returncode}, stdout={r.stdout!r}, stderr={r.stderr!r})"
)
def test_a_base_that_ADVANCED_BEFORE_the_enumeration_STILL_SUCCEEDS(enumerate_files):
"""ersatztv#707, the deadlock this guard must NOT cause — and the reason the check is scoped to
the window rather than to the base being "current".
`main` advances constantly. If the guard compared the base tip against anything other than what
it observed at the START of THIS enumeration, every open PR would fail closed on every unrelated
merge to `main` — losing its exemption for reasons that have nothing to do with it. Here the base
already sits at a tip different from the suite's default before the first page is requested, and
then holds still: that is an ordinary, healthy PR and it must enumerate normally.
Note this is deliberately NOT the same test as the unmoved-base control: that one pins "nothing
happened", this one pins "something happened, but OUTSIDE the window, so it is none of our
business."
"""
enumerate_files.base_sha_is_already("eeee000000000000000000000000000000000e")
enumerate_files.set_pages(_rows(["docs/a.md", "docs/b.md"]), [])
assert enumerate_files.paths() == ["docs/a.md", "docs/b.md"], (
"a PR whose base advanced BEFORE this enumeration began was failed closed; the guard is "
"comparing against a stale expectation instead of the tip it actually started from"
)
def test_an_UNREADABLE_base_sha_on_the_AFTER_read_fails_closed(enumerate_files):
"""Consistent with every other guard in this script: an absent/unparseable field is never read
as 'no movement' — it is fail-closed, same as the head-sha and base-ref binding above."""
enumerate_files.set_pages(_rows(["docs/a.md"]))
enumerate_files.base_sha_missing_after()
assert enumerate_files.fails_closed()
def test_the_base_comparator_is_the_BRANCH_NAME_never_the_TIP_SHA():
"""`.base.ref` is compared, never `.base.sha` — matching `post-review-verdict.sh` (ersatztv#632).
Comparing tips would fail every enumeration on every unrelated merge to `main`: a self-inflicted
deadlock dressed as a security control.
This is a STRUCTURAL assertion on purpose, and the previous version of this test is why. It was
written behaviourally as `head_moves_to(SHA)` — the sha that was ALREADY current — so it modelled
no movement at all and was simply a duplicate positive control. It would have passed just as
happily against a script comparing tip shas. The stub serves only branch names, so no behavioural
test in this harness can distinguish the two comparators; say so and assert the source instead.
"""
src = SCRIPT.read_text()
assert ".base.ref" in src, "the enumeration no longer reads .base.ref"
binding = [ln for ln in src.splitlines() if "base_before=" in ln or "base_after=" in ln]
assert binding, "no base binding assignments found"
for ln in binding:
assert ".base.ref" in ln and ".base.sha" not in ln, (
f"the base binding compares a tip sha, which deadlocks on ordinary churn: {ln.strip()!r}"
)
def test_a_SHORT_page_does_not_end_the_enumeration(enumerate_files):
"""Gitea caps `limit` at the server-wide MAX_RESPONSE_ITEMS and may return fewer than asked.
'Fewer than 50 rows means last page' would complete over a partial list without any transport
error — so termination requires a validated EMPTY page. A 30-row page followed by code must be
seen."""
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]), _rows(["ErsatzTV/Program.cs"]))
assert "ErsatzTV/Program.cs" in enumerate_files.paths()
# --- The same guard, under the runner's jq 1.6 ------------------------------------------------
#
# `test_transport_failure_mid_pagination_fails_closed` above does NOT isolate the explicit
# `if [ -z "${raw//[[:space:]]/}" ]` clause — cold review claimed this and mutation confirmed it:
# deleting that clause leaves the whole suite green on a developer Mac, because jq 1.8 rejects empty
# input on its own. jq 1.6 does not, and the runner ships 1.6 — so the one environment where the
# clause is load-bearing was the one environment with no coverage. That is the #643/#647 failure
# class exactly, reproduced in the test suite meant to prevent it.
#
# The shim is IMPORTED, not copied. A second copy of a version-quirk emulator is the same
# two-implementations-drift problem this whole issue is about, one level down.
from scripts.tests.test_merge_consent_exemption import _JQ16_SHIM # noqa: E402
@pytest.fixture
def enumerate_files_jq16(enumerate_files, tmp_path):
"""The same harness, plus a jq shim reproducing jq 1.6's empty-input `-e` exit status."""
jq = tmp_path / "bin" / "jq"
jq.write_text(_JQ16_SHIM)
jq.chmod(0o755)
return enumerate_files
def test_jq16_shim_actually_reproduces_the_quirk(enumerate_files_jq16, tmp_path):
"""Verify the verifier. A shim that failed to install would make the test below pass vacuously,
reporting the guard safe on jq 1.6 without ever exercising the quirk."""
assert enumerate_files_jq16 is not None # the fixture is what installs the shim
jq = str(tmp_path / "bin" / "jq")
empty = subprocess.run([jq, "-e", "."], input="", capture_output=True, text=True)
assert empty.returncode == 0, "the shim does not reproduce jq 1.6's empty-input exit 0"
real = subprocess.run([jq, "-e", ".a"], input='{"a":1}', capture_output=True, text=True)
assert real.returncode == 0 and real.stdout.strip() == "1", "the shim broke ordinary jq"
false = subprocess.run([jq, "-e", ".a"], input='{"a":false}', capture_output=True, text=True)
assert false.returncode == 1, "the shim broke jq's real -e semantics for a false result"
def test_transport_failure_mid_pagination_fails_closed_on_jq16(enumerate_files_jq16):
"""The property, asserted on the interpreter that actually runs it in CI."""
enumerate_files_jq16.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), "ERROR", _rows(["docs/tail.md"]))
assert enumerate_files_jq16.fails_closed()
def test_a_docs_only_pr_still_enumerates_cleanly_on_jq16(enumerate_files_jq16):
"""Positive control: the shim must not make everything fail, or the test above proves nothing."""
enumerate_files_jq16.set_pages(_rows(["docs/a.md", "docs/b.md"]))
assert enumerate_files_jq16.paths() == ["docs/a.md", "docs/b.md"]
# --- The CALLER contract: a non-zero exit must withhold the exemption, on its own -------------
#
# This is the single line the whole extraction rests on, and it was the one guard nothing pinned. A
# review mutated the hook's `if files=$(...)` into `files=$(...) || true; files_complete=yes` — i.e.
# ignore the exit status entirely — and the ENTIRE suite still passed.
#
# It survives today only by REDUNDANCY: the script writes stdout once, immediately before `exit 0`,
# so every failure path also happens to yield empty stdout, and the hook's independent
# `[ -n "$files" ]` check catches it. That is exactly the shape this PR criticises elsewhere — safe
# by accident rather than by assertion. Any future change that streams pages, or prints a partial
# list before failing, turns it into a live false exemption.
#
# So the stub below FAILS *while emitting a perfectly docs-only list*, which is the one combination
# the redundancy cannot absorb. The positive control immediately after it proves the harness can
# actually observe the difference, rather than reporting "not exempt" for some unrelated reason.
_HOOK_STUB_CURL_TEMPLATE = r"""#!/usr/bin/env python3
import json, sys
url = [a for a in sys.argv[1:] if a.startswith("http")][-1]
if "/pulls/" in url and "/files" not in url:
print(json.dumps({"head": {"sha": "%s"}, "body": "no linked issue"}))
else:
print("{}")
"""
# Percent formatting is required: the template is Python source containing literal `{}`,
# which .format() would eat.
HOOK_STUB_CURL = _HOOK_STUB_CURL_TEMPLATE % SHA
def _mirror_tree(tmp_path, stub_body):
"""Lay out a minimal repo mirror so the hook resolves OUR stub as the shared script.
The hook finds the script via `${BASH_SOURCE[0]}/../..`, so the mirror must reproduce the real
`.claude/hooks/` + `scripts/` shape rather than just dropping the stub anywhere.
"""
hooks = tmp_path / ".claude" / "hooks"
hooks.mkdir(parents=True)
(hooks / "pretooluse-merge-consent.sh").write_text(HOOK.read_text())
scripts = tmp_path / "scripts"
scripts.mkdir()
stub = scripts / "pr-changed-files.sh"
stub.write_text(stub_body)
stub.chmod(0o755)
bindir = tmp_path / "bin"
bindir.mkdir()
curl = bindir / "curl"
curl.write_text(HOOK_STUB_CURL)
curl.chmod(0o755)
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env.pop("ETV_GITEA_BASICAUTH", None)
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(
["bash", str(hooks / "pretooluse-merge-consent.sh")],
input=json.dumps(payload),
env=env,
capture_output=True,
text=True,
)
assert r.returncode == 0, r.stderr
# Exempt == passthrough == exit 0 with no decision JSON on stdout.
return r.stdout.strip() == ""
DOCS_ONLY_OUTPUT = 'printf "docs/a.md\\ndocs/b.md\\n"\n'
def test_a_FAILING_script_withholds_the_exemption_even_when_stdout_looks_docs_only(tmp_path):
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "exit 1\n")
assert exempt is False, (
"the hook granted a docs-only exemption from the stdout of a script that FAILED — the exit "
"status is not being checked"
)
def test_positive_control_the_same_output_with_exit_0_DOES_exempt(tmp_path):
"""Without this, the test above could pass for any unrelated reason and prove nothing."""
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "exit 0\n")
assert exempt is True, (
"the positive control failed, so the negative test above cannot be trusted to be measuring "
"the exit status at all"
)
def test_a_script_that_crashes_also_withholds_the_exemption(tmp_path):
"""Not every failure is a clean `exit 1` — a crash must not read as success either."""
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "kill -TERM $$\n")
assert exempt is False
def test_a_MISSING_script_withholds_the_exemption(tmp_path):
"""The transition window: the shared checkout has no such script until this lands."""
hooks = tmp_path / ".claude" / "hooks"
hooks.mkdir(parents=True)
(hooks / "pretooluse-merge-consent.sh").write_text(HOOK.read_text())
(tmp_path / "scripts").mkdir()
bindir = tmp_path / "bin"
bindir.mkdir()
curl = bindir / "curl"
curl.write_text(HOOK_STUB_CURL)
curl.chmod(0o755)
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(
["bash", str(hooks / "pretooluse-merge-consent.sh")],
input=json.dumps(payload),
env=env,
capture_output=True,
text=True,
)
assert r.returncode == 0, r.stderr
assert r.stdout.strip() != "", "a missing shared script must not grant an exemption"
# --- Drift guard: the reason this file is worth having at all ----------------------------------
#
# Scope is BOTH callers. Until the enforced workflow was rewired (the follow-up half of ersatztv#649)
# this guard could only assert the hook, which left the copy with real authority — the one that
# writes the branch-protection-required `review-verdict/h10` status — unpinned. That asymmetry was
# the entire subject of the issue, so a guard that covered only the advisory side would have been
# the same mistake one level up.
CALLERS = pytest.mark.parametrize("caller", [HOOK, WORKFLOW], ids=["advisory-hook", "enforced-workflow"])
def _code_lines(path: Path) -> str:
"""Strip comment lines.
Both callers' comments legitimately discuss the `pulls/N/files` endpoint, and a future comment
writing `pulls/$pr/files?limit=100` as an example of what NOT to do would redden these tests —
which, since `script-tests` red blocks merges via the combined status, would block the repo over
a piece of prose. `#`-prefixed works for both files: YAML comments and the shell comments inside
the workflow's `run:` block share the marker.
"""
return "\n".join(ln for ln in path.read_text().splitlines() if not ln.lstrip().startswith("#"))
@CALLERS
def test_the_caller_uses_the_shared_script(caller):
assert "scripts/pr-changed-files.sh" in _code_lines(caller), (
f"{caller.relative_to(REPO_ROOT)} no longer calls the shared enumeration"
)
@CALLERS
def test_the_caller_passes_the_BASE_REF_argument(caller):
"""Both callers must invoke the 5-argument form (ersatztv#698 route 1).
The runtime tests prove the SCRIPT rejects a 4-argument call. They cannot prove a caller still
makes a 5-argument one, and the failure is quiet in opposite directions at the two sites: the
workflow would die on `set -u` (fail-closed, but it takes every PR with it), while the hook would
pass an empty base and lose every docs-only exemption. Pin the call site itself.
Matching is deliberately narrow. A first draft keyed on "any line mentioning the script name" and
matched the enum_error MESSAGE string, failing for a reason that had nothing to do with the call.
The workflow also invokes through `"$ENUM"` rather than the literal path, so the alias is resolved
here and asserted to point at the shared script — otherwise this test could be satisfied while
`ENUM` pointed somewhere else entirely.
"""
code = _code_lines(caller)
if '"$ENUM"' in code:
assert re.search(r"^\s*ENUM=\S*scripts/pr-changed-files\.sh\s*$", code, re.M), (
f'{caller.relative_to(REPO_ROOT)} invokes "$ENUM" but ENUM is not assigned the shared enumeration script'
)
invocations = [ln for ln in code.splitlines() if re.search(r'\$\(\s*"(\$ENUM|[^"]*pr-changed-files\.sh)"', ln)]
assert invocations, f"{caller.relative_to(REPO_ROOT)} has no executable call to the shared enumeration"
for ln in invocations:
after = ln.split('"', 2)[2] if '"$ENUM"' in ln else ln.split("pr-changed-files.sh", 1)[1]
args = re.findall(r'"[^"]*\$[^"]*"', after)
assert len(args) >= 5, (
f"{caller.relative_to(REPO_ROOT)} calls the enumeration with {len(args)} quoted "
f"arguments, expected 5 including the expected base ref: {ln.strip()!r}"
)
# Counting five arguments is not enough — cold review caught that passing `"$SHA"` twice
# satisfied the count while stalling every real exemption. Name the fifth.
assert re.search(r"(?i)base", args[4]), (
f"{caller.relative_to(REPO_ROOT)} passes {args[4]} as the 5th argument; it must be the "
f"expected BASE ref: {ln.strip()!r}"
)
@CALLERS
def test_the_caller_does_not_reimplement_the_enumeration(caller):
"""Structural rather than behavioural on purpose. Behavioural equivalence tests would still pass
if someone pasted the loop back inline and kept it correct *that day* — which is exactly how the
drift happened the first time. What must be prevented is a SECOND implementation existing.
"""
# An inline `pulls/<n>/files?` fetch is the signature of a re-inlined copy. Match on the endpoint
# alone, NOT on `?limit=` — an earlier version anchored the query string, so a copy written as
# `files?page=1&limit=50` would have walked straight past a guard that exists to stop exactly
# that. Still evadable by a copy that builds the URL without a literal `?`, so this narrows the
# gap rather than closing it.
assert not re.search(r"pulls/\$?\{?\w+\}?/files\?", _code_lines(caller)), (
f"{caller.relative_to(REPO_ROOT)} appears to enumerate PR files inline again — "
"that is the duplication ersatztv#649 removed"
)
# --- The ENFORCED caller's own preconditions ---------------------------------------------------
def _workflow_steps():
import yaml
wf = yaml.safe_load(WORKFLOW.read_text())
return wf["jobs"]["set-verdict-status"]["steps"]
def _classify_step():
for step in _workflow_steps():
if "review-verdict/h10" in (step.get("run") or ""):
return step
raise AssertionError("no step in review-verdict.yml posts review-verdict/h10")
def _workflow_triggers():
"""The `on:` block, tolerating YAML 1.1's `on` -> True coercion.
`yaml.safe_load` parses the bare key `on` as the BOOLEAN True, not the string "on" — so the
obvious `wf["on"]` raises KeyError against a perfectly valid workflow. Looking up both is not
defensive padding: a test that died on a KeyError here would read as "the trigger assertion is
broken" rather than "the trigger changed", which is the wrong failure to hand a maintainer.
"""
import yaml
wf = yaml.safe_load(WORKFLOW.read_text())
on = wf.get("on", wf.get(True))
assert isinstance(on, dict), f"review-verdict.yml has no parseable `on:` mapping (got {on!r})"
return on
def test_the_workflow_trigger_is_pull_request_TARGET_scoped_to_main():
"""The definition-rewrite hole (ersatztv#672) — sibling of the base-ref checkout below.
That checkout binds the SCRIPTS this job runs to the base. It cannot bind the job DEFINITION:
Gitea resolves a `pull_request` workflow definition from the PR's own head, so a PR editing
review-verdict.yml ran its own rewritten copy and could post `review-verdict/h10=success` for
itself. Branch protection does not care who posted the context, and carries
`required_approvals: 0`.
Both halves are asserted because either alone closes nothing:
- `pull_request_target` resolves the definition from the base.
- `branches: [main]` keeps "the base" from being an attacker-pushed branch. Base resolution
without it merely moves the rewrite from the head to a scratch base — and since a commit
status is repo-global per sha (ersatztv#663), a success forged there is inherited by a later
real PR into `main` with the same head.
Parsed, not substring-matched, for the reason the checkout test gives — and here the text-level
version is actively broken rather than merely weak: `pull_request` is a PREFIX of
`pull_request_target`, so `"pull_request" in text` cannot tell the safe trigger from the
vulnerable one, and `"pull_request_target" in text` stays green when a plain `pull_request`
trigger is ADDED back alongside it.
"""
on = _workflow_triggers()
# EXACT SET, not "target present and plain absent". Cold review found the weaker pair of
# assertions green after ADDING `workflow_dispatch:` or `push:` alongside the safe trigger —
# both are ref-resolved and both get secrets, so either one restores an equivalent
# self-supplied-definition path while the test reports clean. Enumerating the two known-bad
# extra triggers would have the same hole one trigger later; pinning the whole set does not.
assert set(on) == {"pull_request_target"}, (
f"review-verdict.yml must trigger on `pull_request_target` and NOTHING else; got "
f"{sorted(map(str, on))}. Plain `pull_request` takes the workflow DEFINITION from the PR "
"head (ersatztv#672), and `push`/`workflow_dispatch` resolve it from an arbitrary ref — "
"any of them, ADDED ALONGSIDE rather than replacing, reopens the hole"
)
branches = (on["pull_request_target"] or {}).get("branches")
assert branches == ["main"], (
f"`pull_request_target.branches` is {branches!r}; it must be exactly ['main']. Base "
"resolution means the BASE branch supplies the gate, so an unfiltered trigger lets a PR "
"into an attacker-pushed base run that branch's rewritten copy (ersatztv#672)"
)
def test_no_OTHER_workflow_writes_the_review_verdict_status():
"""`review-verdict.yml` is the only workflow allowed to write `review-verdict/h10`.
The trigger test above hardens ONE file, and by construction cannot see the more general
problem: the gate is forgeable by *any* head-resolved workflow holding credentials that can POST
a commit status, not only by the gate's own definition. `docker-build.yml` is exactly that — it
triggers on `pull_request` (correctly; it builds the PR's code) and carries `ETV_STATUS_AUTH`.
A probe once confirmed those credentials could WRITE statuses, not merely read them
(ersatztv#697); a follow-up probe confirmed `REGISTRY_PASSWORD` is now scoped off
`write:repository` and the same POST returns 403 (see `ci.actions-credential-scoping`). That
closes the registry-credential route, not the general problem — `GITEA_TOKEN`, `RENOVATE_TOKEN`,
and a collaborator's own token can all still POST a status, and none of that is closed by this
test.
What this test does close is the cheap regression: a second workflow quietly starting to write
the context. It is a guard against drift, not a security boundary — a workflow can still write
the status through an indirection this scan cannot see (a variable, a script it calls). Said
plainly so nobody reads a green here as "only one writer is possible".
"""
offenders = []
for wf in _workflow_files():
if wf.name == WORKFLOW.name:
continue
# CODE lines only — `docs`-style commentary in a workflow header legitimately discusses the
# context by name, and a raw-text scan would red on the prose explaining this very rule.
if "review-verdict/h10" in _code_lines(wf):
offenders.append(wf.name)
assert not offenders, (
f"{offenders} reference `review-verdict/h10` in executable lines. Only review-verdict.yml "
"may write the gate's own status; another workflow doing so is a forgery path (see "
"ersatztv#697) or, at best, a second implementation of the gate that will drift"
)
def test_the_workflow_checks_out_the_BASE_ref_never_the_head():
"""The self-rewrite hole. This workflow judges the PR, so the PR must not supply the code that
judges it: checking out `head.sha` would let a PR rewrite `scripts/pr-changed-files.sh` to
return an empty list and exempt itself.
Asserted through a YAML parse rather than a substring search, because the dangerous edit is
subtle — `head.sha` for `base.sha` is a nine-character diff in a `with:` block — and a
text-level `"base.sha" in text` check would still pass if a second checkout step took the head
afterwards and won.
"""
checkouts = [s for s in _workflow_steps() if "actions/checkout" in (s.get("uses") or "")]
assert len(checkouts) == 1, (
f"expected exactly one checkout step, found {len(checkouts)} — a second checkout can "
"silently overwrite the base ref with the PR head"
)
with_ = checkouts[0].get("with") or {}
ref = str(with_.get("ref", ""))
assert "pull_request.base.sha" in ref, (
f"the checkout ref is {ref!r}; it must be the PR's BASE sha, so a PR cannot rewrite the gate that judges it"
)
assert "head" not in ref, f"the checkout ref {ref!r} references the PR head"
assert with_.get("persist-credentials") is False, (
"persist-credentials must be false — nothing here pushes, and a token left in .git/config "
"is handed to every script the job runs"
)
def test_the_workflow_runs_the_jq_preflight_in_FLOOR_mode_only():
"""`--expect` pins an exact jq version and fails when it drifts. That is right for the advisory
`script-tests` job and catastrophic here: this workflow writes `review-verdict/h10`, a REQUIRED
check on `main`, so a pin would turn any jq upgrade on the runner into a repo-wide merge
deadlock — a required gate failing because an upstream package manager did its job.
"""
# CODE only, for the reason `_code_lines` documents: the first draft of this assertion read the
# raw text and went red on the workflow's own comment explaining why `--expect` is banned here.
code = _code_lines(WORKFLOW)
# A bare `"jq-preflight.sh" in code` is NOT enough, and cold review was right to say so: the
# path also appears in the `if [ -x ./scripts/jq-preflight.sh ]` presence guard, so deleting the
# actual invocation would leave that substring behind and the assertion green. Require a line
# that INVOKES it.
steps = [s for s in _workflow_steps() if "jq-preflight.sh" in (s.get("run") or "")]
assert steps, (
"review-verdict.yml no longer runs the jq preflight, so the version its shell gates run "
"under is unobservable again (ersatztv#648)"
)
invocations = [
ln.strip() for ln in steps[0]["run"].splitlines() if re.match(r"^\s*(\./)?scripts/jq-preflight\.sh(\s|$)", ln)
]
assert invocations, "the jq preflight is referenced but never actually invoked"
assert "--expect" not in code, (
"review-verdict.yml must run jq-preflight.sh in floor-only mode; --expect here deadlocks "
"every merge on `main` the day the runner's jq changes"
)
assert all("--expect" not in ln for ln in invocations)
# --- The ENFORCED caller's contract, EXECUTED ---------------------------------------------------
#
# The structural guards above prove the workflow *calls* the shared script. They cannot prove it
# reacts correctly when the script FAILS — and that is precisely the mutation that survived the last
# round on the hook side: making the caller ignore the exit status left the entire suite green,
# because every failure path also happened to produce empty stdout. So the stub below FAILS while
# emitting a perfectly docs-only list, the one combination that redundancy cannot absorb.
#
# The step's `run:` block is extracted from the YAML and executed directly. That is a real
# behavioural test of the shipped text — not a paraphrase of it — at the cost of not exercising the
# runner's step wiring, which no local test can reach anyway.
WORKFLOW_STUB_CURL = r"""#!/usr/bin/env python3
import json, os, pathlib, sys
args = sys.argv[1:]
url = [a for a in args if a.startswith("http")][-1]
out = pathlib.Path(os.environ["STUB_DIR"])
if "-X" in args and args[args.index("-X") + 1] == "POST":
payload = args[args.index("-d") + 1]
(out / "posted.json").write_text(payload)
(out / "posted_url.txt").write_text(url)
# EVERY post is also appended (ersatztv#706 race 2): the repair path POSTs a second time, and a
# test that only ever saw the last write could not tell "posted success, then repaired it to
# pending" from "posted pending once".
#
# Re-serialized COMPACT rather than appended verbatim: `jq -n` pretty-prints by default, so the
# payloads arrive spanning several lines and a raw append would produce a file that is not JSONL
# at all. (Found the honest way — the first draft of this recorder broke both race-2 tests.)
with (out / "posted_all.jsonl").open("a") as fh:
fh.write(json.dumps(json.loads(payload)) + "\n")
print("{}")
sys.exit(0)
# The PR timeline, which the retarget fence counts (ersatztv#706 race 1). Checked BEFORE the status
# branch: the fence's URL does not contain "/status", but keeping the order explicit means a later
# edit to either pattern cannot silently reroute one endpoint into the other's handler.
#
# Real shapes, measured on this instance and deliberately mirrored, because the job's guards are
# type-sensitive: a NON-EMPTY page is a bare array of events, each with a `type`, and a retarget is
# `change_target_branch` (confirmed on PR #703, the route-1 reproduction, which carries exactly two;
# and on PR #717, never retargeted, which carries none). A page PAST THE END is the JSON value `null`
# — NOT an empty array (measured again at 1.27.1 on PR #752, ersatztv#751). The earlier version of
# this comment claimed measured fidelity while the terminator below printed `[]`; that discrepancy is
# why the fence's type gate was never exercised and shipped rejecting every real timeline.
if "/timeline" in url:
mode = os.environ.get("STUB_TIMELINE_MODE", "none")
page = 1
for part in url.split("?", 1)[-1].split("&"):
if part.startswith("page="):
page = int(part.split("=", 1)[1])
if mode == "empty-first-page":
# A terminator on PAGE 1 — no real page ever read (ersatztv#751). The job must NOT certify a
# zero retarget count from this, so the exemption is withheld.
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
sys.exit(0)
if mode == "unreadable":
print("<html>502 Bad Gateway</html>")
sys.exit(0)
if mode == "transport-error":
sys.exit(22)
# Page 2+ terminates the walk. THE DEFAULT IS `null`, NOT `[]`, because that is what this
# endpoint really returns past the end — measured at Gitea 1.27.1 (ersatztv#751). This stub
# printed `[]` for two rounds of #706 work while the comment above claimed it mirrored measured
# reality, so every fence test was green against a shape the server never produces, and the
# fence's `array`-only type gate — which reads `null` as unreadable — was never exercised. On the
# real instance that made `rt_ok` false for EVERY pr, so the fence withheld every exemption.
# Parameterised rather than simply corrected: `/issues/{n}/comments` really does return `[]` when
# empty, so both shapes are live on this server and the job must accept either.
if page > 1:
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
sys.exit(0)
n_before, n_after = 0, 0
if mode.startswith("stable:"):
n_before = n_after = int(mode.split(":", 1)[1])
elif mode.startswith("moves:"):
# "moves:A,B" — A retarget events on the fence's FIRST count, B on the re-count taken just
# before the POST. This is the race-1 window: the PR was retargeted while the job classified.
a, b = mode.split(":", 1)[1].split(",")
n_before, n_after = int(a), int(b)
ctr = out / "timeline_reads.txt"
seen = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(seen + 1))
n = n_before if seen == 0 else n_after
print(json.dumps([{"id": 1000 + i, "type": "change_target_branch",
"old_ref": "main", "new_ref": "scratch"} for i in range(n)]
+ [{"id": 900, "type": "comment"}]))
sys.exit(0)
# The per-POST status HISTORY (`/statuses/{sha}`), read by the race-2 post-write verification. It is
# a BARE ARRAY, one row per POST — a different shape AND a different meaning from the combined
# `/commits/{sha}/status` below, which returns an object carrying the LATEST row per context. Both
# shapes were measured on the same live head (24 rows vs 12). Modelling them identically would have
# made the post-write check untestable, since the thing it looks for is precisely a row the combined
# endpoint no longer shows.
if "/statuses/" in url:
mode = os.environ.get("STUB_HISTORY_MODE", "none")
# PAGE, honoured. THE REASON IS THE TERMINATOR, NOT THE COUNTERS — and that distinction was itself
# a finding (#751 round 5). An earlier version of this comment said the guard sits ahead of the
# read-counting modes "so the page-2 probe cannot shift 'raced row appears on read N'", by analogy
# with the combined endpoint. Measured: moving this guard AFTER the counter modes reddens NOTHING,
# because no history mode that counts reads ever issues a page-2 request — `raced=1` on page 1
# short-circuits the probe. So that half of the rationale read as a checked reason and was not,
# which is the precise class this branch exists to retire.
#
# What IS load-bearing: page 2 must terminate for every mode that describes page 1 only, or the job
# re-reads page 1's rows as a second page and concludes the list is longer than it is. Dropping
# just the `print("[]")` below reddens `test_a_PRE_EXISTING_human_row_does_NOT_trigger_a_repair`.
# (The ordering guard on the COMBINED endpoint is a different story and genuinely is counter-
# related — moving that one reddens three mid-run-race tests.)
#
# PAST THE END THIS ENDPOINT RETURNS `[]`, NOT `null` — measured at 1.27.1:
# `/statuses/{sha}?limit=100&page=99` is the two bytes `[]`, while `/commits/{sha}/status` past the
# end returns `{"statuses": null}`. A THIRD distinct empty shape on one server; the job tolerates
# both here precisely because guessing per endpoint has been wrong twice.
hist_page = 1
for part in url.split("?", 1)[-1].split("&"):
if part.startswith("page="):
try:
hist_page = int(part.split("=", 1)[1])
except ValueError:
hist_page = 1
if hist_page > 1 and mode == "second-page-garbage":
print("<html>502 Bad Gateway</html>")
sys.exit(0)
if hist_page > 1 and mode == "second-page-error":
sys.exit(22)
if hist_page > 1 and mode not in ("second-page", "second-page-garbage", "second-page-error"):
print("[]")
sys.exit(0)
if mode == "second-page":
# The history runs past page 1, so "no raced row on page 1" does not establish "no race". The
# rows are deliberately ORDINARY (no verdict, no sentinel): the point is that unread rows EXIST,
# not that a verdict was found, so the repair must fire on uncertainty alone.
print(json.dumps([{"id": 7000 + i, "context": "ci/other", "status": "success",
"creator": None, "description": "unrelated"} for i in range(3)]))
sys.exit(0)
rows = []
if mode.startswith("human-after-post"):
# The raced verdict: absent when the high-water mark is taken, present afterwards. Its id is
# ABOVE the mark, which is what makes it detectable.
ctr = out / "history_reads.txt"
seen = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(seen + 1))
if seen > 0:
rows = [{"id": 5000, "context": "review-verdict/h10", "status": "failure",
"creator": {"login": "timothy"},
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}]
elif mode == "sentinel-after-post":
# Another overlapping run repaired mid-flight: its SENTINEL lands above this run's mark, while
# the human row it records sits BELOW the mark and is therefore invisible here (round 5).
ctr = out / "history_reads.txt"
seen = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(seen + 1))
if seen > 0:
rows = [{"id": 6000, "context": "review-verdict/h10", "status": "pending",
"creator": None,
"description": "Human verdict raced this exemption write — re-post the verdict"}]
elif mode == "stale-human-already-present":
# A human verdict that was ALREADY in the history before this run — e.g. one whose recorded
# base did not match, which `read_existing_verdict` deliberately declines to honour. It must
# NOT be mistaken for a raced write, or every later run of that PR would repair its own
# exemption to pending forever.
rows = [{"id": 10, "context": "review-verdict/h10", "status": "success",
"creator": {"login": "timothy"},
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: other)"}]
print(json.dumps(rows))
sys.exit(0)
if "/status" in url:
# Configurable. Hardcoding "no verdict yet" left the ersatztv#647 emptiness guard and the
# never-overwrite short-circuit unreachable: neither could be made to fire, so mutations
# deleting them survived the whole suite.
mode = os.environ.get("STUB_STATUS_MODE", "none")
# PAGE, honoured — the job now reads page 2 to justify "no verdict exists" (ersatztv#751). The
# real endpoint pages properly: measured at 1.27.1 on a 6-context head, `?limit=3&page=2` returns
# 3 more rows and `page=9` returns the `statuses: null` terminator. Every mode below describes
# page 1 only, so page 2+ must terminate, or the job would read its own page-1 rows again and
# conclude the list is longer than it is. `twopage` is the one mode with a real second page.
status_page = 1
for part in url.split("?", 1)[-1].split("&"):
if part.startswith("page="):
try:
status_page = int(part.split("=", 1)[1])
except ValueError:
status_page = 1
# THE EMPTY SHAPE IS `statuses: null`, NOT `[]` (ersatztv#751). Measured on this instance: a head
# with no statuses returns `{"state":"pending","total_count":0,"statuses":null}` — PR #739's head
# 5fa672e2. This stub printed `{"statuses": []}` at all three no-verdict sites, which is a shape
# the server does not produce for that case, so the job's `.statuses | type == "array"` gate was
# never exercised against reality and its `exit 1` branch — posting nothing at all — was
# unreachable in the suite. Same class as the timeline terminator, one function over.
# Parameterised, not merely corrected: a head that HAS statuses really does return an array, so
# both shapes are live and the job must read either.
def empty_statuses():
if os.environ.get("STUB_STATUS_EMPTY_SHAPE", "null") == "array":
return json.dumps({"state": "pending", "total_count": 0, "statuses": []})
return json.dumps({"state": "pending", "total_count": 0, "statuses": None})
# BEFORE ANY READ-COUNTING MODE. The `appears-on-read:N` modes count how many times the job has
# LOOKED at the combined status, and the page-2 completeness probe is part of the same look, not a
# further one — letting it increment those counters shifted "the verdict appears on read N" by one
# and broke three mid-run-race tests. Page 2 also has to terminate here for every mode that
# describes page 1 only, or the job would re-read page 1's rows as a second page and conclude the
# list is longer than it is.
if status_page > 1 and mode == "page2-garbage":
print("<html>502 Bad Gateway</html>")
sys.exit(0)
if status_page > 1 and mode == "page2-error":
sys.exit(22)
if status_page > 1 and mode != "twopage":
print(empty_statuses())
sys.exit(0)
if mode.startswith("appears-on-read:"):
# A human verdict that does NOT exist at the first read and DOES exist at the re-read made
# immediately before the POST (ersatztv#706). Models a reviewer posting BLOCKED while the job
# is still enumerating — the window the first read structurally cannot see.
nth = int(mode.split(":", 1)[1])
ctr = out / "status_reads.txt"
n = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(n + 1))
if n + 1 < nth:
print(empty_statuses())
sys.exit(0)
print(json.dumps({"statuses": [
{"context": "review-verdict/h10", "status": "failure",
"creator": {"login": "timothy"},
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}]}))
sys.exit(0)
if mode.startswith("sentinel-appears-on-read:"):
# A repair sentinel written by ANOTHER, overlapping run between this job's first read and its
# last-moment re-read (ersatztv#706 round 3). Creator is null: the sentinel is machine-written.
nth = int(mode.split(":", 1)[1])
ctr = out / "status_reads.txt"
n = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(n + 1))
if n + 1 < nth:
print(empty_statuses()); sys.exit(0)
print(json.dumps({"statuses": [
{"context": "review-verdict/h10", "status": "pending", "creator": None,
"description": "Human verdict raced this exemption write — re-post the verdict"}]}))
sys.exit(0)
if mode == "twopage":
# Page 1 AND page 2 both carry decoy contexts and NO `review-verdict/h10`: the list is longer
# than one page, so "no verdict exists" is not established and the job must refuse. Models a
# head with more contexts than the server-wide page cap (measured 50 here) — which is exactly
# why the guard cannot be written as a comparison against a hardcoded limit.
print(json.dumps({"state": "pending", "total_count": 3,
"statuses": [{"context": f"ci/p{status_page}x{i}", "status": "pending"}
for i in range(3)]}))
sys.exit(0)
if mode.startswith("rows:"):
# N decoy contexts and NO `review-verdict/h10`, to exercise the truncation guard. `total_count`
# deliberately MIRRORS the page length, because that is what the real endpoint does — it is the
# count for the page returned, not for the commit (measured at 1.27.1: `?limit=1` on a 6-context
# head returns len=1, total_count=1). A stub that reported the true total would make a
# length-vs-total guard look like it worked, which is the trap this models.
n = int(mode.split(":", 1)[1])
print(json.dumps({"state": "pending", "total_count": n,
"statuses": [{"context": f"ci/decoy{i}", "status": "pending"}
for i in range(n)]}))
sys.exit(0)
if mode == "total-count-string":
# A schema-corrupted `total_count` as a STRING alongside a null `.statuses`. `jq -r` renders 0
# and "0" identically, so a text compare would accept this.
print(json.dumps({"state": "pending", "total_count": "0", "statuses": None}))
sys.exit(0)
if mode == "transport-error":
# Real `gh()` is `curl -sf`: an HTTP error exits 22 with EMPTY stdout.
sys.exit(22)
if mode == "garbage":
print("<html>502 Bad Gateway</html>")
sys.exit(0)
if mode.startswith("existing:"):
# DECOY contexts either side, because real heads carry several (a live head showed 7 rows,
# the first an unrelated CI context). Their status is deliberately `pending`, NOT `success`:
# a `success` decoy triggers the very same short-circuit as a real verdict, so dropping
# `select(.context == $c)` produced an identical outcome and the mutation survived. With
# `pending` decoys, mis-selecting means no short-circuit — the job posts, and the test sees it.
#
# The row's PROVENANCE is configurable (ersatztv#698 route 3). A status POSTed by a user
# credential carries `.creator.login` and, for a real verdict, a `Review-verdict:`
# description; one POSTed by an Actions job carries `"creator": null`. Both were measured on
# the live combined endpoint. Defaults model a HUMAN verdict, so a test that does not opt out
# exercises the never-overwrite property.
creator = os.environ.get("STUB_STATUS_CREATOR", "timothy")
desc = os.environ.get("STUB_STATUS_DESC", "Review-verdict: MERGEABLE @ a9e3e23 (base: main)")
print(json.dumps({"statuses": [
{"context": "Build & test (.NET)", "status": "pending"},
{"context": "review-verdict/h10", "status": mode.split(":", 1)[1],
"creator": ({"login": creator} if creator else None), "description": desc},
{"context": "Functional E2E", "status": "pending"}]}))
sys.exit(0)
print(empty_statuses())
sys.exit(0)
print("{}")
"""
def _run_classify(
tmp_path,
enum_stub: str | None,
author: str = "timothy",
status_mode: str = "none",
jq16: bool = False,
status_creator: str | None = "timothy",
status_desc: str = "Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
timeline_mode: str = "none",
history_mode: str = "none",
timeline_terminator: str = "null",
status_empty_shape: str = "null",
):
"""Execute the workflow's classify `run:` block with a stubbed enumeration script.
Returns the status payload the job POSTed, or None if it posted nothing.
"""
tmp_path.mkdir(parents=True, exist_ok=True) # the chained sentinel test passes sub-paths
bindir = tmp_path / "bin"
bindir.mkdir()
curl = bindir / "curl"
curl.write_text(WORKFLOW_STUB_CURL)
curl.chmod(0o755)
if jq16:
# Reproduce the runner's jq 1.6 (`-e` over EMPTY input exits 0, not 4). The shim is the one
# already used for pr-changed-files.sh in this file, so its fidelity is covered by that
# suite's own verify-the-verifier test.
jq = bindir / "jq"
jq.write_text(_JQ16_SHIM)
jq.chmod(0o755)
scripts = tmp_path / "scripts"
scripts.mkdir()
if enum_stub is not None:
enum = scripts / "pr-changed-files.sh"
enum.write_text(enum_stub)
enum.chmod(0o755)
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(tmp_path)
env["STUB_STATUS_MODE"] = status_mode
env["STUB_STATUS_CREATOR"] = status_creator or ""
env["STUB_STATUS_DESC"] = status_desc
env["STUB_TIMELINE_MODE"] = timeline_mode
env["STUB_TIMELINE_TERMINATOR"] = timeline_terminator
env["STUB_STATUS_EMPTY_SHAPE"] = status_empty_shape
env["STUB_HISTORY_MODE"] = history_mode
env.update(
{
"GITEA_TOKEN": "stub",
"BASE_URL": "http://gitea.example/api/v1",
"GITEA_BASE_URL": "http://gitea.example/api/v1",
"REPO": "timothy/ersatztv",
"PR": "42",
"SHA": SHA,
"BASE_SHA": OTHER_SHA,
# The base branch the event was raised for, threaded to the enumeration (ersatztv#698).
# Omitting it is not a soft failure: `set -u` kills the step, no status is posted, and the
# absent required check blocks the merge — fail-closed, but it would take every PR with it.
"BASE_REF": "main",
"AUTHOR": author,
"PR_URL": "http://gitea.example/timothy/ersatztv/pulls/42",
}
)
script = tmp_path / "step.sh"
script.write_text(_classify_step()["run"])
r = subprocess.run(["bash", str(script)], cwd=tmp_path, env=env, capture_output=True, text=True)
# Assert the WIRING, not only the classification. The stub accepts every POST, so a status aimed
# at the wrong endpoint, sha, host or repo would otherwise leave these tests green while the real
# required check was never written.
#
# Two ways this check could disable itself, both found by cold review of an earlier draft:
# * it was guarded by `if url_file.exists()`, so deleting the recorder in the stub turned it
# into a no-op and every test stayed green — a verifier that silently opts out;
# * it compared only the URL SUFFIX, so a POST to the right path on the WRONG HOST OR REPO
# passed. Compare the whole URL against the env this job was given.
posted = tmp_path / "posted.json"
url_file = tmp_path / "posted_url.txt"
if posted.exists():
assert url_file.exists(), (
"a status was POSTed but its URL was not recorded — the wiring assertion below would have silently skipped"
)
expected = f"{env['BASE_URL']}/repos/{env['REPO']}/statuses/{SHA}"
assert url_file.read_text().strip() == expected, (
f"the status was POSTed to {url_file.read_text().strip()!r}, expected {expected!r}"
)
return (json.loads(posted.read_text()) if posted.exists() else None), r
DOCS_ONLY = 'printf "docs/a.md\\ndocs/b.md\\n"\n'
def test_a_FAILING_enumeration_withholds_the_exemption_even_when_stdout_looks_docs_only(tmp_path):
"""The mutation that previously survived: ignore the exit status, trust stdout."""
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n")
assert posted is not None, f"the job posted no status at all: {r.stderr}"
assert posted["state"] == "pending", (
f"a docs-only exemption was granted from the stdout of a script that FAILED "
f"(state={posted['state']}, desc={posted['description']!r}) — the exit status is not being "
"checked"
)
def test_workflow_positive_control_the_same_output_with_exit_0_DOES_exempt(tmp_path):
"""Without this, the test above could pass for any unrelated reason and prove nothing."""
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 0\n")
assert posted is not None, f"the job posted no status at all: {r.stderr}"
assert posted["state"] == "success", (
"the positive control failed, so the negative test above cannot be trusted to be measuring "
f"the exit status at all (state={posted['state']}, desc={posted['description']!r})"
)
def test_a_crashing_enumeration_also_withholds_the_exemption(tmp_path):
posted, _ = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "kill -TERM $$\n")
assert posted is not None
assert posted["state"] == "pending"
def test_a_MISSING_enumeration_script_withholds_the_exemption(tmp_path):
"""A PR whose BASE predates the script's introduction. It must post an actionable `pending`
rather than dying with no status — an absent required check blocks the merge either way, but a
stalled PR with no explanation is how a gate acquires a reputation for being flaky.
"""
posted, r = _run_classify(tmp_path, None)
assert posted is not None, f"the job posted no status at all: {r.stderr}"
assert posted["state"] == "pending"
def _emitting(*paths):
body = "".join(f'printf "%s\\n" "{p}"\n' for p in paths)
return "#!/usr/bin/env bash\n" + body + "exit 0\n"
def test_a_code_file_defeats_the_docs_only_exemption(tmp_path):
"""The workflow's own classification — deliberately NOT shared with the hook, whose allow-list
is wider because a match there falls through to a human prompt rather than a green status.
"""
posted, _ = _run_classify(tmp_path, _emitting("docs/a.md", "ErsatzTV/Program.cs"))
assert posted is not None
assert posted["state"] == "pending"
def test_a_BOT_pr_touching_a_protected_path_is_NOT_exempt(tmp_path):
"""`PROTECTED` guards the BOT exemption specifically, and mutation testing is how that got
stated correctly. The first version of this test used a docs-only+protected file list and
passed even with the `PROTECTED` clause deleted — `PROTECTED` (`.claude/ .gitea/ .husky/
scripts/ docker/ci/`) and `DOCS_ONLY` (`docs/`, root `*.md`) are DISJOINT, so on the docs-only
path that clause can never fire and the `DOCS_ONLY` check was doing all the work. The test
looked like it covered the self-exemption hole and covered nothing.
Renovate lands patch bumps unattended via Gitea's own auto-merge, so a bot PR that edits the
gate, CI, the hooks, or the scripts they call is the one path where an unreviewed change to the
merge gate could actually merge itself.
"""
posted, _ = _run_classify(
tmp_path, _emitting("ErsatzTV/Program.cs", "scripts/pr-changed-files.sh"), author="renovate"
)
assert posted is not None
assert posted["state"] == "pending", (
"a bot PR editing the shared enumeration was auto-exempted — a PR that weakens the merge "
"gate must never be able to exempt itself from the merge gate"
)
def test_bot_positive_control_a_plain_bot_pr_IS_exempt(tmp_path):
"""Proves the test above measures `PROTECTED` and not merely 'bot PRs are never exempt'."""
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props"), author="renovate")
assert posted is not None
assert posted["state"] == "success", (
"the bot exemption never fires at all, so the protected-path test above proves nothing"
)
# --- The STATUS READ: the ersatztv#647 guard and the never-overwrite short-circuit -------------
#
# These were unreachable until the stub's status response became configurable. Four mutations
# survived the full suite without them, including re-introducing the literal ersatztv#647 fail-open.
def test_a_transport_failure_on_the_STATUS_READ_posts_NOTHING(tmp_path):
"""`gh()` is `curl -sf`, so an HTTP error yields exit 22 and EMPTY stdout. Reading that as "no
verdict exists" would let the job post over a real human verdict. It must fail WITHOUT posting.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="transport-error")
assert r.returncode != 0, "an unreadable status read must fail the job"
assert posted is None, "nothing may be posted when the existing verdict state is unknown"
def test_a_GARBAGE_status_response_posts_NOTHING(tmp_path):
"""A proxy error page is a 200 with a non-JSON body — not an absent verdict."""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="garbage")
assert r.returncode != 0
assert posted is None
# Two mutations to this read are NOT covered, and both are behaviourally equivalent rather than gaps:
# * `first` -> `last`: `select(.context == $c)` yields exactly ONE row, because the job reads the
# COMBINED status endpoint, which returns latest-per-context by contract. A fixture with two
# `review-verdict/h10` rows would be testing something the API does not produce.
# * `test_a_GARBAGE_status_response_posts_NOTHING` does not, on its own, defend the type guard:
# with the guard gone, the following `jq -r` fails on non-JSON and `set -e` kills the job anyway.
# That is fail-closed by REDUNDANCY. The guard's own behaviour is pinned by the jq-1.6 test below,
# which is where it actually matters.
@pytest.mark.parametrize("existing", ["success", "failure"])
@pytest.mark.parametrize(
"paths,author",
[
(("ErsatzTV/Program.cs",), "timothy"), # non-exempt: only the short-circuit can stop it
(("docs/a.md",), "timothy"), # docs-only EXEMPT
(("Directory.Packages.props",), "renovate"), # bot EXEMPT
],
)
def test_an_existing_verdict_on_this_head_is_NEVER_overwritten(tmp_path, existing, paths, author):
"""A human verdict for this exact head may already exist — the reviewer ran
post-review-verdict.sh before this job finished, or the job re-ran. Re-posting would un-approve
a reviewed head, or (worse) approve one a human marked BLOCKED.
`failure` is the sharp case: that is a human saying NO, and an exemption posted over it would
turn a rejection into a merge.
"""
posted, r = _run_classify(tmp_path, _emitting(*paths), author=author, status_mode=f"existing:{existing}")
assert r.returncode == 0, r.stderr
assert posted is None, f"overwrote an existing '{existing}' verdict on this head with {paths} as {author}"
# --- The `count -eq 0` guard, on the path where it is the ONLY guard ---------------------------
def test_an_EMPTY_enumeration_is_not_exempt_even_for_a_BOT(tmp_path):
"""The author must be a BOT for this to test anything.
With a non-bot author the blank line an empty list produces already fails `DOCS_ONLY`, so the
`[ "${count:-0}" -eq 0 ]` guard never decides the outcome — the same short-circuit that made an
earlier `PROTECTED` test vacuous. On the bot path that guard is the ONLY thing between an
unreadable-but-successful enumeration and an unattended `success`.
Verified by mutation: changing `grep -c .` to `grep -c ''` (counting the blank line, so
`count=1`) grants a bot PR `success` here while every other test stays green.
"""
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\nexit 0\n", author="renovate")
assert posted is not None, f"the job posted no status at all: {r.stderr}"
assert posted["state"] == "pending", (
"an empty file list was treated as a bot exemption — the enumeration returning nothing is "
"not evidence that nothing was changed"
)
# --- Anchors in the classifier predicates ------------------------------------------------------
def test_the_BOT_match_is_whole_line_not_substring(tmp_path):
"""`grep -qxF` is anchored; plain `grep -qF` would exempt any author whose name CONTAINS a bot
name. `ova` is a substring of `renovate`."""
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), author="ova")
assert posted is not None
assert posted["state"] == "pending", "a substring of a bot name was granted the bot exemption"
def test_DOCS_ONLY_anchors_the_markdown_extension(tmp_path):
r"""`[^/]*\.md$` must match only a top-level file ENDING in .md. Losing the `$` exempts
`evil.mdx`, and the docs-only exemption posts a green status with nobody in the loop."""
posted, _ = _run_classify(tmp_path, _emitting("evil.mdx"))
assert posted is not None
assert posted["state"] == "pending", "a .mdx file was accepted as docs-only"
def test_a_transport_failure_under_jq_1_6_STILL_posts_nothing(tmp_path):
"""The ersatztv#647 fail-open, tested where it actually lives: jq 1.6.
The shell emptiness check exists because `jq -e` over EMPTY input exits 4 on jq >= 1.7 but **0 on
jq 1.6, which is what the runner ships**. Remove that check and, on a dev Mac's 1.8, the guard
still fires and every test stays green — the bug is invisible locally.
An earlier version of this pinned the construct STRUCTURALLY instead, on the stated grounds that
"no behavioural test can catch this on a dev machine". That was wrong: this file already imports
`_JQ16_SHIM` for `pr-changed-files.sh`, so the runner's quirk is reproducible here. The structural
version was also weaker than it looked — it stripped only FULL-LINE comments, so leaving the
literal as a trailing comment on the surviving `if` satisfied it while the real guard was gone.
This test is strictly stronger: it catches that mutant, needs no comment-stripping, and fails for
the right reason. Verified by mutation under both jq versions.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="transport-error", jq16=True)
assert r.returncode != 0, (
"under jq 1.6 an unreadable status read must still fail the job — this is the exact ersatztv#647 fail-open"
)
assert posted is None, "nothing may be posted when the existing verdict state is unknown"
def test_DOCS_ONLY_anchors_the_START_of_the_path_too(tmp_path):
r"""`^(docs/|...)` must match only at the start. Losing the `^` exempts `ErsatzTV/docs/Evil.cs`,
which is a C# file — fail-OPEN, and the sibling of the `$` case above."""
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/docs/Evil.cs"))
assert posted is not None
assert posted["state"] == "pending", "a path merely CONTAINING docs/ was accepted as docs-only"
# --- Route 2: a bot ACCOUNT does not attribute the CODE (ersatztv#698) --------------------------
#
# `AUTHOR` is `pull_request.user.login` — the PR's CREATOR, which is immutable. The head a PR points
# at is not. Force-push application code onto an open Renovate branch and the PR is still authored by
# `renovate`, still touches no protected path, and was exempted. Checking the PUSHER instead does not
# help: a git author/committer is self-asserted text. So the exemption is constrained by what a
# dependency bump can legitimately BE.
#
# The allow-list is measured, not guessed: across all 11 Renovate PRs this repo has ever had, the
# paths touched were `Directory.Packages.props` (10) and `.config/dotnet-tools.json` (1). The one
# historical outlier, PR #20, touched a `.csproj` AND two C# files — and received an unattended bot
# exemption for a source change.
def test_a_BOT_pr_carrying_a_CODE_file_is_NOT_exempt(tmp_path):
"""Route 2, stated as a behaviour: the hijacked-branch case."""
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), author="renovate")
assert posted is not None
assert posted["state"] == "pending", (
"a PR authored by the renovate account was exempted while changing a C# file — the bot "
"identity is being read as attribution for code it did not write"
)
def test_a_BOT_pr_mixing_a_manifest_WITH_code_is_NOT_exempt(tmp_path):
"""The realistic shape of the attack: keep the manifest edit so the PR still looks like a bump,
and smuggle the code alongside it. A rule that asked 'does it touch a manifest' rather than 'is
EVERY path a manifest' would exempt this."""
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props", "ErsatzTV/Program.cs"), author="renovate")
assert posted is not None
assert posted["state"] == "pending", (
"a manifest edit was enough to carry a C# file through the bot exemption — the allow-list "
"is being applied as 'any' rather than 'all'"
)
def test_bot_positive_control_a_REAL_dependency_bump_IS_still_exempt(tmp_path):
"""Renovate uses platformAutomerge, so breaking this deadlocks every dependency PR. The negative
tests above prove nothing if the exemption no longer works at all."""
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props"), author="renovate")
assert posted is not None
assert posted["state"] == "success", (
f"a plain dependency bump lost its exemption ({posted['description']!r}) — every Renovate PR "
"would now stall waiting on a human verdict"
)
@pytest.mark.parametrize("path", ["Directory.Packages.props", ".config/dotnet-tools.json"])
def test_every_manifest_in_the_allow_list_is_actually_exempt(tmp_path, path):
"""Pin the SET, not one example of it. A test that only ever exercises Directory.Packages.props
cannot see a typo in any of the other three alternations."""
posted, _ = _run_classify(tmp_path, _emitting(path), author="renovate")
assert posted is not None, path
assert posted["state"] == "success", f"{path} is in the allow-list but was not exempted"
def test_the_manifest_allow_list_is_ANCHORED(tmp_path):
"""Unanchored, `Directory.Packages.props` would match `evil/Directory.Packages.props.cs`. Same
class as the two DOCS_ONLY anchoring tests above, which is why it is tested the same way."""
posted, _ = _run_classify(tmp_path, _emitting("evil/Directory.Packages.props.cs"), author="renovate")
assert posted is not None
assert posted["state"] == "pending", "the manifest allow-list matched mid-path"
def test_a_BOT_docs_only_pr_is_STILL_exempt_via_the_docs_rule(tmp_path):
"""The regression the restructure exists to prevent.
Written as an `elif` chain, a Renovate PR touching only `docs/` enters the bot branch, fails the
manifest test, and never reaches the docs-only branch — silently withdrawing an exemption the
docs-only rule grants on its own merits for ANY author. The predicates are therefore evaluated
independently and the decision made afterwards.
"""
posted, _ = _run_classify(tmp_path, _emitting("docs/note.md"), author="renovate")
assert posted is not None
assert posted["state"] == "success", (
"a docs-only PR lost its docs-only exemption merely because its author is a bot — the "
"exemptions are chained rather than composed"
)
def test_a_BOT_pr_touching_a_protected_path_is_still_NOT_exempt(tmp_path):
"""PROTECTED must keep outranking both exemptions, including the manifest one."""
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props", "scripts/evil.sh"), author="renovate")
assert posted is not None
assert posted["state"] == "pending"
# --- Route 3: an inherited `success` is re-derived unless it is an attributable human verdict ----
#
# The short-circuit used to exit on ANY existing `success`, so an exemption this job wrote was
# indistinguishable from a verdict a human wrote. Obtained once, a forged success was thereafter
# accepted unchanged on every later run, because the guard exited before looking at the PR, the base,
# the author or the files.
#
# MEASURED on Gitea 1.25.4, on the COMBINED endpoint this job reads: a status POSTed with a user
# credential carries `.creator.login` (`timothy`), one POSTed by an Actions job carries
# `"creator": null`. Both halves are required, and the test is written in the POSITIVE direction —
# short-circuit only on something identified as a human verdict — so an unrecognised shape is
# re-derived rather than trusted.
def test_a_MACHINE_written_exemption_success_is_RE_DERIVED_not_inherited(tmp_path):
"""Route 3. The status looks exactly like one this job writes: creator null, `Exempt:` wording.
The PR now carries a C# file, so re-deriving must downgrade it to `pending`. Inheriting it would
leave a forged exemption standing forever.
"""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator=None,
status_desc="Exempt: docs-only change (no code, no protected path)",
)
assert posted is not None, (
"an existing machine-written success was inherited unchanged — this is route 3, and it is "
"how a forgery obtained once survives every subsequent run"
)
assert posted["state"] == "pending"
def test_a_success_with_NO_creator_but_a_VERDICT_LOOKING_description_is_re_derived(tmp_path):
"""Isolates the creator half. Both conditions are required; either alone is forgeable by the
other party. If a future Gitea populates `creator` for Actions, the description half still
fails — the guard degrades toward re-deriving, never toward trusting."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator=None,
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
)
assert posted is not None, "a creatorless status was trusted on the strength of its description"
assert posted["state"] == "pending"
def test_a_success_with_a_creator_but_an_EXEMPT_description_is_re_derived(tmp_path):
"""Isolates the description half, the mirror of the test above."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Exempt: docs-only change (no code, no protected path)",
)
assert posted is not None, "a status was trusted on the strength of its creator alone"
assert posted["state"] == "pending"
def test_an_UNRECOGNISED_success_shape_is_re_derived_rather_than_trusted(tmp_path):
"""The direction-of-test check. Written as 'skip if it looks machine-written', anything novel
would fall through to TRUSTED. Written as 'skip only if positively identified as human', novel
shapes are re-derived. This test fails under the first spelling and passes under the second."""
posted, _ = _run_classify(
tmp_path, _emitting("ErsatzTV/Program.cs"), status_mode="existing:success", status_creator="", status_desc=""
)
assert posted is not None
assert posted["state"] == "pending"
@pytest.mark.parametrize("existing", ["success", "failure"])
def test_a_REAL_human_verdict_is_still_NEVER_overwritten(tmp_path, existing):
"""The property the short-circuit exists for, which the fix must not break.
`failure` is the sharp case: that is a human saying NO, and an exemption posted over it would
turn a rejection into a merge. Deliberately paired with a docs-only file list, so the job WOULD
have posted an exemption `success` had it not short-circuited — without that, a passing test
would prove only that nothing was posted for some unrelated reason.
"""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode=f"existing:{existing}",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
)
assert r.returncode == 0, r.stderr
assert posted is None, (
f"overwrote a human '{existing}' verdict written by timothy — the never-overwrite property "
"has been lost while fixing route 3"
)
def test_a_PENDING_status_from_a_previous_run_is_replaced_normally(tmp_path):
"""`pending` was never short-circuited and must stay that way, or a PR that becomes exempt after
an earlier pending run could never reach `success`."""
posted, _ = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc="Awaiting review verdict for a9e3e23",
)
assert posted is not None
assert posted["state"] == "success"
@pytest.mark.parametrize("path", ["web/package.json", "web/package-lock.json"])
def test_the_npm_manifests_are_NOT_exempt(tmp_path, path):
"""Cross-family review called an earlier draft's inclusion of these a Blocker, correctly.
`renovate.json` sets `enabledManagers: ["nuget", "github-actions", "dockerfile"]`, so Renovate does
not manage npm here at all — the entry bought nothing. Meanwhile `package.json` carries `scripts`
that CI EXECUTES (`npm ci`, `npm run build`), so exempting it lets a hijacked bot branch run
arbitrary shell in CI while every path still matches a "manifest" allow-list. Widening an exemption
to a code-execution vector for no operational benefit is strictly worse than the hole being closed.
"""
posted, _ = _run_classify(tmp_path, _emitting(path), author="renovate")
assert posted is not None
assert posted["state"] == "pending", (
f"{path} was exempted; package.json scripts execute in CI, and npm is not a managed ecosystem in this repo"
)
# --- The `grep -q` + `pipefail` inversion (found by cross-family review of this PR) ---------------
#
# `grep -q` exits at its FIRST match, so the upstream writer takes SIGPIPE (141) once the path list
# exceeds the pipe buffer. Under `set -o pipefail` the pipeline is then a FAILURE even though grep
# MATCHED, inverting every guard built on `printf … | grep -q`. Reproduced at 171KB / 1901 paths,
# comfortably inside the enumerator's 2000-file cap.
#
# These are the regression guards, and they are the tests the original round did not have: every
# earlier case used a handful of short paths, far below the buffer, so the whole class was invisible.
# The construct PREDATES #698, so `main` carried this hole with no retarget or bot account required.
def _many_docs(n=1900):
return [f"docs/{'d' * 40}-{i:040d}.md" for i in range(n)]
def test_a_LARGE_pr_with_a_code_file_is_not_classified_docs_only(tmp_path):
"""The code file goes FIRST so `grep -qv` matches immediately and the writer is left with ~171KB
still to push — the exact shape that produced exit 141 and `docs_only=yes`."""
posted, r = _run_classify(tmp_path, _emitting("A.cs", *_many_docs()))
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending", (
"a PR containing A.cs was exempted as docs-only because the classification pipeline was "
f"inverted by SIGPIPE on a large file list (desc={posted['description']!r})"
)
def test_a_LARGE_pr_touching_a_PROTECTED_path_still_voids_the_exemptions(tmp_path):
"""The worse direction: here the inversion makes the PROTECTED guard MISS, so a PR editing the
gate's own workflow falls through to the docs-only exemption and self-exempts."""
posted, r = _run_classify(tmp_path, _emitting(".gitea/workflows/review-verdict.yml", *_many_docs()))
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending", (
"a PR editing .gitea/workflows/ was exempted — the protected-path guard was skipped on a "
f"large file list (desc={posted['description']!r})"
)
def test_a_LARGE_bot_pr_with_a_code_file_is_not_manifest_exempt(tmp_path):
"""The same inversion reached through the third guard added by this change."""
posted, r = _run_classify(tmp_path, _emitting("A.cs", *_many_docs()), author="renovate")
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending"
def test_positive_control_a_LARGE_genuinely_docs_only_pr_IS_still_exempt(tmp_path):
"""Without this the three tests above could pass because large lists now fail outright, which
would be a merge deadlock rather than a fix."""
posted, r = _run_classify(tmp_path, _emitting(*_many_docs()))
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "success", (
f"a large but genuinely docs-only PR lost its exemption (desc={posted['description']!r})"
)
def test_a_human_verdict_landing_MID_RUN_is_not_overwritten(tmp_path):
"""ersatztv#706, the half that IS mitigated here.
The first status read finds nothing, so the job proceeds to classify. A reviewer then posts a human
`failure` (BLOCKED). Without the re-read immediately before the POST, this docs-only PR would post
an exemption `success` OVER an explicit human rejection — turning a "no" into a merge, which is the
worst outcome in this class.
Note precisely what this does and does not prove: it pins the re-read, not atomicity. A verdict
landing between the re-read and the POST is still lost — there is no compare-and-set on Gitea's
status API. That remainder is #706, not a claim made here.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="appears-on-read:2")
assert r.returncode == 0, r.stderr
assert posted is None, (
"an exemption success was posted over a human BLOCKED verdict that landed while the job was "
"classifying — the pre-POST re-read is missing or ineffective"
)
def test_positive_control_no_late_verdict_still_posts_normally(tmp_path):
"""Without this, the test above would pass against a job that had simply stopped posting."""
posted, _ = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="none")
assert posted is not None and posted["state"] == "success"
# --- The PROTECTED branch must actually EXECUTE, not merely coincide with the right answer --------
#
# Round-3 review found `count_matching` being called before its definition, so it was
# `command not found` on every run and the PROTECTED branch never fired. Three "protected path" tests
# passed anyway, because a protected path is also not a manifest and not docs-only, so the job reached
# `pending` down a different route. Asserting the STATE could not see it; the guard was dead and the
# suite was green.
#
# The lesson generalises: when several branches produce the same outcome, asserting the outcome cannot
# tell you which branch ran. Assert the DISCRIMINATOR — here the reason string the branch writes.
def test_a_protected_path_is_rejected_BY_THE_PROTECTED_BRANCH(tmp_path):
posted, r = _run_classify(tmp_path, _emitting("scripts/evil.sh", "docs/a.md"))
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending"
# The DISCRIMINATOR is the job's `Decision:` line, not the status description: for `pending` the
# description is always "Awaiting review verdict for <sha>", identical no matter which branch
# produced it. A first draft of this test asserted on the description and failed against a WORKING
# guard — the assertion has to be aimed at something that actually differs per branch.
assert "protected" in r.stdout.lower(), (
"the PR was not exempted, but NOT via the protected-path branch — it reached the same verdict "
f"by another route, so that guard may be dead. Decision log:\n{r.stdout[-800:]}"
)
def test_a_protected_path_defeats_the_BOT_exemption_by_the_protected_branch(tmp_path):
"""A bot PR whose every path IS a manifest, plus one protected path. Without a live PROTECTED
branch this still lands on `pending` (the protected file is not a manifest), so again only the
reason string distinguishes a working guard from a dead one."""
posted, r = _run_classify(
tmp_path, _emitting("Directory.Packages.props", ".gitea/workflows/renovate.yml"), author="renovate"
)
assert posted is not None
assert posted["state"] == "pending"
assert "protected" in r.stdout.lower(), (
f"the bot exemption was refused, but not by the PROTECTED branch:\n{r.stdout[-800:]}"
)
@pytest.mark.parametrize(
"paths,author",
[
(("docs/a.md",), "timothy"),
(("Directory.Packages.props",), "renovate"),
(("ErsatzTV/Program.cs",), "timothy"),
(("scripts/evil.sh",), "timothy"),
],
)
def test_the_classify_step_runs_without_SHELL_ERRORS(tmp_path, paths, author):
"""A cheap, general trap-catcher for the whole step.
`command not found`, `integer expression expected`, `unbound variable` — each of these silently
skips a branch inside an `if`/`elif` (the condition just evaluates false) while the job exits 0 and
posts a plausible status. That is precisely how the dead PROTECTED guard survived a green suite.
Scope, stated so this is not mistaken for more than it is: it catches guards that die NOISILY on
stderr. It is NOT a general liveness check — a clean mutation such as hardcoding `n_protected=0`
emits none of these and passes here. That case is covered by the branch-discriminator test above,
which is the actual liveness guard; this one is the cheap net for the whole error-emitting family.
"""
_, r = _run_classify(tmp_path, _emitting(*paths), author=author)
bad = [
ln
for ln in r.stderr.splitlines()
if "command not found" in ln
or "integer expression expected" in ln
or "unbound variable" in ln
or "syntax error" in ln
]
assert not bad, f"the classification step emitted shell errors, so a guard is not running: {bad}"
def test_a_human_verdict_formed_against_ANOTHER_BASE_is_not_inherited(tmp_path):
"""Round-4 review: the sha-binding is escapable through the HUMAN verdict path.
Get a genuine `success` on head H while it targets scratch base S (benign diff there), then
retarget H onto `main`, where its diff contains unreviewed code. Creator is real, prefix is real,
so the short-circuit preserved it — a green required check over code nobody reviewed. The
merge-consent hook compares the base, but that is advisory; a merge through the Gitea UI or API
only sees the status.
`post-review-verdict.sh` already records the base it reviewed (`(base: …)`, ersatztv#632); this
asserts the gate actually READS it.
"""
posted, r = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: probe/scratch-base)",
)
assert posted is not None, (
"a human verdict formed against a DIFFERENT base was inherited unchanged — the reviewed diff "
f"is not this PR's diff. Log:\n{r.stdout[-600:]}"
)
assert posted["state"] == "pending"
def test_a_human_verdict_for_THIS_base_is_still_honoured(tmp_path):
"""The positive control the test above needs: matching bases must still short-circuit, or the
check has simply broken every verdict."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
)
assert posted is None, "a verdict formed against THIS base was re-derived; the base check is too strict"
def test_a_LEGACY_verdict_with_no_recorded_base_is_still_honoured(tmp_path):
"""Verdicts predating ersatztv#632 carry no `(base: …)`. Absent is deliberately not treated as a
mismatch: re-deriving over one would un-approve a genuinely reviewed head. Only a base that is
PRESENT and DIFFERENT is rejected."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23",
)
assert posted is None
def test_an_APPENDED_base_cannot_override_the_real_one(tmp_path):
"""The description is attacker-influencable by anyone who can POST a status — the registry
credential's own route closed with #697, but `GITEA_TOKEN`, `RENOVATE_TOKEN`, and a
collaborator's own token still can — so the parse must not be trickable into reading a second,
appended base.
This defeated the greedy `##` parse. The implementation no longer parses at all — it requires the
description to END with the exact literal marker AND to contain exactly one marker — so a second
appended marker makes the count 2 and is rejected outright.
"""
posted, r = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: probe/scratch) (base: main)",
)
assert posted is not None, (
"an appended '(base: main)' overrode the real recorded base, so a verdict formed elsewhere was "
f"inherited. Log:\n{r.stdout[-600:]}"
)
assert posted["state"] == "pending"
def test_an_EMPTY_recorded_base_is_treated_as_a_mismatch(tmp_path):
"""`(base: )` is not 'absent' — it is present and not equal to the PR's base, so it fails closed
rather than being waved through by the legacy-verdict allowance."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: )",
)
assert posted is not None and posted["state"] == "pending"
def test_a_branch_name_containing_a_PAREN_cannot_truncate_into_the_current_base(tmp_path):
"""Round-5 review, and the sharpest finding of the five: it needs no forgery and no #697.
`main)evil` is a VALID git branch name (`git check-ref-format --branch 'main)evil'` succeeds). A
genuine verdict earned while head H targeted it is written `(base: main)evil)`. Any implementation
that extracts the value by truncating at the first `)` gets exactly `main`, matches a PR that now
targets `main`, and inherits a verdict covering a completely different diff.
An earlier comment in the workflow asserted that a `)` in a branch name "mismatches — safe
direction". That was generalised from `feat/foo)bar` (which does mismatch) and is false for every
branch whose name STARTS with the target base. Hence the rule the code now follows: compare against
the exact expected literal, never parse a value out of attacker- or user-influenced text.
"""
posted, r = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)evil)",
)
assert posted is not None, (
"a verdict recorded against branch 'main)evil' was inherited by a PR targeting 'main' — the "
f"base value is being truncated at the first ')'. Log:\n{r.stdout[-600:]}"
)
assert posted["state"] == "pending"
# --- ersatztv#706: the retarget fence (race 1) and the post-write repair (race 2) -----------------
#
# Race 1 was reproduced LIVE before any of this was written, because the fix turns on which Gitea
# behaviours are real rather than on what the docs imply. Measured on Gitea 1.25.4:
# * `pull_request_target` runs for one PR are NOT auto-cancelled — probe PR #722, run 7520
# (`opened`) ran to completion 20s AFTER run 7521 (`synchronize`) started. The older run finished
# LAST, which is exactly the mechanism by which a stale classification overwrites a fresh one.
# * A NON-CANCELLING concurrency group does not serialize them either: with the group active, runs
# 7528 and 7529 still overlapped and 7528 ended 36s after 7529 began. #706's headline proposal is
# therefore refuted, not merely unattractive.
# * Gitea DOES auto-cancel superseded `push` runs on a branch — a control workflow carrying no
# `concurrency:` key at all showed that — which is why the concurrency-group probe was confounded
# until the control separated the two.
# The fence keys on the timeline's `change_target_branch` COUNT because the branch NAME is
# ABA-vulnerable (`main → S → main` reads `main` at both ends), while the count is monotonic.
def _posted_sequence(tmp_path):
"""Every status POST the job made, in order — not just the last one."""
f = tmp_path / "posted_all.jsonl"
return [json.loads(line) for line in f.read_text().splitlines() if line.strip()] if f.exists() else []
def test_a_RETARGET_DURING_the_run_posts_NOTHING(tmp_path):
"""THE race-1 test. A docs-only PR that would otherwise be exempted is retargeted while the job
classifies, so this run's answer describes a base the PR may no longer target. It must write
nothing at all and leave the field to the successor run the retarget's `edited` event queues."""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="moves:0,1")
assert r.returncode == 0, r.stderr
assert posted is None, (
"a stale run posted its classification after the PR was retargeted underneath it — the "
f"retarget fence did not fire. Log:\n{r.stdout[-900:]}"
)
# Assert the DISCRIMINATOR, not just the outcome: several unrelated failures also end in "posted
# nothing", so the state alone cannot tell a working fence from a broken job.
assert "retargeted while this job was classifying" in r.stdout, (
f"nothing was posted, but NOT via the retarget fence. Log:\n{r.stdout[-900:]}"
)
def test_positive_control_a_QUIET_run_still_posts_its_exemption(tmp_path):
"""Without this, the test above passes against a job that simply never posts."""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none")
assert posted is not None and posted["state"] == "success", (
f"an undisturbed docs-only PR lost its exemption. Log:\n{r.stdout[-900:]}"
)
def test_a_PR_retargeted_BEFORE_the_run_but_QUIET_during_it_is_STILL_exempt(tmp_path):
"""The fence keys on MOTION during this run, never on "has ever been retargeted".
A PR legitimately retargeted once, long before this run, carries a permanently non-zero event
count. Keying on the count being non-zero — rather than on it CHANGING — would deadlock that PR's
exemption forever, which is a worse failure than the race being fixed.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="stable:2")
assert posted is not None and posted["state"] == "success", (
"a PR with a settled, non-zero retarget history was refused its exemption — the fence is "
f"testing the count's VALUE instead of its MOVEMENT. Log:\n{r.stdout[-900:]}"
)
@pytest.mark.parametrize("mode", ["unreadable", "transport-error"])
def test_an_UNTRUSTED_retarget_count_withholds_the_EXEMPTION(tmp_path, mode):
"""A `success` that cannot be shown to describe the PR's current base must not be written. An
absent required check blocks the merge, which is the safe direction."""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode=mode)
assert posted is None, (
"an exemption success was posted even though the retarget count could not be established "
f"({mode}). Log:\n{r.stdout[-900:]}"
)
@pytest.mark.parametrize("mode", ["unreadable", "transport-error"])
def test_an_UNTRUSTED_retarget_count_STILL_LETS_PENDING_THROUGH(tmp_path, mode):
"""The other half, and the one that keeps a timeline outage from taking every PR down with it.
`pending` cannot turn an unreviewed head green — it is what blocks the merge — so withholding it
buys no safety and would strand ordinary PRs with no status at all. Only the exemption is gated.
"""
posted, r = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), timeline_mode=mode)
assert posted is not None and posted["state"] == "pending", (
"an ordinary PR was left with NO status because the timeline was unreadable; only the "
f"exemption success should be gated on the count. Log:\n{r.stdout[-900:]}"
)
def test_a_human_verdict_landing_AFTER_the_POST_is_repaired_to_pending(tmp_path):
"""Race 2's remainder. The pre-POST re-read cannot see a verdict that lands between it and the
write, and there is no compare-and-set to make the two one operation. So the write is verified
AFTERWARDS and repaired in the safe direction: silently greening an explicit human BLOCKED is the
worst outcome this gate can produce, and strictly worse than a stall.
The repair is `pending`, never a copy of the human's `failure` — re-posting their state under the
machine credential would attribute a human verdict to this job.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="human-after-post")
assert r.returncode == 0, r.stderr
seq = _posted_sequence(tmp_path)
assert len(seq) == 2, (
"expected the exemption write followed by a repair; the post-write verification did not "
f"fire. Posts: {seq}. Log:\n{r.stdout[-900:]}"
)
assert seq[0]["state"] == "success"
assert seq[1]["state"] == "pending", f"the raced exemption was not repaired: {seq}"
assert posted is not None and posted["state"] == "pending"
def test_a_PRE_EXISTING_human_row_does_NOT_trigger_a_repair(tmp_path):
"""The false-positive guard, and the reason the check compares ids instead of asking "does a human
verdict exist".
`read_existing_verdict` deliberately declines to honour a human verdict whose recorded base does
not match this PR's — so such a row sits in the history forever. A presence test would fire on it
on EVERY later run, repair each exemption to `pending`, and permanently deadlock a PR that had one
mismatched verdict once. Only rows newer than the high-water mark count.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="stale-human-already-present")
seq = _posted_sequence(tmp_path)
assert len(seq) == 1, (
"an old, already-present human verdict row was mistaken for one that raced this run's write, "
f"so the exemption was repaired away. Posts: {seq}. Log:\n{r.stdout[-900:]}"
)
assert posted is not None and posted["state"] == "success"
# --- ersatztv#711: the PROTECTED list must cover every mirror of the enforcement hooks -----------
def test_a_CODEX_hook_copy_is_a_PROTECTED_path(tmp_path):
"""`.codex/hooks/` is a byte-identical mirror of `.claude/hooks/`, including
`pretooluse-merge-consent.sh`. Editing the `.claude/` copy correctly voids both exemptions;
before this, editing its `.codex/` twin did not — the rule "a PR that can weaken the gate must
not exempt itself from the gate" was written as a path list and the list had gone incomplete.
Latent rather than live today (`.codex/` is untracked and gitignored, and a PR cannot touch a
path that is not in the repo), which is exactly why it needs a test: it becomes live silently,
the moment anyone tracks the directory.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md", ".codex/hooks/pretooluse-merge-consent.sh"))
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending", (
"a PR editing a .codex/ copy of the merge-consent hook was granted the docs-only exemption "
f"(desc={posted['description']!r})"
)
# Assert the DISCRIMINATOR: a non-exempt outcome is reachable by several routes, so the state
# alone cannot show the PROTECTED branch is what rejected it.
assert "protected" in r.stdout.lower(), (
f"rejected, but NOT by the protected-path branch. Decision log:\n{r.stdout[-800:]}"
)
def test_BOTH_hook_directories_are_in_the_PROTECTED_pattern():
"""A structural pin over the list itself. The behavioural test above proves `.codex/` is covered
today; this one states the INVARIANT — the two hook directories are mirrors, so any future edit
that drops one while keeping the other is the bug, not a simplification."""
src = _code_lines(WORKFLOW)
protected = [line for line in src.splitlines() if line.strip().startswith("PROTECTED=")]
assert len(protected) == 1, f"expected exactly one PROTECTED definition, found {protected}"
for d in (r"\.claude/", r"\.codex/"):
assert d in protected[0], (
f"{d} is missing from PROTECTED ({protected[0].strip()!r}) — a directory carrying a copy "
"of the enforcement hooks can exempt itself from the gate it enforces"
)
# --- ersatztv#706, round 2: findings from the cold adversarial review ----------------------------
def test_the_high_water_MARK_is_captured_BEFORE_the_last_moment_re_read():
"""The High finding of round 2, pinned as the ORDERING property it actually is.
The mark was originally taken "as late as possible", just before the POST. That leaves everything
between the last-moment re-read and the mark as a blind gap: a human verdict landing there is
invisible to the re-read (already done) and excluded from the post-write check (its id is BELOW a
mark taken afterwards), so it is overwritten with no repair. The gap spanned the whole retarget
re-count — up to 20 timeline round-trips — not the single round-trip that was being claimed.
Taking the mark FIRST closes the read side: any row newer than the mark is caught either by the
re-read (abstain) or by the post-write check (repair). This is asserted structurally because the
defect is an order, not an output — with the mark late the job still posts and still repairs in
every scenario a stub can pose; only the id arithmetic silently changes.
"""
src = _classify_step()["run"]
# Assert on BOTH the init and the FETCH: keying only on the init line would let a refactor that
# splits initialisation from the read slide the actual round-trip back past the re-read while this
# test stayed green (round-3 review nit).
mark = max(src.index("max_id_before=-1"), src.index('hist_before=$(gh "$BASE_URL/repos/$REPO/statuses/$SHA'))
# The LAST-MOMENT re-read is the second bare `read_existing_verdict` call.
calls = [i for i in range(len(src)) if src.startswith("read_existing_verdict\n", i)]
assert len(calls) >= 2, f"expected two read_existing_verdict call sites, found {len(calls)}"
assert mark < calls[-1], (
"the high-water mark is captured AFTER the last-moment re-read, reopening the blind window "
"in which a human verdict is neither seen by the re-read nor repaired by the post-write check"
)
def test_a_previously_REPAIRED_head_is_never_re_exempted(tmp_path):
"""The Medium finding of round 2: the repair used to last exactly one event.
After a repair, the status is a machine-written `pending` — indistinguishable, to the next run,
from an ordinary one. That run re-derived it, posted `success`, and took a fresh high-water mark
ABOVE the human row, so the post-write check stayed silent and the human's rejection went green
again one event later. The repair description is now a sentinel the classification recognises.
"""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc="Human verdict raced this exemption write — re-post the verdict",
)
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
assert posted["state"] == "pending", (
"a head whose human verdict was previously overwritten was granted a FRESH exemption, burying "
f"the rejection again (desc={posted['description']!r})"
)
assert "raced a previous exemption" in r.stdout, (
f"rejected, but not via the repair-sentinel branch. Log:\n{r.stdout[-800:]}"
)
def test_positive_control_an_ORDINARY_machine_pending_is_still_re_derived(tmp_path):
"""Without this, the test above would pass against a job that had stopped exempting anything with
a pre-existing `pending`. An ordinary machine `pending` — no sentinel — must still re-derive to
`success` for a docs-only PR."""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc="Awaiting review verdict for a9e3e23",
)
assert posted is not None and posted["state"] == "success", (
f"an ordinary pending was not re-derived to an exemption. Log:\n{r.stdout[-800:]}"
)
def test_the_fence_gates_PENDING_TOO_not_only_the_exemption(tmp_path):
"""The fence deliberately refuses to write `pending` as well as `success` when the PR was
retargeted mid-run. Untested, a success-only-fence mutant would have passed the whole suite.
A stale `pending` is only a stall rather than a forged green, so gating it is not strictly
required — but the successor run is guaranteed either way, so there is nothing to buy by writing a
value computed against a base the PR may no longer target.
"""
posted, r = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), timeline_mode="moves:0,1")
assert posted is None, (
"the fence let a stale `pending` through; it is gating only the exemption path "
f"(desc={posted['description']!r})"
if posted
else ""
)
assert "retargeted while this job was classifying" in r.stdout
def test_an_UNTRUSTED_count_withholds_the_exemption_BY_THAT_BRANCH(tmp_path):
"""Round 2 test-gap: the existing untrusted-count test asserted only "posted nothing", which a
crash also produces. Assert the discriminator and a clean exit."""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="unreadable")
assert posted is None
assert r.returncode == 0, f"the job died rather than declining cleanly: {r.stderr[-800:]}"
assert "Could not establish a trusted retarget count" in r.stdout, (
f"nothing posted, but not via the untrusted-count branch. Log:\n{r.stdout[-800:]}"
)
def test_the_repair_sentinel_is_a_FIXED_POINT_across_consecutive_runs(tmp_path):
"""Round-3 finding: durability is a fixed point, and only a CHAIN can assert a fixed point.
The first version of the sentinel refused the exemption but posted the GENERIC pending
description, erasing the marker it depends on. The next run then saw an ordinary machine
`pending`, re-derived it, and posted `success` — burying the human rejection two events after the
repair instead of one. The single-hop test passed throughout, and the positive control asserting
that an ordinary machine `pending` DOES re-derive was itself the proof of the second hop.
So chain two runs: feed run N's posted description in as run N+1's existing status. The property
is that the sentinel branch's own output re-triggers the sentinel branch, forever.
"""
first, r1 = _run_classify(
tmp_path / "run1",
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc="Human verdict raced this exemption write — re-post the verdict",
)
assert first is not None and first["state"] == "pending", f"run 1: {r1.stdout[-600:]}"
second, r2 = _run_classify(
tmp_path / "run2",
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc=first["description"],
) # <-- the chain
assert second is not None, f"run 2 posted nothing: {r2.stderr[-800:]}"
assert second["state"] == "pending", (
"the repair decayed: run 1's own posted description did not re-trigger the sentinel, so run 2 "
f"re-derived the exemption and buried the human verdict again (run1 desc={first['description']!r}, "
f"run2 state={second['state']}, run2 desc={second['description']!r})"
)
assert second["description"] == first["description"], (
"the sentinel is not a fixed point — run 2 wrote a different description than run 1, so run 3 "
f"would not recognise it ({first['description']!r} -> {second['description']!r})"
)
def test_a_sentinel_APPEARING_MID_RUN_stops_a_stale_run_overwriting_it(tmp_path):
"""Round-3 finding, and the only one in this series that failed toward SUCCESS.
Two runs overlap for the same sha — the regime this branch measured live (probe PR #722: the older
run finished 20s after the newer one started). Run B catches a raced human BLOCKED and repairs to
the sentinel. Run A is still in flight: its FIRST read predates all of it, so it classified
`success`; its high-water mark was taken after the human row, so the post-write check stays silent;
and the retarget fence sees nothing. Until this guard, A posted its stale `success` straight over
the sentinel — burying the human rejection with no repair and no log.
The re-read recomputes `ex_repair`; the bug was that nothing downstream consulted it.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="sentinel-appears-on-read:2")
assert r.returncode == 0, r.stderr
assert posted is None, (
"a stale run posted its exemption over a repair sentinel written mid-run, burying the human "
f"verdict it records (posted={posted})"
)
assert "repair sentinel was written" in r.stdout, (
f"nothing posted, but not via the mid-run sentinel guard. Log:\n{r.stdout[-800:]}"
)
def test_positive_control_a_sentinel_present_from_the_START_still_posts_pending(tmp_path):
"""Without this, the guard above could be an unconditional abstain whenever a sentinel exists.
A sentinel present at the FIRST read is the ordinary repaired-head case: it must still POST
`pending` carrying the sentinel forward (the fixed point), not abstain. That difference is exactly
what makes the mid-run guard's condition exact rather than conservative.
"""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:pending",
status_creator=None,
status_desc="Human verdict raced this exemption write — re-post the verdict",
)
assert posted is not None, f"the repaired head stopped posting entirely: {r.stdout[-800:]}"
assert posted["state"] == "pending"
assert posted["description"] == "Human verdict raced this exemption write — re-post the verdict"
def test_a_PENDING_path_run_also_refuses_to_clobber_a_mid_run_sentinel(tmp_path):
"""Round-4 finding: the guard's first form tested `state = success`, one branch too narrow.
A run can reach the POST on `pending` carrying the GENERIC description — most realistically after a
transient enumeration failure — and a success-only guard waves it through. It then overwrites the
sentinel with ordinary text, the next run sees no sentinel, re-derives, and posts `success`: the
same buried human rejection, two steps instead of one.
Hence the rule is "never replace a sentinel with a non-sentinel", compared on the DESCRIPTION. That
is strictly more general and exactly as precise, because the carry-forward branch guarantees a
first-read sentinel already sets `desc` to the sentinel.
"""
posted, r = _run_classify(
tmp_path,
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n", # enumeration fails -> pending, generic desc
status_mode="sentinel-appears-on-read:2",
)
assert r.returncode == 0, r.stderr
assert posted is None, (
"a pending-path run overwrote a repair sentinel written mid-run with the generic description; "
f"the next run would re-derive the exemption and bury the human verdict (posted={posted})"
)
assert "repair sentinel was written" in r.stdout, (
f"nothing posted, but not via the mid-run sentinel guard. Log:\n{r.stdout[-800:]}"
)
def test_a_SENTINEL_landing_above_the_mark_also_triggers_the_repair(tmp_path):
"""Round-5 finding: the post-write filter counted only HUMAN rows, and that is not sufficient.
With two overlapping runs A and B, the human BLOCKED can land BELOW A's high-water mark — so A
cannot see it — while B masks it with an exemption `success` and only afterwards writes the
sentinel. A then finds nothing human above its mark, does not repair, and posts its own `success`
on top of the sentinel: a PERMANENT forged green over a human rejection, and the repair race
failing toward success rather than pending, which the decision record explicitly promises it does
not do.
A sentinel above the mark can only have been written by another run mid-flight — a pre-existing one
would have been seen at the first read and forced the pending path — so counting it cannot
false-fire, and it converges both runs on the fixed point.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="sentinel-after-post")
assert r.returncode == 0, r.stderr
seq = _posted_sequence(tmp_path)
assert len(seq) == 2, (
"the run posted its exemption on top of a sentinel written by an overlapping run and did not "
f"repair, leaving a human rejection permanently green. Posts: {seq}\n{r.stdout[-800:]}"
)
assert seq[0]["state"] == "success"
assert seq[1]["state"] == "pending"
assert seq[1]["description"] == "Human verdict raced this exemption write — re-post the verdict", (
f"repaired, but not to the sentinel, so the fixed point is broken: {seq[1]}"
)
# --- The expression-delimiter class (ersatztv#751) ----------------------------------------------
#
# WHAT HAPPENED, because the shape of it is what these tests are written against. A `run:` body is
# not shell yet when the runner reads it. The runner scans the whole scalar for the expression
# opener, and a single occurrence makes it rewrite the ENTIRE body into one `format(...)` call so
# the evaluated result can be spliced back in. That rewrite is all-or-nothing, so a payload that
# does not parse takes the whole step with it — and the runner then DROPS THE STEP AND CONCLUDES THE
# JOB `success`.
#
# The occurrence that did it was in a SHELL COMMENT: the #706 note explaining why a concurrency
# group does not work quoted a `concurrency:` snippet containing a PR-number expression as an
# illustration. `pr number` is not an expression. The step stopped running on 2026-08-03 and nothing
# went red until 2026-08-06.
#
# WHY EVERY EXISTING GUARD IN THIS FILE WAS BLIND TO IT, which is the part worth keeping: they all
# read `_code_lines()`, which strips comment lines. That choice is correct for what it was for — its
# own docstring explains that prose legitimately discusses `pulls/N/files`, and a raw scan would
# redden the repo over a piece of writing. But it encodes an assumption this bug falsifies: that a
# comment in a workflow cannot change behaviour. Inside a `run:` scalar it can. So the test below
# reads the RAW text, deliberately, and is the one test here that must never adopt `_code_lines`.
_EXPR = re.compile(r"\$\{\{(.*?)\}\}", re.S)
# Roots of a dotted context path, and the callable functions. Both lists are what the runner
# actually accepts; anything outside them cannot evaluate, and an expression that cannot evaluate
# does not fail loudly — it silently removes the step it appears in.
_EXPR_CONTEXTS = frozenset(
{
"github",
"env",
"vars",
"secrets",
"inputs",
"runner",
"steps",
"needs",
"matrix",
"job",
"jobs",
"strategy",
}
)
_EXPR_FUNCTIONS = frozenset(
{
"always",
"success",
"failure",
"cancelled",
"hashFiles",
"format",
"toJSON",
"toJson",
"fromJSON",
"fromJson",
"contains",
"startsWith",
"endsWith",
"join",
}
)
_EXPR_LITERALS = frozenset({"true", "false", "null"})
def _yaml_string_scalars(path: Path):
"""Every string scalar in the parsed document — keys and values, recursively.
Deliberately PARSED rather than raw (ersatztv#751, cold re-review). A `${{ … }}` in an ordinary
top-level YAML comment is inert: the runner never evaluates it, so redding on it would be a false
positive of exactly the kind this file has now produced twice. PyYAML drops those comments, which
is the behaviour wanted here.
A `run:` body IS one of these scalars, and it keeps its SHELL comments — which is the whole point,
because inside a `run:` scalar a comment is not inert. So this covers the real defect class in
every workflow while ignoring the one place a delimiter is genuinely harmless.
"""
import yaml
out: list[str] = []
def walk(node):
if isinstance(node, str):
out.append(node)
elif isinstance(node, dict):
for k, v in node.items():
if isinstance(k, str):
out.append(k)
walk(v)
elif isinstance(node, list):
for v in node:
walk(v)
walk(yaml.safe_load(path.read_text()))
return out
def _workflow_files():
# `*.yml` AND `*.yaml`: a workflow added as `.yaml` is just as executable and would otherwise go
# unscanned. From the GIT INDEX rather than a directory listing (ersatztv#806) — a stray
# untracked `.yaml` in `.gitea/workflows/` is not part of the repo, and enumerating it made
# these two completeness claims red on that checkout and green in CI, which is #778's shape.
return tracked_paths(".gitea/workflows", ("*.yml", "*.yaml"))
def test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body():
"""The absolute rule, and it is scoped to this one file on purpose.
Other workflows legitimately interpolate into a `run:` body (5 occurrences today across
ci-image, docker-build's `api-docs`/`format`, and pr-checks' two git-diff gates), so a
repo-wide ban would be false and would be deleted the first time it got in someone's way.
`test`, `migrations` and `build` carry their own absolute ban — see
scripts/tests/test_ci_dropped_step_guard.py (ersatztv#756). This file is different in two ways that justify the
strict rule: it writes the branch-protection-required status, so a dropped step here is a dead
merge gate rather than a failed build; and its `run:` bodies are ~700 lines of dense prose,
which is the only place the delimiter has ever appeared by accident.
The rule is also what keeps the dropped-step guard trustworthy. A guard that the guarded
mechanism can silently delete is worse than no guard, because its absence is silent too.
The `run:` SCALAR AS PARSED, comments and all — never `_code_lines`, which strips them. The
distinction matters and is easy to garble: what must not be filtered is the SHELL comments inside
the body, because those are what the runner scans. Ordinary YAML comments outside a `run:` body are
genuinely inert and are handled by the repo-wide test below, which is why neither test reads the
file as flat text any more.
"""
offenders = []
for job_name, step in _iter_workflow_steps(WORKFLOW):
for m in _EXPR.finditer(step.get("run") or ""):
offenders.append(f"{job_name}/{step.get('name', '?')}: {m.group(1).strip()!r}")
assert not offenders, (
f"review-verdict.yml has an expression delimiter inside a run: body — {offenders}. Even in "
"a comment this is unsafe: the runner rewrites the WHOLE body into a format(...) call, and "
"if the payload does not parse it drops the step and reports the job GREEN, leaving the "
"required review-verdict/h10 unposted (ersatztv#751). To describe an expression in prose, "
"name it (`a github.event.pull_request.number expression`) instead of quoting the "
"delimiters. Pass values in through the step's `env:` block, which is interpolated per "
"value, so a bad payload there cannot take the body with it."
)
# ANTI-VACUITY, and it has to be the right property. A `run:` body the YAML walk never reached
# would make the assertion above vacuously green — the failure mode to guard against. An earlier
# version of this guard compared the file's TOTAL delimiter count against the count inside
# `with:`/`env:` values, which is a different and wrong claim: it banned expressions everywhere
# else in the file too. Both reviewers reproduced the false red — writing `if: ${{ always() }}`,
# the standard and equivalent spelling of the `if:` two steps below, turned this test red, as did
# a delimiter in an inert top-level YAML comment. Neither is unsafe, and a red here blocks every
# merge through the combined status, so the guard was strictly more dangerous than the thing it
# was protecting against.
#
# ANTI-VACUITY WITHOUT A SECOND PARSER. A first attempt counted `run:` keys in the raw text and
# compared that to the walk. Cold re-review showed the regex only recognised an indented `run:`
# whose value starts with `|` or `>`, so legal spellings (`- run: |`, a single-line
# `run: echo ok`) counted as zero declarations and false-redded the file, while a `run: |` line
# sitting INSIDE a shell heredoc counted as a declaration. Hand-parsing YAML to check a YAML parse
# is the wrong shape: it adds a second, worse parser whose disagreements are all false alarms, and
# a red here blocks every merge through the combined status.
#
# The property that actually matters is that the walk reached the body that carries the risk. The
# classifier is ~700 lines; a walk that returned nothing, or only the short steps, is the failure
# to catch. Both are asserted on content, which no spelling change can spoof.
bodies = [s["run"] for _, s in _iter_workflow_steps(WORKFLOW) if s.get("run")]
# NOT `>= 3`, and not merely non-empty either. `>= 3` had zero slack — cold review deleted the
# optional jq-preflight step, a legitimate simplification, and this redded claiming the classifier
# had not been examined, which was untrue. But relaxing it to `assert bodies` threw away the only
# check that the walk reached ALL run-bearing steps: `max(len) > 5000` proves it reached the
# classifier and nothing about the short ones, so a helper that silently stopped yielding them
# would let an unscanned delimiter through.
#
# So count against the job's own step list, read here rather than through the helper under test.
# That catches the failure the count existed for (a helper looking at the wrong key, or dropping
# steps) without breaking when a step is legitimately added or removed.
import yaml as _yaml
_steps = _yaml.safe_load(WORKFLOW.read_text())["jobs"]["set-verdict-status"]["steps"] or []
declared = sum(1 for s in _steps if isinstance(s, dict) and s.get("run"))
assert len(bodies) == declared, (
f"the YAML walk reached {len(bodies)} run: bodies but the job declares {declared} — the "
"assertion above did not examine every body, so a green here proves nothing"
)
assert max(len(b) for b in bodies) > 5000, (
"the YAML walk did not reach a substantial run: body — the ~700-line classifier is the one "
"that must be scanned, so this test would be vacuous"
)
def _iter_workflow_steps(path: Path):
import yaml
doc = yaml.safe_load(path.read_text()) or {}
for job_name, job in (doc.get("jobs") or {}).items():
for step in job.get("steps") or []:
yield job_name, step
def test_every_workflow_expression_names_a_REAL_context_or_function():
"""The general form of #751, across every workflow and every field.
The strict test above hardens the gate file. It cannot see the class, which is not "prose in a
run body" but "a payload that does not evaluate" — a typo in an `if:`, a renamed output, a
context that does not exist. All of them fail the same silent way, and in an `if:` the
consequence is the same shape as #751: the step does not run and nothing is red.
BE PRECISE ABOUT WHAT THIS ENFORCES, because the first version of this docstring was not and
both reviewers caught it: it checks that THE HEAD TOKEN of each dotted path is a known context or
function. Nothing more. That catches the historical defect — `pr number` fails on `pr` — and a
payload naming a context that does not exist. It does NOT catch:
* syntactically invalid expressions whose tokens are all known: `${{ github.ref == }}` and
`${{ github.event.pull_request.head.sha + }}` both pass, verified;
* a renamed or misspelled output or property, because every token after the first is preceded
by `.` and is deliberately skipped: `steps.metadata.outputs.shortsha` passes;
* an unclosed opener, since `_EXPR` requires the closing braces to match at all.
A real fix for those is an expression parser, which is a different and much larger change. This is
a cheap net under the specific class that has bitten us, deliberately kept permissive so it cannot
false-red the repo (a red here blocks every merge through the combined status). Verified against
all 31 distinct payloads in this repo today, which pass. Do not restate this test as "catches any
payload that cannot evaluate" — that claim was in the docs and the decision record and was false.
"""
offenders = []
for wf in _workflow_files():
for scalar in _yaml_string_scalars(wf):
for m in _EXPR.finditer(scalar):
payload = m.group(1).strip()
# Strip string literals first: a path inside `hashFiles('web/package-lock.json')` is
# data, not an identifier, and would otherwise read as an unknown context.
bare = re.sub(r"'[^']*'", "''", payload)
for ident in re.finditer(r"(?<![.\w'])([A-Za-z_][A-Za-z0-9_-]*)", bare):
name = ident.group(1)
if name in _EXPR_CONTEXTS or name in _EXPR_FUNCTIONS or name in _EXPR_LITERALS:
continue
offenders.append(f"{wf.name}: {payload!r} -> unknown '{name}'")
assert not offenders, (
"these workflow expressions name something the runner cannot resolve, so they will fail to "
f"interpolate — which DROPS THE STEP and still reports the job green (ersatztv#751): "
f"{offenders}. If this is prose describing an expression, do not write the delimiters; if "
"it is a real new context or function, add it to _EXPR_CONTEXTS / _EXPR_FUNCTIONS here."
)
def test_a_dropped_classify_step_FAILS_the_job_instead_of_going_green():
"""The silent-green half of #751 — the actual defect, and the only part that generalises.
The stray delimiter was one bug in one comment. What made it cost three days of a dead merge
gate was the REPORTING: a dropped step concludes `success`, the workflow's own status context
goes green, and the required `review-verdict/h10` is merely ABSENT — which on a normal PR is
indistinguishable from the correct "not reviewed yet" state. Nothing surfaced it. So the
classifier writes a start marker and a later `if: always()` step fails the job when it is
missing.
Asserted structurally, and the marker PATH is compared across the two steps rather than
hardcoded here twice: the failure mode of a divergent path is a job that reddens on every run
(fail-closed, but it would look like this guard is broken rather than like a real drop), and a
test carrying its own third copy of the literal could not see the divergence at all.
The premise — that the runner still executes a LATER step after dropping an earlier one — is
MEASURED, not assumed, and could not be settled from the #751 report because the classifier was
the job's last step. Gitea 1.27.1, 2026-08-06: probe run 1863 dropped the classifier on a
reintroduced bad payload, ran the guard anyway (`always()` evaluated true), and the job concluded
`failure`; run 1866 is the positive control. So the guard is load-bearing in production and the
body does not need to move into `scripts/`. This test pins the SHAPE; the probe pinned the
BEHAVIOUR, and neither substitutes for the other — which is why the behavioural test below
EXECUTES the guard body rather than only reading it.
"""
# ONE parse, and both steps located inside it. Using `_classify_step()` here instead was wrong
# in a way worth recording, because it went green-adjacent rather than loud: that helper does its
# own `yaml.safe_load`, so the dict it returns is never the same OBJECT as the equivalent step in
# this list. An `is not` filter against it therefore excluded nothing, the classify step matched
# as its own guard, and the assertions below ran against the wrong step.
steps = [s for _, s in _iter_workflow_steps(WORKFLOW)]
classifiers = [i for i, s in enumerate(steps) if "review-verdict/h10" in (s.get("run") or "")]
assert classifiers, "no step in review-verdict.yml posts review-verdict/h10"
classify = steps[classifiers[0]]
# The FULL path expansion is compared between the two steps, not a basename. The path is keyed on
# the run id, so a basename match would accept two steps that agree on the prefix and disagree on
# the key — which is precisely the divergence that would make the guard fail on every run.
marker_write = re.search(r'(\w+)="(\$\{RUNNER_TEMP[^"]*)"', classify["run"])
assert marker_write, (
"the classify step no longer assigns a start-marker path under RUNNER_TEMP, so a step the "
"runner drops goes green again (ersatztv#751)"
)
var, marker_name = marker_write.group(1), marker_write.group(2)
assert re.search(rf':\s*>\s*"\${var}"', classify["run"]), (
f"the classify step defines {var} but never creates the marker, so the guard below will "
"fail on every run and read as broken rather than as a real dropped step"
)
guard_idx = [i for i, s in enumerate(steps) if i != classifiers[0] and marker_name in (s.get("run") or "")]
assert guard_idx, (
f"no step checks for the {marker_name!r} start marker. Without it a dropped classify step "
"concludes success and the merge gate is silently dead (ersatztv#751)"
)
# AFTER the classifier, not merely present. A guard placed before it would read a marker that
# has not been written yet and fail on every run — fail-closed, but it would deadlock `main` and
# read as this guard being broken, which is how a correct-looking guard gets deleted.
assert guard_idx[0] > classifiers[0], (
f"the dropped-step guard is step {guard_idx[0]} but the classifier is step "
f"{classifiers[0]} — a guard that runs first always fails"
)
guard = steps[guard_idx[0]]
# `always()` and `${{ always() }}` are the same condition; the runner accepts both and the second
# is the more common spelling. Pinning the bare form EXACTLY would red the repo over a
# semantically identical edit, so normalise instead. What must not change is that the guard runs
# when the classifier failed.
guard_if = re.sub(r"\s+", "", str(guard.get("if", "")))
assert guard_if in ("always()", "${{always()}}"), (
f"the dropped-step guard's `if:` is {guard.get('if')!r}; it must be `always()` (bare or "
"wrapped), or it will be skipped on exactly the runs where the classifier failed"
)
# INSIDE the missing-marker branch, not merely somewhere in the body. Cold review pointed out
# that a bare `exit 1` substring is satisfied by an unreachable `if false; then exit 1; fi` while
# the real branch says `exit 0` — the test passes and a dropped classifier goes green again. The
# behavioural test below is the real proof; this keeps the structural one from being satisfiable
# by dead code.
missing_branch = re.search(r'if \[ ! -f "\$marker" \]; then(.*?)\bfi\b', guard["run"], re.S)
assert missing_branch, (
'the dropped-step guard no longer tests for a MISSING marker with `if [ ! -f "$marker" ]`, '
"so the assertion below cannot locate the branch that must fail the job"
)
assert re.search(r"exit\s+1", missing_branch.group(1)), (
"the dropped-step guard detects the missing marker but does not `exit 1` inside that branch, "
"so it observes the failure and still lets the job go green — which is the whole defect"
)
assert not _EXPR.search(guard["run"]), (
"the dropped-step guard's own run body contains an expression delimiter, so the mechanism "
"it guards against can drop the guard too — and that absence would be silent as well"
)
@pytest.mark.parametrize("terminator", ["null", "[]"], ids=["null-page", "empty-array-page"])
def test_the_fence_TRUSTS_the_count_and_POSTS_when_the_timeline_terminates(tmp_path, terminator):
"""The second defect found by the ersatztv#751 probe, and the one that actually kept the gate
from posting anything.
A page past the end of `/issues/{n}/timeline` is the JSON value `null` on this instance, not `[]`.
`count_retargets` gated on `type == "array"` and so treated the real terminator as unreadable: the
walk never reached a validated empty page, `rt_ok` was never `yes` for ANY pull request, and the
fence therefore withheld every exemption `success`. Renovate and docs-only PRs got NO status —
the same user-visible outcome as #751, by an unrelated route.
It hid for two reasons worth keeping written down. It shipped in the same commit (8f6d4f443) that
stopped the step from executing, so the fence had never once run in production; and the test
double printed `[]` while its comment claimed to mirror measured reality, so the type gate was
never exercised by the suite either. A green suite over an unfaithful double is what let a
fail-closed-by-accident branch look deliberate.
Both shapes are asserted because both are live on this server: comments really do return `[]`.
Asserting the POSTED STATUS, not the log, because the log said `Decision: state=success` on the
real probe run and the job still posted nothing — the decision and the write are different events,
and only the write is what a merge reads.
"""
posted, r = _run_classify(
tmp_path, _emitting("docs/a.md"), timeline_mode="stable:0", timeline_terminator=terminator
)
assert r.returncode == 0, r.stderr
assert posted is not None, (
f"a docs-only PR whose timeline terminates with `{terminator}` got NO status at all. The "
"fence could not establish a trusted retarget count, so it withheld the exemption — which "
"leaves the required review-verdict/h10 absent and the PR unmergeable with no bypass "
f"(ersatztv#751).\n{r.stdout[-1200:]}"
)
assert posted["state"] == "success", f"expected the docs-only exemption, got {posted}"
assert "trusted=yes" in r.stdout, (
"the exemption was posted but the fence did not report a trusted count — the two must agree, "
f"or this test is passing for a different reason than it claims.\n{r.stdout[-800:]}"
)
def _marker_steps():
"""(classify body PREFIX through the marker write, guard body that CHECKS it) — from one parse.
THE PREFIX, NOT THE MATCHING LINES. An earlier version picked out the lines matching `RAN_MARKER=`
and `: > "$RAN_MARKER"` and ran those two alone. Cold re-review showed that passes even if the
write is moved into a function nobody calls, or under `if false`: the extractor finds the text,
runs it at top level, the marker appears, and the test reports the guard proven while production
writes no marker at all. Executing the real prefix — everything from the top of the body down to
and including the write — reproduces the production control flow instead of a reconstruction of
it, so a write that is defined-but-not-reached simply does not happen and case B fails.
#
It also pins the property the marker needs anyway: that the write happens EARLY, before anything
in the classifier can fail. If it drifts down past code that needs credentials or network, this
prefix stops executing cleanly and the test says so.
"""
steps = [s for _, s in _iter_workflow_steps(WORKFLOW)]
idx = [i for i, s in enumerate(steps) if "review-verdict/h10" in (s.get("run") or "")]
classify = steps[idx[0]]
lines = classify["run"].splitlines()
write_at = [i for i, ln in enumerate(lines) if re.match(r'\s*:\s*>\s*"\$RAN_MARKER"', ln)]
assert len(write_at) == 1, f'expected exactly one `: > "$RAN_MARKER"` in the classify body, found {len(write_at)}'
prefix = "\n".join(lines[: write_at[0] + 1])
guard = next(s for i, s in enumerate(steps) if i != idx[0] and "h10-classifier-started" in (s.get("run") or ""))
return prefix, guard["run"]
def test_the_dropped_step_guard_BEHAVIOURALLY_fails_without_the_marker_and_passes_with_it(tmp_path):
"""Executes the guard, instead of reading it — the structural test above cannot prove the exit
code, and cold review was right that a bare `exit 1` substring is satisfiable by dead code.
The marker is created by running THE CLASSIFY STEP'S OWN two prologue lines under the same
environment, never by rebuilding the path in Python. That is the point: it proves the two steps
AGREE on the path by construction. A test that computed the name itself would keep passing after
the two steps drifted apart, which is the one divergence that makes the guard fail on every run
and get deleted as broken.
`GITHUB_RUN_ID`/`GITHUB_RUN_ATTEMPT` are set to fixed values, so this also covers the run-keying
added after the live probes: if either step stopped interpolating them, the paths would differ and
case B would fail.
"""
prologue, guard = _marker_steps()
assert prologue.count("RAN_MARKER=") == 1 and ': > "$RAN_MARKER"' in prologue, (
f"could not extract the classify step's marker prologue; got {prologue!r}"
)
# The prefix must be the REAL top of the body, so a write hidden in an uncalled function is not
# executed by this test either.
assert prologue.lstrip().startswith("set -euo pipefail"), (
"the extracted prefix does not start at the top of the classify body, so it is a "
f"reconstruction rather than the production path: {prologue[:120]!r}"
)
env = {
"PATH": os.environ["PATH"],
"RUNNER_TEMP": str(tmp_path),
"GITHUB_RUN_ID": "424242",
"GITHUB_RUN_ATTEMPT": "7",
}
# A — the step was DROPPED: no marker exists. The job must fail.
a = subprocess.run(["bash", "-c", guard], env=env, capture_output=True, text=True)
assert a.returncode != 0, (
"the guard exited 0 with NO start marker present — a dropped classify step would go green "
f"again, which is the whole defect (ersatztv#751).\nstdout: {a.stdout}\nstderr: {a.stderr}"
)
assert "did not execute" in (a.stdout + a.stderr), (
f"the guard failed but without an actionable message: {a.stdout!r} {a.stderr!r}"
)
assert not list(tmp_path.glob("h10-classifier-started*")), (
"the guard itself created the marker it is supposed to be checking for"
)
# B — the classifier RAN: its own prologue created the marker. The guard must pass.
b = subprocess.run(["bash", "-c", prologue + "\n" + guard], env=env, capture_output=True, text=True)
assert b.returncode == 0, (
"the guard rejected a marker written by the classify step's OWN prologue — the two steps "
f"disagree on the path, so this guard would fail on every run.\nstdout: {b.stdout}\n"
f"stderr: {b.stderr}"
)
# C — the write must be reached by STRAIGHT-LINE code. Guarding the prefix trick itself: if the
# write were wrapped in a function or an `if`, the prefix would still contain it but production
# might not reach it. Executing the prefix with the function/conditional intact is the test; this
# assertion makes the intent explicit and fails loudly rather than subtly.
body_before = "\n".join(ln for ln in prologue.splitlines() if not ln.lstrip().startswith("#"))
# Covers all four bash spellings: `mk() {`, `mk(){`, `function mk {`, `function mk() {`. The third
# was added when review found the first regex missed it, and the FOURTH still slipped that fix —
# the union form is the natural next spelling once `function mk {` is caught. Budget three rounds
# for any string-matching predicate.
assert not re.search(r"^\s*(function\s+)?\w+\s*(\(\s*\))?\s*\{", body_before, re.M), (
"a function is defined before the marker write, so the write may be inside it and unreached "
f"in production while this test still passes:\n{body_before}"
)
written = [q.name for q in tmp_path.glob("h10-classifier-started*")]
assert written == ["h10-classifier-started-424242-7"], (
f"the marker is not keyed on the run id/attempt as intended; found {written}. A fixed name in "
"a shared RUNNER_TEMP lets a stale marker satisfy this guard on a run whose step was dropped"
)
@pytest.mark.parametrize("shape", ["null", "array"], ids=["statuses-null", "statuses-empty-array"])
def test_a_head_with_NO_statuses_YET_is_readable_and_still_gets_its_exemption(tmp_path, shape):
"""The twin of the timeline terminator, found by cold review of the fix for that one (#751).
`GET /commits/{sha}/status` returns `{"state":"pending","total_count":0,"statuses":null}` for a
head that has no statuses yet — measured on PR #739's head 5fa672e2. `read_existing_verdict`
gated on `.statuses | type == "array"`, so it read that as unreadable and took its `exit 1`
path: the job posted NOTHING. Fail-closed, but the user-visible result is exactly the outcome
this issue is about — an exempt PR with no status and, since #743, no bypass.
Both shapes are asserted because a head that HAS statuses really does return an array, so the job
must read either. Restoring the `array`-only gate turns 40+ tests red with the corrected double,
which is the measure of how thoroughly the unfaithful stub was hiding this.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_empty_shape=shape)
assert r.returncode == 0, (
f"the classifier died reading a `statuses: {shape}` body instead of treating it as "
f"'no verdict yet', so nothing was posted at all.\n{r.stdout[-1000:]}\n{r.stderr[-600:]}"
)
assert posted is not None, (
f"a docs-only PR whose head has no statuses yet (shape: {shape}) got NO status at all.\n{r.stdout[-1000:]}"
)
assert posted["state"] == "success", f"expected the docs-only exemption, got {posted}"
@pytest.mark.parametrize("path", ["CLAUDE.md", "AGENTS.md"])
def test_the_GOVERNANCE_docs_are_protected_and_get_no_docs_only_exemption(tmp_path, path):
"""`CLAUDE.md`/`AGENTS.md` are not prose about the project — they define the completion protocol,
the merge-consent convention and the H10 rule itself. `DOCS_ONLY` (`^(docs/|[^/]*\\.md$)`) matched
them, so a PR editing the document that specifies what `.claude/` enforces was auto-exemptible
while `.claude/` itself was protected: the same self-exemption the workflow header rules out, one
directory over. Found by cold review of #751 by driving this exact case through the real body.
Latent until #751, because no exemption `success` was writable at all while the classify step was
dropped — restoring the exemptions is what makes it reachable, which is why it is fixed there.
Asserted through the PROTECTED branch specifically, not merely "not exempt": `pending` is reached
by several routes and a test that accepted any of them could not tell a working guard from a dead
one.
"""
posted, r = _run_classify(tmp_path, _emitting(path))
assert posted is not None, f"the job posted nothing: {r.stderr[-1500:]}"
assert posted["state"] == "pending", (
f"{path} was granted an exemption ({posted}) — the document that DEFINES the merge gate must "
f"not be able to exempt itself from it.\n{r.stdout[-800:]}"
)
assert "protected" in r.stdout.lower(), (
f"{path} was not exempted, but not via the protected-path branch either, so that guard may be "
f"dead for it.\n{r.stdout[-800:]}"
)
def test_a_terminator_on_PAGE_ONE_does_not_certify_a_zero_retarget_count(tmp_path):
"""`null` means "exhausted" only after a real page has been read.
THE INVARIANT, not a figure that rots: a real PR's timeline always carries at least one event on
page 1, because the PR is created by a push and that is itself an event. Spot-checked non-empty
across #752/#753/#749/#739/#717; the counts are not recorded, because an earlier version of this
docstring cited five and three were stale within days. So a terminator on page 1 is anomalous, not
empty. Trusting a zero count from it would mean
certifying that no retarget happened on the strength of a response we cannot explain, which is the
one thing the fence exists to refuse. Withholding the exemption is the safe direction: the PR asks
for a human verdict instead.
This narrows rather than closes the concern — a wrong `null` on page 3 is still read as exhaustion,
and no bounded number of round-trips can rule that out.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="empty-first-page")
assert r.returncode == 0, r.stderr
assert posted is None, (
"an exemption `success` was posted from a timeline whose FIRST page was already the "
f"terminator, so no page of events was ever actually read: {posted}\n{r.stdout[-800:]}"
)
assert "trusted=no" in r.stdout, (
f"the exemption was withheld, but not because the count was untrusted:\n{r.stdout[-800:]}"
)
def test_a_SECOND_PAGE_of_statuses_refuses_to_conclude_that_no_verdict_exists(tmp_path):
"""Completeness for the "no verdict exists" conclusion — the one that licenses posting over a
verdict this job cannot see.
THE TWO OBVIOUS GUARDS ARE BOTH NO-OPS HERE, which is why this costs a round-trip:
* `.statuses | length` vs `.total_count` — `total_count` is the count for the PAGE RETURNED, not
the commit. Measured at 1.27.1 on 3aed43c6 (6 contexts): `?limit=1` -> `len=1, total_count=1`.
Equal by construction. The stub mirrors that, so this test cannot pass for that wrong reason.
* "refuse when the page came back full at the requested limit of 100" — this instance caps `limit`
at `MAX_RESPONSE_ITEMS`, measured at 50 (`/issues?limit=100` returns 50), so a response can never
hold 100 rows and the comparison was DEAD CODE. The repo already documented that cap in three
places; the guard was written against 100 anyway and a cold review caught it.
So the job asks the server, and only when the row is absent from page 1. Any rows on page 2 mean
the list is longer than one page and the verdict may be beyond it.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="twopage")
assert posted is None, (
"an exemption was posted while the status list ran to a second page, so an existing verdict "
f"beyond page 1 would have been silently overwritten: {posted}\n{r.stdout[-800:]}"
)
assert "page 2" in (r.stdout + r.stderr).lower(), (
f"nothing was posted, but not because of the page-2 completeness probe:\n{r.stdout[-800:]}"
)
def test_a_SINGLE_page_of_statuses_reads_normally(tmp_path):
"""Positive control for the probe above: without it, "refuse when page 2 has rows" could be
satisfied by refusing always, which deadlocks every PR while looking safe. 40 decoy contexts on
page 1, nothing on page 2.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="rows:40")
assert r.returncode == 0, r.stderr
assert posted is not None and posted["state"] == "success", (
f"a single page of statuses should read normally and still exempt; got {posted}\n{r.stdout[-800:]}"
)
def test_a_STRING_total_count_is_not_accepted_as_numeric_zero(tmp_path):
"""`jq -r` renders the JSON number 0 and the JSON string "0" identically, so a text compare
accepts a schema-corrupted `"total_count": "0"` as "no statuses" (cold re-review reproduced it).
The live schema uses an integer, so this is not a live failure — it is the difference between a
guard that holds because the input happens to be well-formed and one that holds because it checks.
The accept path requires the TYPE to be number.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="total-count-string")
assert posted is None, (
f"a string total_count was accepted as numeric zero, so a body that merely lost its statuses "
f"array reads as 'no verdict exists': {posted}\n{r.stdout[-800:]}"
)
@pytest.mark.parametrize(
"mode,why",
[
("page2-garbage", "a non-JSON page 2"),
("page2-error", "an HTTP error on page 2"),
],
ids=["garbage", "transport-error"],
)
def test_an_UNREADABLE_page_2_refuses_to_conclude_that_no_verdict_exists(tmp_path, mode, why):
"""The two refuse branches of the completeness probe, which cold review found untested.
Worth a test rather than trusting the shape: this file's history is two consecutive guards that
were UNREACHABLE — the `count_retargets` type gate that never saw a real terminator, and a
full-page check written against a limit of 100 on a server that caps at 50. An unexercised branch
here has a track record.
Both must fail CLOSED: the point of reading page 2 is to justify "no verdict exists", so a page 2
that cannot be read justifies nothing.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode=mode)
assert posted is None, (
f"an exemption was posted despite {why}, so 'no verdict exists' was concluded without "
f"evidence: {posted}\n{r.stdout[-800:]}"
)
assert "page 2" in (r.stdout + r.stderr).lower(), (
f"nothing was posted, but not because of the page-2 probe:\n{r.stdout[-800:]}"
)
def test_a_status_history_RUNNING_PAST_PAGE_1_repairs_rather_than_leaving_green(tmp_path):
"""The post-write race check reads ONE page of `/statuses/{sha}`, and `limit=100` clamps to the
server-wide cap (measured 50). So "no raced row on page 1" does not establish "no race" — a raced
human `failure` can sit on a page this job never reads.
This is the ONE path in the design whose failure direction is toward SUCCESS: missing a raced
verdict leaves a forged green over a human rejection, permanently. So unread rows are treated as a
race and the exemption is repaired to `pending`, which is the conservative direction — a stall a
reviewer can clear, rather than a rejection silently turned green.
The stub's second page carries ORDINARY rows, no verdict and no sentinel: what must trigger the
repair is the mere existence of rows this job did not read, not the discovery of a verdict.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="second-page")
assert r.returncode == 0, r.stderr
seq = _posted_sequence(tmp_path)
assert len(seq) == 2, (
"the exemption was posted and left standing even though the status history ran past page 1, so "
f"a raced verdict beyond it would be buried. Posts: {seq}\n{r.stdout[-900:]}"
)
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
f"expected an exemption then a repair to pending; got {seq}"
)
assert seq[1]["description"] == "Human verdict raced this exemption write — re-post the verdict", (
f"repaired, but not to the sentinel, so the fixed point is broken: {seq[1]}"
)
@pytest.mark.parametrize(
"mode,why",
[
("second-page-garbage", "a non-JSON page 2"),
("second-page-error", "an HTTP error on page 2"),
],
ids=["garbage", "transport-error"],
)
def test_an_UNREADABLE_history_page_2_also_repairs_rather_than_leaving_green(tmp_path, mode, why):
"""Found by mutating the branch and watching nothing go red — my own coverage gap, in the same
class the review had just flagged twice.
This is the fail-toward-SUCCESS path, so uncertainty must resolve to `pending`. "I could not read
the rest of the history" is not evidence that no verdict raced this write; treating it as such is
exactly how a forged green survives over a human rejection.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode=mode)
assert r.returncode == 0, r.stderr
seq = _posted_sequence(tmp_path)
assert len(seq) == 2, (
f"the exemption was left standing despite {why} — a raced verdict beyond page 1 would be "
f"buried. Posts: {seq}\n{r.stdout[-900:]}"
)
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
f"expected an exemption then a repair to pending; got {seq}"
)