timothy/ersatztv
1
0
Fork 0
Code Issues 58 Pull Requests 8 Actions Packages Projects Releases 9 Wiki Activity

fix(751): review round — the twin null-shape, a governance self-exemption, and four over-claims

Browse Source 
Two independent cold reviews (a cross-family GPT-5.6 pass and an isolated Opus pass).
Neither found a path to a green `review-verdict/h10` on an unreviewed head. Both found
real defects BESIDE the fix, which is the failure mode this file keeps producing.

THE TWIN, and the reason not to trust "I fixed the two I could see". `GET
/commits/{sha}/status` returns `statuses: null` — not `[]` — for a head with no
statuses yet: `{"state":"pending","total_count":0,"statuses":null}`, measured on PR
#739's head. `read_existing_verdict` gated on `.statuses | type == "array"` and took
its `exit 1` path, posting NOTHING. Fail-closed, but the user-visible outcome is the
one this issue is about: an exempt PR with no status and, since #743, no bypass. Its
double printed `{"statuses": []}` at all three no-verdict sites, so that branch was
unreachable in the suite — the same unfaithful-double story as the timeline, one
function over. `null` is accepted only when `total_count` is 0, so a body that merely
lost its array is still refused and an existing verdict is still protected. Swept
`scripts/pr-changed-files.sh` too: `pulls/{n}/files` returns `[]`, unaffected. The
generalisable rule is that a nil Go slice serialises to `null`, so every list-shaped
field on this API is suspect and only a per-endpoint measurement settles it.

A GOVERNANCE SELF-EXEMPTION, reachable again precisely because this change works.
`DOCS_ONLY` matched `CLAUDE.md` and `AGENTS.md` — the documents that DEFINE the
completion protocol, the merge-consent convention and the H10 rule. Driving the real
classify body with a lone `CLAUDE.md` change produced `review-verdict/h10=success`.
Protecting `.claude/` while the file specifying what it enforces stayed exemptible is
the same self-exemption the header rules out, one directory over. Both added to
PROTECTED; `README.md` deliberately not (ordinary prose, no enforcement).

FOUR OVER-CLAIMS, corrected rather than defended:

* The repo-wide expression test does NOT catch "any payload that cannot evaluate".
  It checks the HEAD TOKEN of each dotted path. `${{ github.ref == }}` and
  `${{ …head.sha + }}` pass; so does a renamed output, since tokens after the first
  are skipped by design. Claim corrected in the docstring, `docs/ci-cd.md` and the
  record. The test is kept permissive on purpose: a red here blocks every merge.
* The strict test's anti-vacuity half banned expressions ANYWHERE outside
  `with:`/`env:`, so the standard `if: ${{ always() }}` spelling and even a delimiter
  in an inert top-level comment went red — a guard more dangerous than its target.
  Replaced with the honest property: the YAML walk saw every `run:` body it declares.
* The `if:` assertion demanded the bare `always()` exactly; now normalised, since the
  wrapped form is identical to the runner.
* `exit 1` was matched anywhere in the guard body, so an unreachable
  `if false; then exit 1; fi` satisfied it while the real branch said `exit 0`. Now
  required INSIDE the missing-marker branch — and the new behavioural test settles it
  properly by EXECUTING the guard body both ways.
* The record asserted a repo-wide obligation to guard consequential steps. It is not
  repo-wide: `docker-build.yml`'s `test`/`migrations` are also required contexts and a
  dropped step there is fail-OPEN (green having done no work), strictly worse than
  here. Scoped to this file and tracked as #756 rather than asserted as done.

Also: comments in both files still said it was unestablished whether a later step runs
after a drop — runs 1863/1866 established it, so they now record the measurement; a
cited test name that never existed; `kind` leaked to global scope; a mangled comment
wrap; and an already-false "one event on page 1".

Hardening of my own: `null` now counts as exhaustion only from page 2 ON. Every real
PR's first page carries events (4, 2, 9, 5, 10 across #752/#753/#749/#739/#717), so a
terminator on page 1 means no page was ever read, and certifying "no retarget" from a
response we cannot explain is the one thing the fence exists to refuse. Narrows rather
than closes it: a wrong `null` on page 3 still reads as exhaustion.

Verification: 137 in this file / 452 total green; ELEVEN mutations each red —
reintroducing the defect, deleting the guard, deleting the marker write, removing
`if: always()`, `exit 1`→`exit 0`, a delimiter in the guard body, the fence gate (20
red), the TWIN gate (70 red), dropping the governance paths, accepting a null first
page, and diverging the marker path between the two steps.

Refs: #751
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
timothy
2026-08-06 21:35:56 +02:00
co-authored by Claude Opus 5
parent 6b743be988
commit c710db4a1d
6 changed files with 342 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/tests/test_pr_changed_files.py
+213 -29
View File
@@ -923,6 +923,11 @@ if "/timeline" in url:
for part in url.split("?", 1)[-1].split("&"):
if part.startswith("page="):
page = int(part.split("=", 1)[1])
if mode == "empty-first-page":
# A terminator on PAGE 1 — no real page ever read (ersatztv#751). The job must NOT certify a
# zero retarget count from this, so the exemption is withheld.
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
sys.exit(0)
if mode == "unreadable":
print("<html>502 Bad Gateway</html>")
sys.exit(0)
@@ -1001,6 +1006,19 @@ if "/status" in url:
# never-overwrite short-circuit unreachable: neither could be made to fire, so mutations
# deleting them survived the whole suite.
mode = os.environ.get("STUB_STATUS_MODE", "none")
# THE EMPTY SHAPE IS `statuses: null`, NOT `[]` (ersatztv#751). Measured on this instance: a head
# with no statuses returns `{"state":"pending","total_count":0,"statuses":null}` — PR #739's head
# 5fa672e2. This stub printed `{"statuses": []}` at all three no-verdict sites, which is a shape
# the server does not produce for that case, so the job's `.statuses | type == "array"` gate was
# never exercised against reality and its `exit 1` branch — posting nothing at all — was
# unreachable in the suite. Same class as the timeline terminator, one function over.
# Parameterised, not merely corrected: a head that HAS statuses really does return an array, so
# both shapes are live and the job must read either.
def empty_statuses():
if os.environ.get("STUB_STATUS_EMPTY_SHAPE", "null") == "array":
return json.dumps({"state": "pending", "total_count": 0, "statuses": []})
return json.dumps({"state": "pending", "total_count": 0, "statuses": None})
if mode.startswith("appears-on-read:"):
# A human verdict that does NOT exist at the first read and DOES exist at the re-read made
# immediately before the POST (ersatztv#706). Models a reviewer posting BLOCKED while the job
@@ -1010,7 +1028,7 @@ if "/status" in url:
n = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(n + 1))
if n + 1 < nth:
print(json.dumps({"statuses": []}))
print(empty_statuses())
sys.exit(0)
print(json.dumps({"statuses": [
{"context": "review-verdict/h10", "status": "failure",
@@ -1025,7 +1043,7 @@ if "/status" in url:
n = int(ctr.read_text()) if ctr.exists() else 0
ctr.write_text(str(n + 1))
if n + 1 < nth:
print(json.dumps({"statuses": []})); sys.exit(0)
print(empty_statuses()); sys.exit(0)
print(json.dumps({"statuses": [
{"context": "review-verdict/h10", "status": "pending", "creator": None,
"description": "Human verdict raced this exemption write — re-post the verdict"}]}))
@@ -1056,7 +1074,7 @@ if "/status" in url:
"creator": ({"login": creator} if creator else None), "description": desc},
{"context": "Functional E2E", "status": "pending"}]}))
sys.exit(0)
print(json.dumps({"statuses": []}))
print(empty_statuses())
sys.exit(0)
print("{}")
@@ -1068,7 +1086,8 @@ def _run_classify(tmp_path, enum_stub: str | None, author: str = "timothy",
status_creator: str | None = "timothy",
status_desc: str = "Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
timeline_mode: str = "none", history_mode: str = "none",
timeline_terminator: str = "null"):
timeline_terminator: str = "null",
status_empty_shape: str = "null"):
"""Execute the workflow's classify `run:` block with a stubbed enumeration script.
Returns the status payload the job POSTed, or None if it posted nothing.
@@ -1095,6 +1114,7 @@ def _run_classify(tmp_path, enum_stub: str | None, author: str = "timothy",
env["STUB_STATUS_DESC"] = status_desc
env["STUB_TIMELINE_MODE"] = timeline_mode
env["STUB_TIMELINE_TERMINATOR"] = timeline_terminator
env["STUB_STATUS_EMPTY_SHAPE"] = status_empty_shape
env["STUB_HISTORY_MODE"] = history_mode
env.update({
"GITEA_TOKEN": "stub",
@@ -2202,15 +2222,25 @@ def test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body():
"delimiters. Pass values in through the step's `env:` block, which is interpolated per "
"value, so a bad payload there cannot take the body with it."
)
# Belt-and-braces on the parse above: a `run:` body the YAML walk failed to reach would make
# this test vacuously green, which is the failure mode `eight ways a test passes for the wrong
# reason` warns about. The gate file's expressions all live in `with:`/`env:` blocks, so the raw
# count and the per-field count must agree exactly.
in_fields = sum(1 for _ in _EXPR.finditer(_workflow_expression_fields_text(WORKFLOW)))
assert len(list(_EXPR.finditer(raw))) == in_fields, (
"review-verdict.yml contains expression delimiters that are not in a `with:`/`env:` value. "
"The per-step scan above may not have reached them, so it cannot be trusted — locate them "
"before relying on a green here")
# ANTI-VACUITY, and it has to be the right property. A `run:` body the YAML walk never reached
# would make the assertion above vacuously green — the failure mode to guard against. An earlier
# version of this guard compared the file's TOTAL delimiter count against the count inside
# `with:`/`env:` values, which is a different and wrong claim: it banned expressions everywhere
# else in the file too. Both reviewers reproduced the false red — writing `if: ${{ always() }}`,
# the standard and equivalent spelling of the `if:` two steps below, turned this test red, as did
# a delimiter in an inert top-level YAML comment. Neither is unsafe, and a red here blocks every
# merge through the combined status, so the guard was strictly more dangerous than the thing it
# was protecting against.
#
# The honest anti-vacuity check is that the walk SAW every `run:` body: compare the number of
# bodies the parse produced against the number of `run:` keys in the raw text. That fails if the
# walk misses a step and says nothing about delimiters anywhere else.
walked = sum(1 for _, s in _iter_workflow_steps(WORKFLOW) if s.get("run"))
declared = len(re.findall(r"^\s{4,}run:\s*[|>]", raw, re.M))
assert walked == declared, (
f"the YAML walk reached {walked} run: bodies but the file declares {declared}. The assertion "
"above may not have examined every body, so a green here proves nothing — find the step the "
"walk missed before trusting this test")
def _iter_workflow_steps(path: Path):
@@ -2242,10 +2272,22 @@ def test_every_workflow_expression_names_a_REAL_context_or_function():
context that does not exist. All of them fail the same silent way, and in an `if:` the
consequence is the same shape as #751: the step does not run and nothing is red.
A full expression parser is not needed to catch it. Every payload the runner accepts begins its
dotted paths at a known context and calls only known functions, so checking the head token of
each path is enough. `pr number` — the real defect — fails here on `pr`. Verified against all
31 distinct payloads in this repo today, which pass.
BE PRECISE ABOUT WHAT THIS ENFORCES, because the first version of this docstring was not and
both reviewers caught it: it checks that THE HEAD TOKEN of each dotted path is a known context or
function. Nothing more. That catches the historical defect — `pr number` fails on `pr` — and a
payload naming a context that does not exist. It does NOT catch:
* syntactically invalid expressions whose tokens are all known: `${{ github.ref == }}` and
`${{ github.event.pull_request.head.sha + }}` both pass, verified;
* a renamed or misspelled output or property, because every token after the first is preceded
by `.` and is deliberately skipped: `steps.metadata.outputs.shortsha` passes;
* an unclosed opener, since `_EXPR` requires the closing braces to match at all.
A real fix for those is an expression parser, which is a different and much larger change. This is
a cheap net under the specific class that has bitten us, deliberately kept permissive so it cannot
false-red the repo (a red here blocks every merge through the combined status). Verified against
all 31 distinct payloads in this repo today, which pass. Do not restate this test as "catches any
payload that cannot evaluate" — that claim was in the docs and the decision record and was false.
"""
offenders = []
for wf in _workflow_files():
@@ -2282,12 +2324,14 @@ def test_a_dropped_classify_step_FAILS_the_job_instead_of_going_green():
(fail-closed, but it would look like this guard is broken rather than like a real drop), and a
test carrying its own third copy of the literal could not see the divergence at all.
What this test does NOT establish, said plainly: that the runner still executes a LATER step
after dropping an earlier one. Nothing in the #751 evidence answers that — the classifier was
the last step in the job, so there was never a subsequent step to observe. Only the scratch-base
probe can answer it, with a negative control that reintroduces a bad payload and must produce a
RED job. If the runner drops the remaining steps too, this guard is inert and the body has to
move into `scripts/`, where a one-line `run:` makes the class unreachable.
The premise — that the runner still executes a LATER step after dropping an earlier one — is
MEASURED, not assumed, and could not be settled from the #751 report because the classifier was
the job's last step. Gitea 1.27.1, 2026-08-06: probe run 1863 dropped the classifier on a
reintroduced bad payload, ran the guard anyway (`always()` evaluated true), and the job concluded
`failure`; run 1866 is the positive control. So the guard is load-bearing in production and the
body does not need to move into `scripts/`. This test pins the SHAPE; the probe pinned the
BEHAVIOUR, and neither substitutes for the other — which is why the behavioural test below
EXECUTES the guard body rather than only reading it.
"""
# ONE parse, and both steps located inside it. Using `_classify_step()` here instead was wrong
# in a way worth recording, because it went green-adjacent rather than loud: that helper does its
@@ -2322,12 +2366,26 @@ def test_a_dropped_classify_step_FAILS_the_job_instead_of_going_green():
f"the dropped-step guard is step {guard_idx[0]} but the classifier is step "
f"{classifiers[0]} — a guard that runs first always fails")
guard = steps[guard_idx[0]]
assert str(guard.get("if", "")).strip() == "always()", (
f"the dropped-step guard's `if:` is {guard.get('if')!r}; it must be `always()`, or it will "
"be skipped on exactly the runs where the classifier failed")
assert re.search(r"exit\s+1", guard["run"]), (
"the dropped-step guard never exits non-zero, so it observes the failure and still lets the "
"job go green — which is the whole defect")
# `always()` and `${{ always() }}` are the same condition; the runner accepts both and the second
# is the more common spelling. Pinning the bare form EXACTLY would red the repo over a
# semantically identical edit, so normalise instead. What must not change is that the guard runs
# when the classifier failed.
guard_if = re.sub(r"\s+", "", str(guard.get("if", "")))
assert guard_if in ("always()", "${{always()}}"), (
f"the dropped-step guard's `if:` is {guard.get('if')!r}; it must be `always()` (bare or "
"wrapped), or it will be skipped on exactly the runs where the classifier failed")
# INSIDE the missing-marker branch, not merely somewhere in the body. Cold review pointed out
# that a bare `exit 1` substring is satisfied by an unreachable `if false; then exit 1; fi` while
# the real branch says `exit 0` — the test passes and a dropped classifier goes green again. The
# behavioural test below is the real proof; this keeps the structural one from being satisfiable
# by dead code.
missing_branch = re.search(r'if \[ ! -f "\$marker" \]; then(.*?)\bfi\b', guard["run"], re.S)
assert missing_branch, (
"the dropped-step guard no longer tests for a MISSING marker with `if [ ! -f \"$marker\" ]`, "
"so the assertion below cannot locate the branch that must fail the job")
assert re.search(r"exit\s+1", missing_branch.group(1)), (
"the dropped-step guard detects the missing marker but does not `exit 1` inside that branch, "
"so it observes the failure and still lets the job go green — which is the whole defect")
assert not _EXPR.search(guard["run"]), (
"the dropped-step guard's own run body contains an expression delimiter, so the mechanism "
"it guards against can drop the guard too — and that absence would be silent as well")
@@ -2367,3 +2425,129 @@ def test_the_fence_TRUSTS_the_count_and_POSTS_when_the_timeline_terminates(tmp_p
assert "trusted=yes" in r.stdout, (
"the exemption was posted but the fence did not report a trusted count — the two must agree, "
f"or this test is passing for a different reason than it claims.\n{r.stdout[-800:]}")
def _marker_steps():
"""(classify prologue that CREATES the marker, guard body that CHECKS it) — from one parse."""
steps = [s for _, s in _iter_workflow_steps(WORKFLOW)]
idx = [i for i, s in enumerate(steps) if "review-verdict/h10" in (s.get("run") or "")]
classify = steps[idx[0]]
prologue = [ln for ln in classify["run"].splitlines()
if re.match(r'\s*RAN_MARKER=', ln) or re.match(r'\s*:\s*>\s*"\$RAN_MARKER"', ln)]
guard = next(s for i, s in enumerate(steps)
if i != idx[0] and "h10-classifier-started" in (s.get("run") or ""))
return "\n".join(ln.strip() for ln in prologue), guard["run"]
def test_the_dropped_step_guard_BEHAVIOURALLY_fails_without_the_marker_and_passes_with_it(tmp_path):
"""Executes the guard, instead of reading it — the structural test above cannot prove the exit
code, and cold review was right that a bare `exit 1` substring is satisfiable by dead code.
The marker is created by running THE CLASSIFY STEP'S OWN two prologue lines under the same
environment, never by rebuilding the path in Python. That is the point: it proves the two steps
AGREE on the path by construction. A test that computed the name itself would keep passing after
the two steps drifted apart, which is the one divergence that makes the guard fail on every run
and get deleted as broken.
`GITHUB_RUN_ID`/`GITHUB_RUN_ATTEMPT` are set to fixed values, so this also covers the run-keying
added after the live probes: if either step stopped interpolating them, the paths would differ and
case B would fail.
"""
prologue, guard = _marker_steps()
assert prologue.count("RAN_MARKER=") == 1 and ': > "$RAN_MARKER"' in prologue, (
f"could not extract the classify step's marker prologue; got {prologue!r}")
env = {"PATH": os.environ["PATH"], "RUNNER_TEMP": str(tmp_path),
"GITHUB_RUN_ID": "424242", "GITHUB_RUN_ATTEMPT": "7"}
# A — the step was DROPPED: no marker exists. The job must fail.
a = subprocess.run(["bash", "-c", guard], env=env, capture_output=True, text=True)
assert a.returncode != 0, (
"the guard exited 0 with NO start marker present — a dropped classify step would go green "
f"again, which is the whole defect (ersatztv#751).\nstdout: {a.stdout}\nstderr: {a.stderr}")
assert "did not execute" in (a.stdout + a.stderr), (
f"the guard failed but without an actionable message: {a.stdout!r} {a.stderr!r}")
assert not list(tmp_path.glob("h10-classifier-started*")), (
"the guard itself created the marker it is supposed to be checking for")
# B — the classifier RAN: its own prologue created the marker. The guard must pass.
b = subprocess.run(["bash", "-c", prologue + "\n" + guard], env=env,
capture_output=True, text=True)
assert b.returncode == 0, (
"the guard rejected a marker written by the classify step's OWN prologue — the two steps "
f"disagree on the path, so this guard would fail on every run.\nstdout: {b.stdout}\n"
f"stderr: {b.stderr}")
written = [q.name for q in tmp_path.glob("h10-classifier-started*")]
assert written == ["h10-classifier-started-424242-7"], (
f"the marker is not keyed on the run id/attempt as intended; found {written}. A fixed name in "
"a shared RUNNER_TEMP lets a stale marker satisfy this guard on a run whose step was dropped")
@pytest.mark.parametrize("shape", ["null", "array"],
ids=["statuses-null", "statuses-empty-array"])
def test_a_head_with_NO_statuses_YET_is_readable_and_still_gets_its_exemption(tmp_path, shape):
"""The twin of the timeline terminator, found by cold review of the fix for that one (#751).
`GET /commits/{sha}/status` returns `{"state":"pending","total_count":0,"statuses":null}` for a
head that has no statuses yet — measured on PR #739's head 5fa672e2. `read_existing_verdict`
gated on `.statuses | type == "array"`, so it read that as unreadable and took its `exit 1`
path: the job posted NOTHING. Fail-closed, but the user-visible result is exactly the outcome
this issue is about — an exempt PR with no status and, since #743, no bypass.
Both shapes are asserted because a head that HAS statuses really does return an array, so the job
must read either. Restoring the `array`-only gate turns 40+ tests red with the corrected double,
which is the measure of how thoroughly the unfaithful stub was hiding this.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_empty_shape=shape)
assert r.returncode == 0, (
f"the classifier died reading a `statuses: {shape}` body instead of treating it as "
f"'no verdict yet', so nothing was posted at all.\n{r.stdout[-1000:]}\n{r.stderr[-600:]}")
assert posted is not None, (
f"a docs-only PR whose head has no statuses yet (shape: {shape}) got NO status at "
f"all.\n{r.stdout[-1000:]}")
assert posted["state"] == "success", f"expected the docs-only exemption, got {posted}"
@pytest.mark.parametrize("path", ["CLAUDE.md", "AGENTS.md"])
def test_the_GOVERNANCE_docs_are_protected_and_get_no_docs_only_exemption(tmp_path, path):
"""`CLAUDE.md`/`AGENTS.md` are not prose about the project — they define the completion protocol,
the merge-consent convention and the H10 rule itself. `DOCS_ONLY` (`^(docs/|[^/]*\\.md$)`) matched
them, so a PR editing the document that specifies what `.claude/` enforces was auto-exemptible
while `.claude/` itself was protected: the same self-exemption the workflow header rules out, one
directory over. Found by cold review of #751 by driving this exact case through the real body.
Latent until #751, because no exemption `success` was writable at all while the classify step was
dropped — restoring the exemptions is what makes it reachable, which is why it is fixed there.
Asserted through the PROTECTED branch specifically, not merely "not exempt": `pending` is reached
by several routes and a test that accepted any of them could not tell a working guard from a dead
one.
"""
posted, r = _run_classify(tmp_path, _emitting(path))
assert posted is not None, f"the job posted nothing: {r.stderr[-1500:]}"
assert posted["state"] == "pending", (
f"{path} was granted an exemption ({posted}) — the document that DEFINES the merge gate must "
f"not be able to exempt itself from it.\n{r.stdout[-800:]}")
assert "protected" in r.stdout.lower(), (
f"{path} was not exempted, but not via the protected-path branch either, so that guard may be "
f"dead for it.\n{r.stdout[-800:]}")
def test_a_terminator_on_PAGE_ONE_does_not_certify_a_zero_retarget_count(tmp_path):
"""`null` means "exhausted" only after a real page has been read.
Every real PR's first page carries events — measured 4, 2, 9, 5, 10 on #752/#753/#749/#739/#717 —
so a terminator on page 1 is anomalous, not empty. Trusting a zero count from it would mean
certifying that no retarget happened on the strength of a response we cannot explain, which is the
one thing the fence exists to refuse. Withholding the exemption is the safe direction: the PR asks
for a human verdict instead.
This narrows rather than closes the concern — a wrong `null` on page 3 is still read as exhaustion,
and no bounded number of round-trips can rule that out.
"""
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="empty-first-page")
assert r.returncode == 0, r.stderr
assert posted is None, (
"an exemption `success` was posted from a timeline whose FIRST page was already the "
f"terminator, so no page of events was ever actually read: {posted}\n{r.stdout[-800:]}")
assert "trusted=no" in r.stdout, (
f"the exemption was withheld, but not because the count was untrusted:\n{r.stdout[-800:]}")