Files
ersatztv/scripts/tests/test_post_review_verdict.py
T
timothyandClaude Fable 5.1 a7d91bf15a
Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 35s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 57s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 37s
PR Gates / Docs update reminder (pull_request) Successful in 1m0s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 17s
review-verdict/h10 Review-verdict: MERGEABLE @ a7d91bf (base: main)
Review verdict / Set review-verdict status (pull_request_target) Successful in 45s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m25s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m17s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 19m27s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m4s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
fix(876): sweep session narrative out of hooks, workflows, scripts, tests and code comments; grow the detector to the process corpus
`docs.no-session-narrative` reaches every durable artifact, but its detector scanned only
`docs/**/*.md` and root markdown, and nothing had ever swept the rest. The issue named four sites
from one grep and called them a floor. Deriving the population instead — a whitespace-joined sweep
over every tracked file outside the detector, for the detector's own phrasings plus the attribution
and review-round class #812 found — gave 453 sites in 108 files at `fb5592971`, and a second pass
for phrasings the first list missed (hyphenated `round-N`, "an earlier version", "the reviewer
proved") added residuals in the same files. Every site was classified with #812's three
dispositions (CUT / SEVER / KEEP with its sub-kind) under the who-benefits test; the per-site
manifests are on the PR. The rejected designs, tested-and-rejected fixtures, measurements and
traps stay; the attribution of who found them and the round in which they were found go.

The detector's population grows to `.claude/`, `.gitea/`, `.husky/` and `scripts/` regardless
of extension, minus the detector and its own test (whose fixtures ARE the phrasings) and minus
`scripts/tests/fixtures/` (test data, including decision-record copies — the same reasoning as
the records' own exemption, and what keeps the record's depth measurement true), and `--all`
lists tracked REGULAR files only — a symlink's content is its target and a gitlink has none. The #812
argument for leaving `docs/superpowers/**` in the population runs the other way here: `--diff`
sees only ADDED lines, and 287 of the 453 sites were under 30 days old — this corpus is where
narrative is being added, so the advisory nudge has reach. Density agrees: 56 line-mode hits over
the 113 regular files the predicate admits, against 9 over 66 docs files before #812. `web/` and C# stay out on the same
measurement (3 of 74 PATTERNS-matching sites, ~4,600 files). The predicate did not grow: PATTERNS
matched 74 of 453 sites, and widening the word list to the attribution class is the treadmill
the withdrawn parity test ran on. The population oracle is restated over segments with the new
arms, the synthetic cross product gains the process heads and non-markdown extensions, a fixture
witnesses that a tracked symlink is neither scanned nor counted, a `.py.bak` axis separates a
by-name exemption from a `startswith` over the same tuple, and eight mutants (drop the process
arm, drop the by-name exemption, exempt by `startswith`, drop or add a prefix, drop the fixtures
exemption, list only markdown, drop the symlink filter, test the mode per row instead of per
path) each
redden it. A pre-existing silent drop in `--diff` goes with it: git tab-terminates a `+++`
filename that contains a space, and the kept tab made `is_scanned_path` refuse the file with no
notice — fixed, with a positive control and its own mutant.

Code is unchanged by construction, measured per file type against `origin/main`: Python modules
are AST-equal with docstrings stripped, except `#` lines inside the embedded fixture programs
(string literals) of three test modules; workflows differ only in `#` lines inside `run:` block
scalars; shell, C#, TypeScript and jq are equal with comment lines stripped. The stated
exceptions: the detector and its test, 26 vitest titles that carried review-round or severity
labels or a reviewer attribution (call sites whose title changed — every changed title line
walked back to its `it(` / `it.each(...)(` anchor, so a `' + '` concatenation counts once), two
registry note strings and the mutation manifest's prose fields. scripts/tests: 1565 passed.
Web: lint, typecheck, 1319 tests green. Closes #876.

Decisions-Edit: yes
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PEcBoFw7ctrf3Nb7R7x7wk
2026-09-03 20:51:39 +02:00

1283 lines
59 KiB
Python

"""Tests for `scripts/post-review-verdict.sh` (ersatztv#622).
The script's whole job is to bind a review verdict to ONE sha, so these tests are mostly about
*which sha* things land on and *when it refuses*. Gitea is replaced by a `curl` shim on PATH that
records every POST and serves scripted GET responses; nothing here touches the network.
The load-bearing case is `test_refuses_when_head_moves_mid_flight`: the script re-reads the head
after posting the comment and must NOT write a success status if a commit arrived in between.
Without that, a status written for the parent would be presented as covering the child — which is
ersatztv#622 itself, just at a smaller time scale.
`test_comment_matches_the_hook_parser` is a cross-check rather than a unit test: it runs the comment
body this script produces through the *actual* H10 classifier the hook uses for condition (c),
`scripts/check-review-verdict.sh`. A drift between the two would be invisible until a merge
mysteriously stalled.
It originally re-implemented the hook's regexes in Python and asserted the shell source still
contained them. #629 removed that mirror: three false-opens had survived precisely because the
grammar existed in two places, and a Python copy would have kept passing while the shell drifted.
The grammar now lives in one tested script, so this calls it.
"""
from __future__ import annotations
import json
import os
import re
import shutil
import subprocess
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT = REPO_ROOT / "scripts" / "post-review-verdict.sh"
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
CLASSIFIER = REPO_ROOT / "scripts" / "check-review-verdict.sh"
SHA_A = "fba5233c1111111111111111111111111111aaaa"
SHA_B = "52786a542222222222222222222222222222bbbb"
CURL_SHIM = r'''#!/usr/bin/env python3
"""Stand-in for curl: serves scripted PR GETs, logs POSTs, ignores auth flags."""
import json, os, sys, pathlib, urllib.parse
state = pathlib.Path(os.environ["STUB_DIR"])
args = sys.argv[1:]
url = [a for a in args if a.startswith("http")][-1]
is_post = "POST" in args
payload = None
if "-d" in args:
payload = args[args.index("-d") + 1]
if is_post:
# A POST can be scripted to fail (`post_fail` holds a URL substring), so the two write paths
# can be broken independently — the shape ersatztv#792 is about.
fail_on = (state / "post_fail").read_text().strip() if (state / "post_fail").exists() else ""
if fail_on and fail_on in url:
sys.exit(22)
with (state / "posts.jsonl").open("a") as fh:
fh.write(json.dumps({"url": url, "payload": json.loads(payload)}) + "\n")
print("{}")
sys.exit(0)
# GET /repos/<owner>/<repo>/pulls/<n> -> consume the next scripted head sha.
if "/pulls/" in url and not url.endswith("/files"):
shas = (state / "pr_shas").read_text().split()
counter = state / "pr_get_count"
n = int(counter.read_text()) if counter.exists() else 0
counter.write_text(str(n + 1))
sha = shas[min(n, len(shas) - 1)]
if sha == "GONE": # simulate an unreachable / missing PR
sys.exit(22)
# The base branch is scripted on the same consume-one-per-GET schedule as the head, so a
# RETARGET mid-flight can be modelled independently of a push mid-flight (ersatztv#632).
bases = (state / "pr_bases").read_text().split()
base = bases[min(n, len(bases) - 1)]
body = {
"head": {"sha": sha},
"state": (state / "pr_state").read_text().strip(),
"html_url": "http://gitea.example/timothy/ersatztv/pulls/42",
}
# A 2xx body that merely LOST the field, as distinct from an unreachable PR ('GONE' above).
# This is the shape the `[ -n "$x" ] &&` conjunct used to wave through (ersatztv#778).
if sha == "NOHEAD":
body["head"] = {}
if base != "MISSING":
body["base"] = {"ref": base}
print(json.dumps(body))
sys.exit(0)
# GET /repos/<owner>/<repo>/commits/<sha>/status -> the COMBINED status (latest per context).
# It REPLAYS the status that was actually POSTed, so the script's identify-our-own-write comparison
# is satisfied by construction and tests only have to vary the one field under test: the CREATOR
# Gitea recorded (ersatztv#845). `status_body` overrides the whole response for the shapes a replay
# cannot express — an overwrite, a `statuses: null` body, a malformed one.
if "/commits/" in url and url.split("?")[0].endswith("/status"):
if (state / "status_fail").exists():
sys.exit(22)
raw = state / "status_body"
if raw.exists():
print(raw.read_text())
sys.exit(0)
entry = None
log = state / "posts.jsonl"
if log.exists():
for line in log.read_text().splitlines():
if not line.strip():
continue
post = json.loads(line)
if "/statuses/" in post["url"]:
entry = dict(post["payload"]) # last write wins, as the combined endpoint does
if entry is None:
print(json.dumps({"total_count": 0, "statuses": None}))
sys.exit(0)
who = (state / "status_creator").read_text().strip() if (state / "status_creator").exists() else "timothy"
# 'NULL' is how a status POSTed by an ACTIONS token reads back — the case the gate re-derives.
entry["creator"] = None if who == "NULL" else {"login": who}
# RENAME state -> status. This is the one place the replay must NOT mirror the POST body: a row
# inside `.statuses[]` serialises its state under `status`, while `state` is only the aggregate
# at the top level. Replaying the POST payload verbatim would make this shim agree with the
# script's parser by construction and test nothing — the shape below is what Gitea 1.27.1
# actually returns, measured 2026-08-29.
entry["status"] = entry.pop("state")
# PAGING, modelled from the measured server. Gitea selects each context's MAX row id, orders
# those DESCENDING, then paginates — so the status just POSTed, being the newest id on the head,
# sorts FIRST and is on page 1 whatever the page size. Measured against the live instance: a head
# with ids [17,19,...,41,43] returns 41 and 43 at `?limit=2`, the two highest.
#
# Appending the new row LAST and slicing from the front makes it fall off the page and
# manufactures a truncation the server cannot produce, and tests then get written against that
# fiction. Getting the ORDER right is what makes the paging model faithful;
# `DEFAULT_PAGING_NUM` (30) and the `MAX_RESPONSE_ITEMS` (50) clamp are modelled too, and
# `total_count` reports the PAGE rather than the total, as measured.
filler = int((state / "status_filler").read_text()) if (state / "status_filler").exists() else 0
older = [
{"context": "ci/filler-%d" % i, "status": "success", "description": "f", "creator": None}
for i in range(filler)
]
rows = [entry] + list(reversed(older)) # newest first
query = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
limit = min(int(query.get("limit", ["30"])[0]), 50)
rows = rows[:limit]
print(json.dumps({"state": entry["status"], "total_count": len(rows), "statuses": rows}))
sys.exit(0)
print("{}")
'''
@pytest.fixture
def gitea(tmp_path):
"""A fake Gitea + the env the script needs. Returns a small control/inspection handle."""
bindir = tmp_path / "bin"
bindir.mkdir()
shim = bindir / "curl"
shim.write_text(CURL_SHIM)
shim.chmod(0o755)
state = tmp_path / "state"
state.mkdir()
(state / "pr_shas").write_text(SHA_A)
(state / "pr_bases").write_text("main")
(state / "pr_state").write_text("open")
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["ETV_GITEA_TOKEN"] = "stub-token" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env["ETV_GITEA_REPO"] = "timothy/ersatztv"
env.pop("ETV_GITEA_BASICAUTH", None)
class Handle:
def __init__(self):
self.env = env
self.state = state
def set_head_sequence(self, *shas):
(state / "pr_shas").write_text(" ".join(shas))
def set_pr_state(self, value):
(state / "pr_state").write_text(value)
def fail_posts_to(self, url_substring):
"""Make POSTs whose URL contains this substring fail, as curl -f does on a 4xx/5xx."""
(state / "post_fail").write_text(url_substring)
def set_base_sequence(self, *refs):
"""Base branch per PR GET. 'MISSING' omits `.base` from the response entirely."""
(state / "pr_bases").write_text(" ".join(refs))
def set_status_creator(self, login):
"""Who Gitea records as the author of the read-back status. 'NULL' models an Actions
token, whose statuses carry `creator: null` (ersatztv#742/#845)."""
(state / "status_creator").write_text(login)
def fail_status_readback(self):
"""Make the combined-status GET fail, as curl -f does on a 4xx/5xx."""
(state / "status_fail").write_text("1")
def set_status_filler(self, n):
"""Prepend N unrelated status contexts, so the read-back has to page past them."""
(state / "status_filler").write_text(str(n))
def set_status_body(self, body):
"""Replace the whole combined-status response with this literal JSON text."""
(state / "status_body").write_text(body)
def run(self, *args):
return subprocess.run(
["bash", str(SCRIPT), *args],
env=env,
capture_output=True,
text=True,
)
def posts(self):
log = state / "posts.jsonl"
if not log.exists():
return []
return [json.loads(line) for line in log.read_text().splitlines() if line.strip()]
def statuses(self):
return [p for p in self.posts() if "/statuses/" in p["url"]]
def comments(self):
return [p for p in self.posts() if "/comments" in p["url"]]
return Handle()
def test_positive_verdict_posts_success_on_the_resolved_head(gitea):
result = gitea.run("42", "MERGEABLE")
assert result.returncode == 0, result.stderr
statuses = gitea.statuses()
assert len(statuses) == 1
assert statuses[0]["url"].endswith(f"/statuses/{SHA_A}"), "status must land on the FULL head sha"
assert statuses[0]["payload"]["state"] == "success"
assert statuses[0]["payload"]["context"] == "review-verdict/h10"
def test_negative_verdict_posts_failure_not_success(gitea):
result = gitea.run("42", "BLOCKED", "two findings outstanding")
assert result.returncode == 0, result.stderr
payload = gitea.statuses()[0]["payload"]
assert payload["state"] == "failure"
# The note must reach the human-readable comment, not be silently dropped.
assert "two findings outstanding" in gitea.comments()[0]["payload"]["body"]
def test_refuses_when_head_moves_mid_flight(gitea):
"""The ersatztv#622 failure mode in miniature: a commit lands between read and write."""
gitea.set_head_sequence(SHA_A, SHA_B)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "UNREVIEWED" in result.stderr
assert gitea.statuses() == [], "no status may be written once the reviewed head is stale"
# And no comment either, since ersatztv#792: with the comment written FIRST, a refusal left
# `Review-verdict: MERGEABLE @ <sha>` on the PR with no
# status behind it, which reads to an operator as consent that was never granted.
assert gitea.comments() == [], "a refusal must leave no verdict comment standing in for a status"
def test_never_retargets_the_verdict_at_the_new_head(gitea):
"""Refusing is not enough — it must also not 'helpfully' green the commit that arrived."""
gitea.set_head_sequence(SHA_A, SHA_B)
gitea.run("42", "MERGEABLE")
assert not any(SHA_B in s["url"] for s in gitea.statuses())
@pytest.mark.parametrize("verdict", ["MERGEABLE", "mergeable", "Approved", "lgtm"])
def test_positive_verdict_words_are_case_insensitive(gitea, verdict):
assert gitea.run("42", verdict).returncode == 0
assert gitea.statuses()[0]["payload"]["state"] == "success"
@pytest.mark.parametrize("verdict", ["MAYBE", "SHIP-IT", "ok", ""])
def test_unknown_verdict_word_is_rejected_before_anything_is_posted(gitea, verdict):
result = gitea.run("42", verdict)
assert result.returncode != 0
assert gitea.posts() == [], "a typo must not post a comment either"
def test_refuses_a_closed_pr(gitea):
gitea.set_pr_state("closed")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert gitea.posts() == []
def test_refuses_a_non_numeric_pr_number(gitea):
result = gitea.run("../../etc/passwd", "MERGEABLE")
assert result.returncode != 0
assert gitea.posts() == []
def test_fails_loudly_without_credentials(gitea):
gitea.env.pop("ETV_GITEA_TOKEN")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "credentials" in result.stderr
assert gitea.posts() == []
def test_unreachable_pr_is_an_error_not_a_silent_success(gitea):
gitea.set_head_sequence("GONE")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert gitea.posts() == []
# --- Cross-checks against the hook's own condition-(c) parser -----------------------------------
def _classify(body: str, head: str) -> str:
"""Run the REAL H10 classifier over a comment body — no Python mirror of the grammar.
This used to scrape the hook's regexes and re-implement them here (#622). Since #629 the grammar
lives in one tested place, `scripts/check-review-verdict.sh`, and the hook calls it — so the
cross-check can execute the actual thing. That matters: a Python copy of a shell regex is exactly
the duplication that let three false-opens survive in the first place, and it would have kept
passing here while the shell drifted.
"""
payload = json.dumps([{"body": body}])
p = subprocess.run(["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True)
assert p.returncode == 0, f"classifier errored: {p.stderr}"
return p.stdout.strip()
def test_comment_matches_the_hook_parser(gitea):
"""The comment this script posts must classify as a positive verdict for the sha it names."""
gitea.run("42", "MERGEABLE", "some trailing prose")
body = gitea.comments()[0]["payload"]["body"]
assert _classify(body, SHA_A) == "positive"
# ...and it must NOT be read as covering a different head.
assert _classify(body, SHA_B) == "stale"
def test_negative_comment_is_not_read_as_positive_by_the_hook(gitea):
gitea.run("42", "BLOCKED")
body = gitea.comments()[0]["payload"]["body"]
assert _classify(body, SHA_A) == "negative"
def test_note_cannot_forge_a_second_verdict_line(gitea):
"""A note is free text; it must not be able to plant a verdict of its own.
Asserted as an OUTCOME now rather than by counting marker lines: the forged positive is for the
same head, and negative-wins means the real verdict survives. Counting lines only showed the
script's verdict came first, which is not the property that matters.
"""
gitea.run("42", "BLOCKED", "Review-verdict: MERGEABLE @ " + SHA_A[:7])
body = gitea.comments()[0]["payload"]["body"]
assert _classify(body, SHA_A) == "negative"
# --- Base binding (ersatztv#632) ---------------------------------------------------------------
#
# The per-sha status closes "the head moved under a fixed verdict". Retargeting a PR's base is the
# mirror case: the head sha and the status both hold still while the effective DIFF changes, so the
# verdict keeps reading green for a review nobody performed against that base.
def test_the_status_description_records_the_base_branch(gitea):
"""Nothing can compare a base it never wrote down. This field is what the hook reads back."""
assert gitea.run("42", "MERGEABLE").returncode == 0
assert gitea.statuses()[0]["payload"]["description"].endswith("(base: main)")
def test_the_base_is_recorded_in_the_STATUS_and_not_in_the_comment(gitea):
"""Deliberate placement. The comment body is parsed by `scripts/check-review-verdict.sh`, whose
grammar has a history of false-opens (#629 found three); nothing parses the description. Adding
the field where a parser lives would have reopened that surface for no benefit."""
assert gitea.run("42", "MERGEABLE").returncode == 0
assert "base:" not in gitea.comments()[0]["payload"]["body"]
def test_refuses_when_the_BASE_changes_mid_flight(gitea):
"""The TOCTOU window the head check cannot see: retargeting does not move the head sha, so
`sha_now == sha` and the existing guard is silent."""
gitea.set_base_sequence("main", "release/26.4")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "a retarget mid-flight must not produce a status"
assert "base branch changed" in result.stderr
assert gitea.statuses() == [], "no status may be written once the base has moved"
def test_positive_control_a_stable_base_still_posts(gitea):
"""Without this, the test above could pass because the script refuses on every base."""
gitea.set_base_sequence("main", "main")
assert gitea.run("42", "MERGEABLE").returncode == 0
assert len(gitea.statuses()) == 1
def test_refuses_when_the_pr_has_no_resolvable_base(gitea):
"""A verdict that cannot record what it was formed against is not a verdict this gate can
later re-check, so it fails closed rather than posting an unbindable success."""
gitea.set_base_sequence("MISSING")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert gitea.statuses() == []
def test_a_failed_HEAD_RECHECK_writes_no_status(gitea):
"""Fail-closed on the re-read itself, not just on a moved head.
Left IMPLICIT, this guard is only a side effect: `sha_now=$(api_get ... | jq ...)` aborts under
`set -e` + `pipefail` when the GET fails. Asserted by nothing, folding the head and base
re-reads into one `$(... || true)` variable silently converts it to fail-OPEN — both guards see
an empty string, both no-op, and the status is written having confirmed nothing. Asserted here
so the behaviour is a contract rather than a side effect of a shell option.
"""
gitea.set_head_sequence(SHA_A, "GONE")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert gitea.statuses() == [], (
"a status was written even though the head/base re-read failed — nothing was confirmed"
)
# --- write-side polarity, the half the #774 rescue initially missed ------------------------------
# Kept deliberately in step with POSITIVE_WORDS/NEGATIVE_WORDS in test_check_review_verdict.py.
# The two lists are NOT compared to each other, and this file does not claim they agree — proving
# that needs one shared vocabulary both scripts read, which is ersatztv#788. What each side proves
# is its OWN polarity: that these established tokens still map the way reviewers rely on.
WRITE_POSITIVE = ["MERGEABLE", "APPROVED", "LGTM"]
WRITE_NEGATIVE = ["BLOCKED", "NOT-MERGEABLE"]
@pytest.mark.parametrize(
("word", "expected"),
[(w, "success") for w in WRITE_POSITIVE] + [(w, "failure") for w in WRITE_NEGATIVE],
)
def test_each_verdict_word_posts_its_established_polarity(word, expected, gitea):
"""The WRITE side of the polarity pair.
ersatztv#774 withdrew a test that compared the two verdict vocabularies by parsing shell, and
rescued its disjointness half into test_check_review_verdict.py. That rescue
covered only `check-review-verdict.sh`: an in-memory mutation adding `BLOCKED` to the SUCCESS
arm here produced a write-side overlap the deleted test caught and the rescue did not, because
the rescue never executes this script. That was a real, undisclosed loss and this closes it.
`case` takes the FIRST matching arm, so a token listed in both arms is not ambiguous — it
resolves to whichever comes first, exactly as `is_pos` wins on the read side. The consequence is
the same and it is the one that matters: a word a reviewer means as BLOCKED silently posting
`success` writes a green `review-verdict/h10`, which is the required context branch protection
honours. That flips this assertion.
What this does NOT prove, stated because the parent test was withdrawn for overclaiming: it is
a polarity regression over five established tokens, not a universal disjointness property and
not parity with the read side. A token added to only one script is untested here, not caught.
"""
result = gitea.run("42", word)
assert result.returncode == 0, result.stderr
payload = gitea.statuses()[0]["payload"]
assert payload["state"] == expected, (
f"'{word}' posted state {payload['state']!r}, expected {expected!r}. `case` takes the first "
"matching arm, so a token that has appeared in the other arm resolves there silently — and "
"a BLOCKED verdict posting `success` writes a green required context."
)
@pytest.mark.parametrize(
("head_seq", "base_seq", "field"),
[
(("NOHEAD",), ("main", "main"), "head sha"),
((SHA_A,), ("main", "MISSING"), "base ref"),
],
)
def test_a_reread_that_LOSES_a_field_refuses_instead_of_posting(head_seq, base_seq, field, gitea):
"""The fail-OPEN one level below the TOCTOU guard (ersatztv#778).
Both re-read checks were written as `[ -n "$x" ] && [ "$x" != "$want" ]`. That conjunct makes an
EMPTY value a no-op: a well-formed 2xx response that merely omits `.head.sha` or `.base.ref`
yields an empty variable, neither comparison runs, and the status is posted having confirmed
NOTHING about the head or the base — while the script's whole purpose at that point is to refuse
unless it can confirm. The transport failure one line above was already fatal, which is exactly
what made this shape easy to miss: the loud case was handled and the quiet one was not.
Asserted on the OBSERVABLE outcome — no status written — rather than on message text, so it
still holds if the wording changes.
"""
gitea.set_head_sequence(SHA_A, *head_seq)
gitea.set_base_sequence(*base_seq)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, (
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed nothing about it"
)
assert not gitea.statuses(), (
f"a status was written despite the re-read carrying no {field} — this is the fail-open the -n conjunct created"
)
# --- ersatztv#792: no path may write no status and report success ------------------------------
#
# The issue was filed on an observed "printed the refusal AND exited 0". Re-measured on the tree
# that fixed the re-read fence: every refusal already exits NON-ZERO, and the exit-0 came from the
# caller's pipeline, not from the script. That is worth an executed contract rather than a second
# reading of the source — `die` is one line away from being edited into a `return`, and this file is
# where that would be caught. The parametrisation covers each refusal REASON, not one
# representative, because those paths were added at four different times and only the shared helper
# makes them agree today.
#
# SCOPE, since "every path" would overclaim: these are the eight refusal MODES reachable through the
# real entry point. The source also exits non-zero for a usage error (2), a failing `jq` (its own
# status, under `set -e` + `pipefail`), and a signal (128+n); none of those is a refusal DECISION,
# and only the non-zero-ness is common to all of them.
def _drive(gitea, mode):
if mode == "head-moved":
gitea.set_head_sequence(SHA_A, SHA_B)
elif mode == "reread-failed":
gitea.set_head_sequence(SHA_A, "GONE")
elif mode == "reread-lost-head":
gitea.set_head_sequence(SHA_A, "NOHEAD")
elif mode == "base-retargeted":
gitea.set_base_sequence("main", "some-feature-branch")
elif mode == "reread-lost-base":
gitea.set_base_sequence("main", "MISSING")
elif mode == "first-read-failed":
gitea.set_head_sequence("GONE")
elif mode == "pr-closed":
gitea.set_pr_state("closed")
elif mode == "status-post-failed":
gitea.fail_posts_to("/statuses/")
else: # pragma: no cover - a typo in the parametrisation must not pass silently
raise AssertionError(f"unknown mode {mode}")
return gitea.run("42", "MERGEABLE")
@pytest.mark.parametrize(
"mode",
[
"head-moved",
"reread-failed",
"reread-lost-head",
"base-retargeted",
"reread-lost-base",
"first-read-failed",
"pr-closed",
"status-post-failed",
],
)
def test_every_REFUSAL_MODE_that_writes_no_status_exits_non_zero(gitea, mode):
result = _drive(gitea, mode)
assert gitea.statuses() == [], f"{mode} wrote a status it had no business writing"
assert result.returncode != 0, (
f"{mode} wrote no status and reported SUCCESS — anything checking $? concludes the verdict "
f"posted. stdout={result.stdout!r} stderr={result.stderr!r}"
)
@pytest.mark.parametrize(
"mode",
[
"head-moved",
"reread-failed",
"reread-lost-head",
"base-retargeted",
"reread-lost-base",
"status-post-failed",
],
)
def test_no_refusal_leaves_a_VERDICT_COMMENT_standing_in_for_the_status(gitea, mode):
"""The half-state, which is the part of #792 that was really broken.
The comment is not the gate — `review-verdict/h10` is — but `Review-verdict: MERGEABLE @ <head>`
sitting on a PR reads exactly like consent. Every mode here is one where the status is refused
after the head has been resolved, i.e. every mode that could once have left that comment behind.
"""
_drive(gitea, mode)
assert gitea.comments() == [], f"{mode} left an orphaned verdict comment: {gitea.comments()}"
def test_the_status_is_written_BEFORE_the_comment(gitea):
"""Ordering is the mechanism, so it is asserted rather than described.
Status-then-comment makes the only reachable half-state the safe one: a status with no comment
leaves the merge hook's condition (c) with nothing to classify, which is an `ask`. The reverse
order manufactures the appearance of a granted verdict.
"""
assert gitea.run("42", "MERGEABLE").returncode == 0
urls = [p["url"] for p in gitea.posts()]
assert len(urls) == 2, urls
assert "/statuses/" in urls[0], f"the status must be written first, got {urls}"
assert "/comments" in urls[1], f"the comment must be written second, got {urls}"
def test_a_failed_COMMENT_after_a_written_status_is_still_an_error(gitea):
"""The surviving half-state is safe, not silent: the operator is told to re-run."""
gitea.fail_posts_to("/comments")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert len(gitea.statuses()) == 1, "the status was already written and must not be rolled back"
assert gitea.comments() == []
assert "COMMENT could not be posted" in result.stderr
assert "ask" in result.stderr.lower(), "it must say what the gate will do, not just that a call failed"
# ==================================================================================================
# Is the verdict this tool writes one the GATE will actually inherit? (ersatztv#845)
# ==================================================================================================
#
# `.gitea/workflows/review-verdict.yml` inherits an existing `review-verdict/h10=success` only from a
# status whose `.creator.login` is on its `H10_REVIEWERS` allow-list (ersatztv#742). This script
# posts with whatever account owns the credential in the environment. Those two values were coupled
# with nothing asserting the coupling, and the failure was silent in the worst way: the status is
# written, THIS TOOL REPORTS SUCCESS, and the next PR event re-derives it and posts over it. Forever.
#
# The account below is written as the literal `timothy` on purpose, and it is the same deliberate
# hardcode `test_pr_changed_files.py::test_the_allowlisted_reviewer_in_the_TESTS_is_the_one_the_
# WORKFLOW_ships` exists to pin. Deriving it here would make these tests vacuous: the mutation proof
# works precisely BECAUSE changing the workflow literal moves the allow-list out from under a fixed
# creator. Derive both and they would move together and nothing would ever redden.
ALLOWLISTED = "timothy"
OFF_LIST = "renovate"
def test_a_verdict_posted_by_an_ALLOWLISTED_account_is_accepted(gitea):
"""The accept path — and the mutation proof for the whole derivation.
This is the test `mutation_manifest.py` reddens by rewriting `H10_REVIEWERS` in the SHIPPED
workflow. That single mutation proves two things at once, which is why it is the declared one:
the script must be reading the allow-list LIVE from the gate (a hand-copied list would not
move), and the membership comparison must actually gate the outcome (a no-op comparison would
not care that the list moved).
"""
gitea.set_status_creator(ALLOWLISTED)
result = gitea.run("42", "MERGEABLE")
assert result.returncode == 0, f"the verdict writer REFUSED an allow-listed account: {result.stderr}"
assert len(gitea.statuses()) == 1
assert len(gitea.comments()) == 1, "an accepted verdict must still post its comment"
def test_a_verdict_posted_by_an_account_OFF_the_allowlist_is_refused_before_the_comment(gitea):
"""The whole point of ersatztv#845: this used to report success and deadlock the PR later."""
gitea.set_status_creator(OFF_LIST)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "a verdict the gate will not inherit must not report success"
assert "H10_REVIEWERS" in result.stderr, (
"the diagnostic must name the allow-list, because the repair is to edit it or to re-post "
"with another credential — an operator cannot act on 'refused'"
)
assert OFF_LIST in result.stderr, "the diagnostic must name the account that actually posted"
assert gitea.comments() == [], (
"the refusal must land BEFORE the comment: a verdict comment with no honoured status is the "
"half-state ersatztv#792 removed, and it reads to a human as consent that was never granted"
)
# Stated rather than fixed. The status IS written by the time the author can be measured, and it
# is left standing deliberately — see the residual note in the script.
assert len(gitea.statuses()) == 1
def test_a_status_that_reads_back_with_NO_creator_is_refused(gitea):
"""`creator: null` is how a status POSTed by an ACTIONS token appears, and the gate never
inherits one of those as a reviewer verdict. Distinct from the off-list case: there is no
account to name, so 'add them to the allow-list' would be the wrong advice."""
gitea.set_status_creator("NULL")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "ACTIONS token" in result.stderr
assert gitea.comments() == []
def test_a_status_OVERWRITTEN_between_the_write_and_the_read_is_not_blamed_on_its_author(gitea):
"""`/commits/{sha}/status` returns the LATEST status per context. If the gate overwrote ours
between the POST and the read-back, judging THAT status's author would accuse the wrong party —
and an Actions write carries no creator, so it would surface as a credential problem that does
not exist. Identify our own write first, then judge it."""
gitea.set_status_body(
json.dumps(
{
"total_count": 1,
"statuses": [
{
"context": "review-verdict/h10",
"status": "pending",
"description": "Awaiting review verdict",
"creator": None,
}
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "it was overwritten" in result.stderr
assert "ACTIONS token" not in result.stderr, (
"an overwritten status must not be reported as a wrong-credential problem"
)
assert gitea.comments() == []
def test_an_unreadable_readback_refuses_rather_than_reporting_the_verdict_posted(gitea):
"""UNKNOWN must fail. A read that cannot be performed leaves it unproven that the verdict is in
force, and reporting success on unproven is exactly the ersatztv#845 shape."""
gitea.fail_status_readback()
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
# Same reason as above: `set -e` plus the absent-entry check already refuse, so without naming
# the message this test passed with the explicit `|| die` removed.
assert "could not be read back" in result.stderr
assert gitea.comments() == []
def test_a_readback_whose_statuses_array_is_NULL_is_refused(gitea):
"""Gitea serialises a nil slice as `null`, not `[]` (ersatztv#751). The script tests the array
TYPE rather than assuming one; a body that merely lost its array must refuse, not be read as
'no verdict, carry on'."""
gitea.set_status_body(json.dumps({"total_count": 0, "statuses": None}))
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
# The DIAGNOSTIC, not merely a non-zero exit. Asserting only the exit code let this pass on the
# downstream absent-entry check, so the array-TYPE clause it names could be disarmed with the
# suite still green — an outcome the script produces anyway is not evidence about a clause.
assert "carries no `.statuses` ARRAY" in result.stderr
assert gitea.comments() == []
# --- The allow-list is DERIVED from the gate, not restated beside it. ----------------------------
def _sandbox(tmp_path, reviewers):
"""A throwaway tree carrying the real `scripts/` and the real gate workflow, with the workflow's
`H10_REVIEWERS` literal rewritten. Returns a runner over the SANDBOX copy of the script.
The whole `scripts/` directory is copied rather than the three files in play, because the script
resolves both of its libraries relative to `${BASH_SOURCE[0]}` — copying selectively would leave
it sourcing the real repo's copies and the rewrite would prove nothing.
"""
root = tmp_path / "sandbox"
shutil.copytree(REPO_ROOT / "scripts", root / "scripts")
workflow = root / ".gitea" / "workflows" / "review-verdict.yml"
workflow.parent.mkdir(parents=True)
source = (REPO_ROOT / ".gitea" / "workflows" / "review-verdict.yml").read_text()
rewritten, n = re.subn(
r'^(\s*)H10_REVIEWERS="[^"]*"$',
lambda m: f'{m.group(1)}H10_REVIEWERS="{reviewers}"',
source,
flags=re.MULTILINE,
)
assert n == 1, f"expected exactly one H10_REVIEWERS assignment to rewrite, substituted {n}"
workflow.write_text(rewritten)
return root / "scripts" / "post-review-verdict.sh"
def test_the_allowlist_MOVES_when_the_workflow_moves(gitea, tmp_path):
"""The single-declaration property, measured end to end rather than argued.
Rewrite the gate's own literal to an account this repo does not have, drive the REAL script, and
the accounts swap roles: the one the shipped workflow accepts is refused, and the invented one is
accepted. A copy of the list living beside the script could not produce that.
"""
invented = "not-a-real-account"
script = _sandbox(tmp_path, invented)
def run_with(creator):
gitea.set_status_creator(creator)
(gitea.state / "posts.jsonl").unlink(missing_ok=True)
(gitea.state / "pr_get_count").unlink(missing_ok=True)
return subprocess.run(
["bash", str(script), "42", "MERGEABLE"],
env=gitea.env,
capture_output=True,
text=True,
)
accepted = run_with(invented)
assert accepted.returncode == 0, f"the rewritten allow-list was not the one used: {accepted.stderr}"
refused = run_with(ALLOWLISTED)
assert refused.returncode != 0, (
f"{ALLOWLISTED} was accepted against a workflow that no longer lists them, so the script is "
"not deriving the allow-list from the gate"
)
assert invented in refused.stderr
def test_a_workflow_whose_allowlist_cannot_be_DERIVED_refuses_before_any_write(gitea, tmp_path):
"""Fail-closed, and early. Two assignments means the shell would run the last one executed, so
which value the gate uses is unknowable from here — and a writer that cannot tell whether its
verdict is inheritable must not post one. Nothing is written at all, so there is no half-state
to clean up.
"""
script = _sandbox(tmp_path, ALLOWLISTED)
text = script.parent.parent.joinpath(".gitea/workflows/review-verdict.yml")
body = text.read_text()
marker = f'H10_REVIEWERS="{ALLOWLISTED}"'
assert body.count(marker) == 1
text.write_text(body.replace(marker, f'{marker}\n H10_REVIEWERS="someone-else"'))
result = subprocess.run(
["bash", str(script), "42", "MERGEABLE"],
env=gitea.env,
capture_output=True,
text=True,
)
assert result.returncode != 0
assert gitea.statuses() == [], "a derivation it cannot trust must refuse before writing anything"
assert gitea.comments() == []
def test_a_VERBATIM_live_gitea_status_body_is_accepted(gitea):
"""Pins the real response shape, independently of the shim that serves it.
The shim replays what was POSTed, so it can only ever disagree with the parser about a field it
is told to rename — which is a fixture agreeing with the code by construction. This body is the
shape `GET /repos/{o}/{r}/commits/{sha}/status` actually returned on Gitea 1.27.1 (measured
2026-08-29 against a real `review-verdict/h10` row), trimmed of fields nothing reads. The key
that matters is `status`: a row inside `.statuses[]` carries its state under that name, while
`state` is only the top-level AGGREGATE. Parsing `.state` per row yields "" for every
well-formed response, refuses every verdict, and deadlocks the repo — no PR merges without one.
"""
short = SHA_A[:7]
gitea.set_status_body(
json.dumps(
{
"sha": SHA_A,
"state": "success", # the AGGREGATE, deliberately present and deliberately not read
"total_count": 2,
"statuses": [
{
"id": 32,
"status": "success",
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
"context": "review-verdict/h10",
"creator": {"id": 1, "login": ALLOWLISTED, "username": ALLOWLISTED},
"created_at": "2026-08-29T21:40:22+02:00",
},
{
"id": 31,
"status": "success",
"description": "Build & test",
"context": "ci/build",
"creator": None,
},
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode == 0, f"the verdict writer REFUSED a verbatim live Gitea response: {result.stderr}"
assert len(gitea.comments()) == 1
def test_a_status_row_carrying_NO_state_field_says_so_instead_of_blaming_a_race(gitea):
"""A shape it cannot read is not an overwrite. Stating a cause that did not happen is its own
defect class here (#859) — 'something overwrote it' would send the reader hunting a race that
never occurred, on what is really a Gitea-version problem."""
short = SHA_A[:7]
gitea.set_status_body(
json.dumps(
{
"total_count": 1,
"statuses": [
{
"context": "review-verdict/h10",
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
"creator": {"login": ALLOWLISTED},
}
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "no state field" in result.stderr
assert "overwrote it" not in result.stderr, (
"an unreadable response shape must not be reported as a race that did not happen"
)
assert gitea.comments() == []
def test_a_readback_MISSING_our_context_reports_the_row_count_and_names_no_cause(gitea):
"""The row this run wrote is the newest on the head and sorts first, so its absence means
something removed or replaced it — the diagnostic reports how many rows it read and does not
guess at which. Paging is deliberately NOT offered as a cause: `total_count` reports the page
rather than the total, which suggests it, but the selection ORDER — max id per context,
descending — makes that truncation unproducible for this row.
This also witnesses the row count, which the empty-body case cannot reach."""
gitea.set_status_body(
json.dumps(
{
"total_count": 2,
"statuses": [
{"context": "ci/build", "status": "success", "creator": None},
{"context": "ci/lint", "status": "success", "creator": None},
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "not among the 2 status rows" in result.stderr, (
"the row count it actually read must be in the message, or a paging overflow is unreadable"
)
assert gitea.comments() == []
# --- The derivation library itself. --------------------------------------------------------------
#
# Driven directly rather than through the script, because three of its clauses cannot be reached from
# the script at all: the script only ever runs against the real workflow, which is well-formed. Each
# of these was measured to survive a disarm of its clause when only the end-to-end tests existed.
LIB = REPO_ROOT / "scripts" / "lib" / "h10-reviewers.sh"
def _lib(snippet, cwd, workflow_body=None):
"""Source the real library and run `snippet`. Returns the CompletedProcess.
`cwd` matters: pathname expansion is resolved against it, which is the whole point of the glob
test below. A workflow body, when given, is written to `cwd/wf.yml`.
"""
if workflow_body is not None:
(cwd / "wf.yml").write_text(workflow_body)
script = f'set -euo pipefail\n. "{LIB}"\n{snippet}\n'
return subprocess.run(["bash", "-c", script], cwd=str(cwd), capture_output=True, text=True)
def test_a_GLOB_entry_is_rejected_rather_than_expanded_against_the_working_directory(tmp_path):
"""The documented fail-open, one file over from where it is recorded.
`for x in $list` and `$(printf %s "$list")` both perform PATHNAME EXPANSION, so a `*` entry
expands to the FILENAMES around it — and any of those matching the plain-login class would
validate cleanly and enter the allow-list as a real reviewer. `review-verdict-vocabulary.sh`
records this happening to its own word list. The cwd below is stocked with names that would
validate, so an expanding implementation loads a non-empty list instead of refusing.
"""
for name in ("timothy", "renovate", "attacker"):
(tmp_path / name).write_text("")
result = _lib(
'etv_h10_reviewers_load "$PWD/wf.yml" && echo "LOADED=[$ETV_H10_REVIEWERS]"',
tmp_path,
' H10_REVIEWERS="*"\n',
)
assert result.returncode != 0, (
f"a `*` entry was accepted; it expanded to {result.stdout.strip()!r} in a directory "
"containing plausible login names"
)
assert "LOADED=" not in result.stdout
assert "glob metacharacters" in result.stderr
def test_an_EMPTY_or_whitespace_only_allowlist_is_rejected(tmp_path):
"""`" "` is a non-empty STRING that splits to nothing, so emptiness is checked on the split
count. An empty allow-list is not a harmless default at either end: the gate's `for rv in ""`
iterates zero times, and here every account would be judged not-a-member."""
for body in (' H10_REVIEWERS=""\n', ' H10_REVIEWERS=" "\n'):
result = _lib('etv_h10_reviewers_load "$PWD/wf.yml"', tmp_path, body)
assert result.returncode != 0, f"accepted an empty allow-list from {body!r}"
assert "EMPTY" in result.stderr
def test_membership_asked_before_a_successful_load_answers_CANNOT_TELL_not_NOT_A_MEMBER(tmp_path):
"""Three outcomes, not two. Folding "cannot tell" into "not a member" would make a broken
checkout indistinguishable from a wrong credential, and the script would then tell the operator
to fix their credential — a cause that did not happen. Exit 2 is what keeps them apart."""
result = _lib('set +e; etv_h10_reviewers_contains timothy; echo "RC=$?"', tmp_path)
assert "RC=2" in result.stdout, f"expected cannot-tell (2), got {result.stdout.strip()!r}"
# And after a FAILED load, not merely before any load at all.
result = _lib(
'set +e; etv_h10_reviewers_load "$PWD/nonexistent.yml"; etv_h10_reviewers_contains timothy; echo "RC=$?"',
tmp_path,
)
assert "RC=2" in result.stdout, f"a failed load must not leave the list queryable: {result.stdout!r}"
def test_a_successful_load_answers_member_and_non_member_apart(tmp_path):
"""The positive control for the three-outcome contract above: 0 and 1 must both be reachable,
or the exit-2 assertion could be passing because the function only ever returns 2."""
result = _lib(
'etv_h10_reviewers_load "$PWD/wf.yml"; set +e; '
'etv_h10_reviewers_contains alice; echo "ALICE=$?"; '
'etv_h10_reviewers_contains carol; echo "CAROL=$?"',
tmp_path,
' H10_REVIEWERS="alice bob"\n',
)
assert "ALICE=0" in result.stdout, result.stdout
assert "CAROL=1" in result.stdout, result.stdout
def test_sourcing_the_library_does_not_disturb_the_callers_globbing_setting(tmp_path):
"""The membership split turns pathname expansion off and must put it back exactly as found —
`post-review-verdict.sh` runs under `set -euo pipefail` with globbing ON and interpolates API
values into messages afterwards."""
(tmp_path / "wf.yml").write_text(' H10_REVIEWERS="alice"\n')
on = _lib(
'etv_h10_reviewers_load "$PWD/wf.yml"; set +e; etv_h10_reviewers_contains alice; '
'case "$-" in *f*) echo "GLOB=off" ;; *) echo "GLOB=on" ;; esac',
tmp_path,
)
assert "GLOB=on" in on.stdout, on.stdout
off = _lib(
'set -f; etv_h10_reviewers_load "$PWD/wf.yml"; set +e; etv_h10_reviewers_contains alice; '
'case "$-" in *f*) echo "GLOB=off" ;; *) echo "GLOB=on" ;; esac',
tmp_path,
)
assert "GLOB=off" in off.stdout, off.stdout
def test_the_verdict_row_is_found_however_many_other_contexts_the_head_carries(gitea):
"""A POSITIVE CONTROL over a full page, not a property pin — said plainly because the obvious
reading is wrong.
The shim places our row at index 0 unconditionally (it models Gitea's real ordering: max id per
context, DESCENDING, and our just-POSTed row is the newest). So this cannot fail for the reason
its name suggests — deleting `?limit=100` leaves it green, and it would stay green against a
naive `.statuses[0]` implementation too. What it does check is that `map(select(...))` still
finds our context when the page is full of other rows.
It replaces a test that asserted the OPPOSITE — that with 40 other contexts the row was reachable
only because of `?limit=100` — which was an artifact of a shim that appended the new row LAST and
sliced from the front, manufacturing a truncation the server cannot produce. `?limit=100` has no
observable effect under the real ordering and is declared unwitnessed insurance in the script.
"""
gitea.set_status_filler(80)
result = gitea.run("42", "MERGEABLE")
assert result.returncode == 0, (
f"the verdict row was not found behind 80 other contexts, though it is the newest on the "
f"head and should sort first: {result.stderr}"
)
assert len(gitea.comments()) == 1
def test_TWO_rows_for_our_context_are_ambiguous_rather_than_resolved_by_taking_the_first(gitea):
"""`[0]` would pick a row and judge ITS author while a different row for the same context is the
one actually standing. Unreachable against Gitea as measured (the combined endpoint returns the
latest row per context), so this guards the property rather than an observed shape — which is
why the message says so instead of blaming a race."""
short = SHA_A[:7]
desc = f"Review-verdict: MERGEABLE @ {short} (base: main)"
gitea.set_status_body(
json.dumps(
{
"total_count": 2,
"statuses": [
{
"context": "review-verdict/h10",
"status": "success",
"description": desc,
"creator": {"login": ALLOWLISTED},
},
{
"context": "review-verdict/h10",
"status": "success",
"description": desc,
"creator": {"login": OFF_LIST},
},
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "an allow-listed FIRST row let a second row with an off-list creator through"
assert "2 separate rows" in result.stderr
assert gitea.comments() == []
def test_an_EMPTY_response_body_is_refused_as_a_SHAPE_problem(gitea):
"""An empty body yields no `.statuses` array, so it must refuse on the shape branch.
This does NOT witness the row-count fallback: an empty body exits
at the array-TYPE branch and never reaches the count. The count IS witnessed, by
`test_a_readback_MISSING_our_context_...`, which supplies a real array with no matching row. That
fallback was dead code and is gone — `.statuses` is
known to be an array by the time the count runs.
"""
gitea.set_status_body("")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "carries no `.statuses` ARRAY" in result.stderr
assert gitea.comments() == []
def test_the_split_pins_IFS_so_a_callers_separator_cannot_merge_or_split_entries(tmp_path):
"""Globbing is only half of what an unquoted expansion depends on; the other half is the ambient
`IFS`. With `IFS=-` a caller made `renovate` read as a member of the single entry `renovate-bot`
— measured against the real predecessor, not a hand-written mutant.
The mechanism is the split inside `etv_h10_reviewers_contains`, NOT the one in
`etv_h10_reviewers_load`: `load` stores the RAW string, so its own split can only affect
validation, never membership. Both are pinned, but only the `contains` one is witnessed by this
test, and that is the honest split of the two. Not reachable from the script today (bash resets
IFS at startup and the script never assigns it) — this pins the hardening the library's header
invites by inviting reuse."""
(tmp_path / "wf.yml").write_text(' H10_REVIEWERS="renovate-bot"\n')
result = _lib(
'IFS="-"; etv_h10_reviewers_load "$PWD/wf.yml"; set +e; '
'etv_h10_reviewers_contains renovate; echo "SPLIT=$?"; '
'etv_h10_reviewers_contains renovate-bot; echo "WHOLE=$?"',
tmp_path,
)
assert "SPLIT=1" in result.stdout, f"`renovate` read as a member under IFS=-: {result.stdout!r}"
assert "WHOLE=0" in result.stdout, result.stdout
def _standing_status(gitea, *, state, description):
"""A read-back carrying one `review-verdict/h10` row with exactly these two fields."""
gitea.set_status_body(
json.dumps(
{
"total_count": 1,
"statuses": [
{
"context": "review-verdict/h10",
"status": state,
"description": description,
"creator": {"login": ALLOWLISTED},
}
],
}
)
)
def test_a_row_differing_ONLY_in_state_is_refused(gitea):
"""Witnesses the STATE arm on its own.
`test_a_status_OVERWRITTEN_...` changes state AND description, so either arm could be deleted
with the suite green — the two overlap and mask each other (#685). Each arm therefore gets a
fixture that differs in exactly one field. The creator here is ALLOW-LISTED on purpose: the
refusal must come from identifying the row as not-ours, not from its author.
"""
short = SHA_A[:7]
_standing_status(gitea, state="pending", description=f"Review-verdict: MERGEABLE @ {short} (base: main)")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "a row whose STATE is not the one just written was accepted"
assert "it was overwritten" in result.stderr
assert gitea.comments() == []
def test_a_row_differing_ONLY_in_description_is_refused(gitea):
"""Witnesses the DESCRIPTION arm on its own — the one that actually identifies our write, since
it carries the verdict word, the short sha and the base."""
_standing_status(gitea, state="success", description="Review-verdict: MERGEABLE @ deadbee (base: main)")
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "a row whose DESCRIPTION names a different head was accepted as this run's write"
assert "it was overwritten" in result.stderr
assert gitea.comments() == []
def test_an_off_list_account_may_still_record_a_REJECTION(gitea):
"""The allow-list applies to `success` ONLY, because that is what the gate does.
`review-verdict.yml` short-circuits on an existing `success` only for an allow-listed creator,
but on an existing `failure` for any ATTRIBUTABLE one — inheriting a rejection can only withhold
an exemption, while re-deriving one can turn it green. Enforcing membership on a `failure` here
refused a verdict the gate honours, told the reviewer their rejection would be re-derived when it
would not, and left a real reviewer no supported way to record a rejection at all.
"""
gitea.set_status_creator(OFF_LIST)
result = gitea.run("42", "BLOCKED", "two findings outstanding")
assert result.returncode == 0, (
f"an off-list account could not record a BLOCKED verdict, which the gate honours: {result.stderr}"
)
assert gitea.statuses()[0]["payload"]["state"] == "failure"
assert len(gitea.comments()) == 1, "a rejection must still get its human-readable comment"
assert "two findings outstanding" in gitea.comments()[0]["payload"]["body"]
def test_an_off_list_account_still_cannot_record_an_APPROVAL(gitea):
"""The other side of the same asymmetry — the positive direction still requires membership, so
the fix above did not widen the check into a no-op."""
gitea.set_status_creator(OFF_LIST)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert "H10_REVIEWERS" in result.stderr
assert gitea.comments() == []
def test_a_row_carrying_ONLY_the_top_level_state_key_is_refused(gitea):
"""No `.state` fallback. The gate reads `.status` and nothing else, so tolerating a shape it
cannot read would make this tool report success on a verdict the gate re-derives — ersatztv#845,
recreated by the check meant to prevent it. On a shape neither understands, refusing is correct.
"""
short = SHA_A[:7]
gitea.set_status_body(
json.dumps(
{
"total_count": 1,
"statuses": [
{
"context": "review-verdict/h10",
"state": "success", # the key the GATE does not read
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
"creator": {"login": ALLOWLISTED},
}
],
}
)
)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, "a row the gate cannot read was accepted, which is the ersatztv#845 stall re-created"
assert "no state field" in result.stderr
assert gitea.comments() == []
def test_an_ENVIRONMENT_variable_cannot_steer_which_workflow_the_allowlist_comes_from(gitea, tmp_path):
"""A declared security property that had no guard: `scripts/lib/h10-reviewers.sh` resolves the
gate definition from its OWN location and offers no `ETV_*_WORKFLOW` override, because an env var
choosing which tree answers a security question is an unsound input (ersatztv#858 is the same
shape one file over).
It held, but nothing pinned it — and restoring the override the way a refactor naturally would
(`ETV_H10_REVIEWERS_WORKFLOW="${ETV_H10_REVIEWERS_WORKFLOW:-}"`) left the whole suite green while
an environment variable installed an arbitrary allow-list and an off-list account was accepted as
a reviewer. That is the same class this branch removed elsewhere, in a clause labelled as a
deliberate decision rather than as untested.
"""
hostile = tmp_path / "hostile.yml"
hostile.write_text(f' H10_REVIEWERS="{OFF_LIST}"\n')
env = dict(gitea.env)
env["ETV_H10_REVIEWERS_WORKFLOW"] = str(hostile)
gitea.set_status_creator(OFF_LIST)
result = subprocess.run(["bash", str(SCRIPT), "42", "MERGEABLE"], env=env, capture_output=True, text=True)
assert result.returncode != 0, (
f"an environment variable steered the allow-list: {OFF_LIST} was accepted as a reviewer"
)
assert OFF_LIST in result.stderr and "H10_REVIEWERS" in result.stderr
assert gitea.comments() == []
def test_a_REJECTION_survives_a_workflow_whose_allowlist_cannot_be_derived(gitea, tmp_path):
"""Membership never applies to a `failure`, so a broken `H10_REVIEWERS` declaration must not
block one. It used to: the derivation was loaded unconditionally, so a workflow with two
assignments refused REJECTIONS as well — the exact outcome the success-only rule exists to avoid
("no supported way to record a rejection"), one condition earlier, and on the branch most likely
to have broken that declaration: the one editing it.
"""
script = _sandbox(tmp_path, ALLOWLISTED)
workflow = script.parent.parent / ".gitea" / "workflows" / "review-verdict.yml"
body = workflow.read_text()
marker = f'H10_REVIEWERS="{ALLOWLISTED}"'
assert body.count(marker) == 1
workflow.write_text(body.replace(marker, f'{marker}\n H10_REVIEWERS="someone-else"'))
gitea.set_status_creator(OFF_LIST)
result = subprocess.run(
["bash", str(script), "42", "BLOCKED", "findings outstanding"],
env=gitea.env,
capture_output=True,
text=True,
)
assert result.returncode == 0, f"a rejection was blocked by an allow-list it does not depend on: {result.stderr}"
assert len(gitea.comments()) == 1
# And the positive direction still refuses on the same tree, so the guard was narrowed, not removed.
(gitea.state / "posts.jsonl").unlink(missing_ok=True)
(gitea.state / "pr_get_count").unlink(missing_ok=True)
approval = subprocess.run(["bash", str(script), "42", "MERGEABLE"], env=gitea.env, capture_output=True, text=True)
assert approval.returncode != 0
assert gitea.statuses() == [], "an underivable allow-list must refuse an APPROVAL before writing"