Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 35s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 57s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 37s
PR Gates / Docs update reminder (pull_request) Successful in 1m0s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 17s
review-verdict/h10 Review-verdict: MERGEABLE @ a7d91bf (base: main)
Review verdict / Set review-verdict status (pull_request_target) Successful in 45s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m25s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m17s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 19m27s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m4s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
`docs.no-session-narrative` reaches every durable artifact, but its detector scanned only `docs/**/*.md` and root markdown, and nothing had ever swept the rest. The issue named four sites from one grep and called them a floor. Deriving the population instead — a whitespace-joined sweep over every tracked file outside the detector, for the detector's own phrasings plus the attribution and review-round class #812 found — gave 453 sites in 108 files at `fb5592971`, and a second pass for phrasings the first list missed (hyphenated `round-N`, "an earlier version", "the reviewer proved") added residuals in the same files. Every site was classified with #812's three dispositions (CUT / SEVER / KEEP with its sub-kind) under the who-benefits test; the per-site manifests are on the PR. The rejected designs, tested-and-rejected fixtures, measurements and traps stay; the attribution of who found them and the round in which they were found go. The detector's population grows to `.claude/`, `.gitea/`, `.husky/` and `scripts/` regardless of extension, minus the detector and its own test (whose fixtures ARE the phrasings) and minus `scripts/tests/fixtures/` (test data, including decision-record copies — the same reasoning as the records' own exemption, and what keeps the record's depth measurement true), and `--all` lists tracked REGULAR files only — a symlink's content is its target and a gitlink has none. The #812 argument for leaving `docs/superpowers/**` in the population runs the other way here: `--diff` sees only ADDED lines, and 287 of the 453 sites were under 30 days old — this corpus is where narrative is being added, so the advisory nudge has reach. Density agrees: 56 line-mode hits over the 113 regular files the predicate admits, against 9 over 66 docs files before #812. `web/` and C# stay out on the same measurement (3 of 74 PATTERNS-matching sites, ~4,600 files). The predicate did not grow: PATTERNS matched 74 of 453 sites, and widening the word list to the attribution class is the treadmill the withdrawn parity test ran on. The population oracle is restated over segments with the new arms, the synthetic cross product gains the process heads and non-markdown extensions, a fixture witnesses that a tracked symlink is neither scanned nor counted, a `.py.bak` axis separates a by-name exemption from a `startswith` over the same tuple, and eight mutants (drop the process arm, drop the by-name exemption, exempt by `startswith`, drop or add a prefix, drop the fixtures exemption, list only markdown, drop the symlink filter, test the mode per row instead of per path) each redden it. A pre-existing silent drop in `--diff` goes with it: git tab-terminates a `+++` filename that contains a space, and the kept tab made `is_scanned_path` refuse the file with no notice — fixed, with a positive control and its own mutant. Code is unchanged by construction, measured per file type against `origin/main`: Python modules are AST-equal with docstrings stripped, except `#` lines inside the embedded fixture programs (string literals) of three test modules; workflows differ only in `#` lines inside `run:` block scalars; shell, C#, TypeScript and jq are equal with comment lines stripped. The stated exceptions: the detector and its test, 26 vitest titles that carried review-round or severity labels or a reviewer attribution (call sites whose title changed — every changed title line walked back to its `it(` / `it.each(...)(` anchor, so a `' + '` concatenation counts once), two registry note strings and the mutation manifest's prose fields. scripts/tests: 1565 passed. Web: lint, typecheck, 1319 tests green. Closes #876. Decisions-Edit: yes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEcBoFw7ctrf3Nb7R7x7wk
1283 lines
59 KiB
Python
1283 lines
59 KiB
Python
"""Tests for `scripts/post-review-verdict.sh` (ersatztv#622).
|
|
|
|
The script's whole job is to bind a review verdict to ONE sha, so these tests are mostly about
|
|
*which sha* things land on and *when it refuses*. Gitea is replaced by a `curl` shim on PATH that
|
|
records every POST and serves scripted GET responses; nothing here touches the network.
|
|
|
|
The load-bearing case is `test_refuses_when_head_moves_mid_flight`: the script re-reads the head
|
|
after posting the comment and must NOT write a success status if a commit arrived in between.
|
|
Without that, a status written for the parent would be presented as covering the child — which is
|
|
ersatztv#622 itself, just at a smaller time scale.
|
|
|
|
`test_comment_matches_the_hook_parser` is a cross-check rather than a unit test: it runs the comment
|
|
body this script produces through the *actual* H10 classifier the hook uses for condition (c),
|
|
`scripts/check-review-verdict.sh`. A drift between the two would be invisible until a merge
|
|
mysteriously stalled.
|
|
|
|
It originally re-implemented the hook's regexes in Python and asserted the shell source still
|
|
contained them. #629 removed that mirror: three false-opens had survived precisely because the
|
|
grammar existed in two places, and a Python copy would have kept passing while the shell drifted.
|
|
The grammar now lives in one tested script, so this calls it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
SCRIPT = REPO_ROOT / "scripts" / "post-review-verdict.sh"
|
|
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
|
CLASSIFIER = REPO_ROOT / "scripts" / "check-review-verdict.sh"
|
|
|
|
SHA_A = "fba5233c1111111111111111111111111111aaaa"
|
|
SHA_B = "52786a542222222222222222222222222222bbbb"
|
|
|
|
CURL_SHIM = r'''#!/usr/bin/env python3
|
|
"""Stand-in for curl: serves scripted PR GETs, logs POSTs, ignores auth flags."""
|
|
import json, os, sys, pathlib, urllib.parse
|
|
|
|
state = pathlib.Path(os.environ["STUB_DIR"])
|
|
args = sys.argv[1:]
|
|
url = [a for a in args if a.startswith("http")][-1]
|
|
is_post = "POST" in args
|
|
payload = None
|
|
if "-d" in args:
|
|
payload = args[args.index("-d") + 1]
|
|
|
|
if is_post:
|
|
# A POST can be scripted to fail (`post_fail` holds a URL substring), so the two write paths
|
|
# can be broken independently — the shape ersatztv#792 is about.
|
|
fail_on = (state / "post_fail").read_text().strip() if (state / "post_fail").exists() else ""
|
|
if fail_on and fail_on in url:
|
|
sys.exit(22)
|
|
with (state / "posts.jsonl").open("a") as fh:
|
|
fh.write(json.dumps({"url": url, "payload": json.loads(payload)}) + "\n")
|
|
print("{}")
|
|
sys.exit(0)
|
|
|
|
# GET /repos/<owner>/<repo>/pulls/<n> -> consume the next scripted head sha.
|
|
if "/pulls/" in url and not url.endswith("/files"):
|
|
shas = (state / "pr_shas").read_text().split()
|
|
counter = state / "pr_get_count"
|
|
n = int(counter.read_text()) if counter.exists() else 0
|
|
counter.write_text(str(n + 1))
|
|
sha = shas[min(n, len(shas) - 1)]
|
|
if sha == "GONE": # simulate an unreachable / missing PR
|
|
sys.exit(22)
|
|
# The base branch is scripted on the same consume-one-per-GET schedule as the head, so a
|
|
# RETARGET mid-flight can be modelled independently of a push mid-flight (ersatztv#632).
|
|
bases = (state / "pr_bases").read_text().split()
|
|
base = bases[min(n, len(bases) - 1)]
|
|
body = {
|
|
"head": {"sha": sha},
|
|
"state": (state / "pr_state").read_text().strip(),
|
|
"html_url": "http://gitea.example/timothy/ersatztv/pulls/42",
|
|
}
|
|
# A 2xx body that merely LOST the field, as distinct from an unreachable PR ('GONE' above).
|
|
# This is the shape the `[ -n "$x" ] &&` conjunct used to wave through (ersatztv#778).
|
|
if sha == "NOHEAD":
|
|
body["head"] = {}
|
|
if base != "MISSING":
|
|
body["base"] = {"ref": base}
|
|
print(json.dumps(body))
|
|
sys.exit(0)
|
|
|
|
# GET /repos/<owner>/<repo>/commits/<sha>/status -> the COMBINED status (latest per context).
|
|
# It REPLAYS the status that was actually POSTed, so the script's identify-our-own-write comparison
|
|
# is satisfied by construction and tests only have to vary the one field under test: the CREATOR
|
|
# Gitea recorded (ersatztv#845). `status_body` overrides the whole response for the shapes a replay
|
|
# cannot express — an overwrite, a `statuses: null` body, a malformed one.
|
|
if "/commits/" in url and url.split("?")[0].endswith("/status"):
|
|
if (state / "status_fail").exists():
|
|
sys.exit(22)
|
|
raw = state / "status_body"
|
|
if raw.exists():
|
|
print(raw.read_text())
|
|
sys.exit(0)
|
|
entry = None
|
|
log = state / "posts.jsonl"
|
|
if log.exists():
|
|
for line in log.read_text().splitlines():
|
|
if not line.strip():
|
|
continue
|
|
post = json.loads(line)
|
|
if "/statuses/" in post["url"]:
|
|
entry = dict(post["payload"]) # last write wins, as the combined endpoint does
|
|
if entry is None:
|
|
print(json.dumps({"total_count": 0, "statuses": None}))
|
|
sys.exit(0)
|
|
who = (state / "status_creator").read_text().strip() if (state / "status_creator").exists() else "timothy"
|
|
# 'NULL' is how a status POSTed by an ACTIONS token reads back — the case the gate re-derives.
|
|
entry["creator"] = None if who == "NULL" else {"login": who}
|
|
# RENAME state -> status. This is the one place the replay must NOT mirror the POST body: a row
|
|
# inside `.statuses[]` serialises its state under `status`, while `state` is only the aggregate
|
|
# at the top level. Replaying the POST payload verbatim would make this shim agree with the
|
|
# script's parser by construction and test nothing — the shape below is what Gitea 1.27.1
|
|
# actually returns, measured 2026-08-29.
|
|
entry["status"] = entry.pop("state")
|
|
# PAGING, modelled from the measured server. Gitea selects each context's MAX row id, orders
|
|
# those DESCENDING, then paginates — so the status just POSTed, being the newest id on the head,
|
|
# sorts FIRST and is on page 1 whatever the page size. Measured against the live instance: a head
|
|
# with ids [17,19,...,41,43] returns 41 and 43 at `?limit=2`, the two highest.
|
|
#
|
|
# Appending the new row LAST and slicing from the front makes it fall off the page and
|
|
# manufactures a truncation the server cannot produce, and tests then get written against that
|
|
# fiction. Getting the ORDER right is what makes the paging model faithful;
|
|
# `DEFAULT_PAGING_NUM` (30) and the `MAX_RESPONSE_ITEMS` (50) clamp are modelled too, and
|
|
# `total_count` reports the PAGE rather than the total, as measured.
|
|
filler = int((state / "status_filler").read_text()) if (state / "status_filler").exists() else 0
|
|
older = [
|
|
{"context": "ci/filler-%d" % i, "status": "success", "description": "f", "creator": None}
|
|
for i in range(filler)
|
|
]
|
|
rows = [entry] + list(reversed(older)) # newest first
|
|
query = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
|
|
limit = min(int(query.get("limit", ["30"])[0]), 50)
|
|
rows = rows[:limit]
|
|
print(json.dumps({"state": entry["status"], "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
|
|
print("{}")
|
|
'''
|
|
|
|
|
|
@pytest.fixture
|
|
def gitea(tmp_path):
|
|
"""A fake Gitea + the env the script needs. Returns a small control/inspection handle."""
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
shim = bindir / "curl"
|
|
shim.write_text(CURL_SHIM)
|
|
shim.chmod(0o755)
|
|
|
|
state = tmp_path / "state"
|
|
state.mkdir()
|
|
(state / "pr_shas").write_text(SHA_A)
|
|
(state / "pr_bases").write_text("main")
|
|
(state / "pr_state").write_text("open")
|
|
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["STUB_DIR"] = str(state)
|
|
env["ETV_GITEA_TOKEN"] = "stub-token" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
|
env["ETV_GITEA_URL"] = "http://gitea.example"
|
|
env["ETV_GITEA_REPO"] = "timothy/ersatztv"
|
|
env.pop("ETV_GITEA_BASICAUTH", None)
|
|
|
|
class Handle:
|
|
def __init__(self):
|
|
self.env = env
|
|
self.state = state
|
|
|
|
def set_head_sequence(self, *shas):
|
|
(state / "pr_shas").write_text(" ".join(shas))
|
|
|
|
def set_pr_state(self, value):
|
|
(state / "pr_state").write_text(value)
|
|
|
|
def fail_posts_to(self, url_substring):
|
|
"""Make POSTs whose URL contains this substring fail, as curl -f does on a 4xx/5xx."""
|
|
(state / "post_fail").write_text(url_substring)
|
|
|
|
def set_base_sequence(self, *refs):
|
|
"""Base branch per PR GET. 'MISSING' omits `.base` from the response entirely."""
|
|
(state / "pr_bases").write_text(" ".join(refs))
|
|
|
|
def set_status_creator(self, login):
|
|
"""Who Gitea records as the author of the read-back status. 'NULL' models an Actions
|
|
token, whose statuses carry `creator: null` (ersatztv#742/#845)."""
|
|
(state / "status_creator").write_text(login)
|
|
|
|
def fail_status_readback(self):
|
|
"""Make the combined-status GET fail, as curl -f does on a 4xx/5xx."""
|
|
(state / "status_fail").write_text("1")
|
|
|
|
def set_status_filler(self, n):
|
|
"""Prepend N unrelated status contexts, so the read-back has to page past them."""
|
|
(state / "status_filler").write_text(str(n))
|
|
|
|
def set_status_body(self, body):
|
|
"""Replace the whole combined-status response with this literal JSON text."""
|
|
(state / "status_body").write_text(body)
|
|
|
|
def run(self, *args):
|
|
return subprocess.run(
|
|
["bash", str(SCRIPT), *args],
|
|
env=env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
|
|
def posts(self):
|
|
log = state / "posts.jsonl"
|
|
if not log.exists():
|
|
return []
|
|
return [json.loads(line) for line in log.read_text().splitlines() if line.strip()]
|
|
|
|
def statuses(self):
|
|
return [p for p in self.posts() if "/statuses/" in p["url"]]
|
|
|
|
def comments(self):
|
|
return [p for p in self.posts() if "/comments" in p["url"]]
|
|
|
|
return Handle()
|
|
|
|
|
|
def test_positive_verdict_posts_success_on_the_resolved_head(gitea):
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode == 0, result.stderr
|
|
|
|
statuses = gitea.statuses()
|
|
assert len(statuses) == 1
|
|
assert statuses[0]["url"].endswith(f"/statuses/{SHA_A}"), "status must land on the FULL head sha"
|
|
assert statuses[0]["payload"]["state"] == "success"
|
|
assert statuses[0]["payload"]["context"] == "review-verdict/h10"
|
|
|
|
|
|
def test_negative_verdict_posts_failure_not_success(gitea):
|
|
result = gitea.run("42", "BLOCKED", "two findings outstanding")
|
|
assert result.returncode == 0, result.stderr
|
|
|
|
payload = gitea.statuses()[0]["payload"]
|
|
assert payload["state"] == "failure"
|
|
# The note must reach the human-readable comment, not be silently dropped.
|
|
assert "two findings outstanding" in gitea.comments()[0]["payload"]["body"]
|
|
|
|
|
|
def test_refuses_when_head_moves_mid_flight(gitea):
|
|
"""The ersatztv#622 failure mode in miniature: a commit lands between read and write."""
|
|
gitea.set_head_sequence(SHA_A, SHA_B)
|
|
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "UNREVIEWED" in result.stderr
|
|
assert gitea.statuses() == [], "no status may be written once the reviewed head is stale"
|
|
# And no comment either, since ersatztv#792: with the comment written FIRST, a refusal left
|
|
# `Review-verdict: MERGEABLE @ <sha>` on the PR with no
|
|
# status behind it, which reads to an operator as consent that was never granted.
|
|
assert gitea.comments() == [], "a refusal must leave no verdict comment standing in for a status"
|
|
|
|
|
|
def test_never_retargets_the_verdict_at_the_new_head(gitea):
|
|
"""Refusing is not enough — it must also not 'helpfully' green the commit that arrived."""
|
|
gitea.set_head_sequence(SHA_A, SHA_B)
|
|
gitea.run("42", "MERGEABLE")
|
|
assert not any(SHA_B in s["url"] for s in gitea.statuses())
|
|
|
|
|
|
@pytest.mark.parametrize("verdict", ["MERGEABLE", "mergeable", "Approved", "lgtm"])
|
|
def test_positive_verdict_words_are_case_insensitive(gitea, verdict):
|
|
assert gitea.run("42", verdict).returncode == 0
|
|
assert gitea.statuses()[0]["payload"]["state"] == "success"
|
|
|
|
|
|
@pytest.mark.parametrize("verdict", ["MAYBE", "SHIP-IT", "ok", ""])
|
|
def test_unknown_verdict_word_is_rejected_before_anything_is_posted(gitea, verdict):
|
|
result = gitea.run("42", verdict)
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == [], "a typo must not post a comment either"
|
|
|
|
|
|
def test_refuses_a_closed_pr(gitea):
|
|
gitea.set_pr_state("closed")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_refuses_a_non_numeric_pr_number(gitea):
|
|
result = gitea.run("../../etc/passwd", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_fails_loudly_without_credentials(gitea):
|
|
gitea.env.pop("ETV_GITEA_TOKEN")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert "credentials" in result.stderr
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_unreachable_pr_is_an_error_not_a_silent_success(gitea):
|
|
gitea.set_head_sequence("GONE")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
# --- Cross-checks against the hook's own condition-(c) parser -----------------------------------
|
|
|
|
|
|
def _classify(body: str, head: str) -> str:
|
|
"""Run the REAL H10 classifier over a comment body — no Python mirror of the grammar.
|
|
|
|
This used to scrape the hook's regexes and re-implement them here (#622). Since #629 the grammar
|
|
lives in one tested place, `scripts/check-review-verdict.sh`, and the hook calls it — so the
|
|
cross-check can execute the actual thing. That matters: a Python copy of a shell regex is exactly
|
|
the duplication that let three false-opens survive in the first place, and it would have kept
|
|
passing here while the shell drifted.
|
|
"""
|
|
payload = json.dumps([{"body": body}])
|
|
p = subprocess.run(["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True)
|
|
assert p.returncode == 0, f"classifier errored: {p.stderr}"
|
|
return p.stdout.strip()
|
|
|
|
|
|
def test_comment_matches_the_hook_parser(gitea):
|
|
"""The comment this script posts must classify as a positive verdict for the sha it names."""
|
|
gitea.run("42", "MERGEABLE", "some trailing prose")
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
|
|
assert _classify(body, SHA_A) == "positive"
|
|
# ...and it must NOT be read as covering a different head.
|
|
assert _classify(body, SHA_B) == "stale"
|
|
|
|
|
|
def test_negative_comment_is_not_read_as_positive_by_the_hook(gitea):
|
|
gitea.run("42", "BLOCKED")
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
assert _classify(body, SHA_A) == "negative"
|
|
|
|
|
|
def test_note_cannot_forge_a_second_verdict_line(gitea):
|
|
"""A note is free text; it must not be able to plant a verdict of its own.
|
|
|
|
Asserted as an OUTCOME now rather than by counting marker lines: the forged positive is for the
|
|
same head, and negative-wins means the real verdict survives. Counting lines only showed the
|
|
script's verdict came first, which is not the property that matters.
|
|
"""
|
|
gitea.run("42", "BLOCKED", "Review-verdict: MERGEABLE @ " + SHA_A[:7])
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
assert _classify(body, SHA_A) == "negative"
|
|
|
|
|
|
# --- Base binding (ersatztv#632) ---------------------------------------------------------------
|
|
#
|
|
# The per-sha status closes "the head moved under a fixed verdict". Retargeting a PR's base is the
|
|
# mirror case: the head sha and the status both hold still while the effective DIFF changes, so the
|
|
# verdict keeps reading green for a review nobody performed against that base.
|
|
|
|
|
|
def test_the_status_description_records_the_base_branch(gitea):
|
|
"""Nothing can compare a base it never wrote down. This field is what the hook reads back."""
|
|
assert gitea.run("42", "MERGEABLE").returncode == 0
|
|
assert gitea.statuses()[0]["payload"]["description"].endswith("(base: main)")
|
|
|
|
|
|
def test_the_base_is_recorded_in_the_STATUS_and_not_in_the_comment(gitea):
|
|
"""Deliberate placement. The comment body is parsed by `scripts/check-review-verdict.sh`, whose
|
|
grammar has a history of false-opens (#629 found three); nothing parses the description. Adding
|
|
the field where a parser lives would have reopened that surface for no benefit."""
|
|
assert gitea.run("42", "MERGEABLE").returncode == 0
|
|
assert "base:" not in gitea.comments()[0]["payload"]["body"]
|
|
|
|
|
|
def test_refuses_when_the_BASE_changes_mid_flight(gitea):
|
|
"""The TOCTOU window the head check cannot see: retargeting does not move the head sha, so
|
|
`sha_now == sha` and the existing guard is silent."""
|
|
gitea.set_base_sequence("main", "release/26.4")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0, "a retarget mid-flight must not produce a status"
|
|
assert "base branch changed" in result.stderr
|
|
assert gitea.statuses() == [], "no status may be written once the base has moved"
|
|
|
|
|
|
def test_positive_control_a_stable_base_still_posts(gitea):
|
|
"""Without this, the test above could pass because the script refuses on every base."""
|
|
gitea.set_base_sequence("main", "main")
|
|
assert gitea.run("42", "MERGEABLE").returncode == 0
|
|
assert len(gitea.statuses()) == 1
|
|
|
|
|
|
def test_refuses_when_the_pr_has_no_resolvable_base(gitea):
|
|
"""A verdict that cannot record what it was formed against is not a verdict this gate can
|
|
later re-check, so it fails closed rather than posting an unbindable success."""
|
|
gitea.set_base_sequence("MISSING")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.statuses() == []
|
|
|
|
|
|
def test_a_failed_HEAD_RECHECK_writes_no_status(gitea):
|
|
"""Fail-closed on the re-read itself, not just on a moved head.
|
|
|
|
Left IMPLICIT, this guard is only a side effect: `sha_now=$(api_get ... | jq ...)` aborts under
|
|
`set -e` + `pipefail` when the GET fails. Asserted by nothing, folding the head and base
|
|
re-reads into one `$(... || true)` variable silently converts it to fail-OPEN — both guards see
|
|
an empty string, both no-op, and the status is written having confirmed nothing. Asserted here
|
|
so the behaviour is a contract rather than a side effect of a shell option.
|
|
"""
|
|
gitea.set_head_sequence(SHA_A, "GONE")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.statuses() == [], (
|
|
"a status was written even though the head/base re-read failed — nothing was confirmed"
|
|
)
|
|
|
|
|
|
# --- write-side polarity, the half the #774 rescue initially missed ------------------------------
|
|
|
|
|
|
# Kept deliberately in step with POSITIVE_WORDS/NEGATIVE_WORDS in test_check_review_verdict.py.
|
|
# The two lists are NOT compared to each other, and this file does not claim they agree — proving
|
|
# that needs one shared vocabulary both scripts read, which is ersatztv#788. What each side proves
|
|
# is its OWN polarity: that these established tokens still map the way reviewers rely on.
|
|
WRITE_POSITIVE = ["MERGEABLE", "APPROVED", "LGTM"]
|
|
WRITE_NEGATIVE = ["BLOCKED", "NOT-MERGEABLE"]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("word", "expected"),
|
|
[(w, "success") for w in WRITE_POSITIVE] + [(w, "failure") for w in WRITE_NEGATIVE],
|
|
)
|
|
def test_each_verdict_word_posts_its_established_polarity(word, expected, gitea):
|
|
"""The WRITE side of the polarity pair.
|
|
|
|
ersatztv#774 withdrew a test that compared the two verdict vocabularies by parsing shell, and
|
|
rescued its disjointness half into test_check_review_verdict.py. That rescue
|
|
covered only `check-review-verdict.sh`: an in-memory mutation adding `BLOCKED` to the SUCCESS
|
|
arm here produced a write-side overlap the deleted test caught and the rescue did not, because
|
|
the rescue never executes this script. That was a real, undisclosed loss and this closes it.
|
|
|
|
`case` takes the FIRST matching arm, so a token listed in both arms is not ambiguous — it
|
|
resolves to whichever comes first, exactly as `is_pos` wins on the read side. The consequence is
|
|
the same and it is the one that matters: a word a reviewer means as BLOCKED silently posting
|
|
`success` writes a green `review-verdict/h10`, which is the required context branch protection
|
|
honours. That flips this assertion.
|
|
|
|
What this does NOT prove, stated because the parent test was withdrawn for overclaiming: it is
|
|
a polarity regression over five established tokens, not a universal disjointness property and
|
|
not parity with the read side. A token added to only one script is untested here, not caught.
|
|
"""
|
|
result = gitea.run("42", word)
|
|
assert result.returncode == 0, result.stderr
|
|
payload = gitea.statuses()[0]["payload"]
|
|
assert payload["state"] == expected, (
|
|
f"'{word}' posted state {payload['state']!r}, expected {expected!r}. `case` takes the first "
|
|
"matching arm, so a token that has appeared in the other arm resolves there silently — and "
|
|
"a BLOCKED verdict posting `success` writes a green required context."
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("head_seq", "base_seq", "field"),
|
|
[
|
|
(("NOHEAD",), ("main", "main"), "head sha"),
|
|
((SHA_A,), ("main", "MISSING"), "base ref"),
|
|
],
|
|
)
|
|
def test_a_reread_that_LOSES_a_field_refuses_instead_of_posting(head_seq, base_seq, field, gitea):
|
|
"""The fail-OPEN one level below the TOCTOU guard (ersatztv#778).
|
|
|
|
Both re-read checks were written as `[ -n "$x" ] && [ "$x" != "$want" ]`. That conjunct makes an
|
|
EMPTY value a no-op: a well-formed 2xx response that merely omits `.head.sha` or `.base.ref`
|
|
yields an empty variable, neither comparison runs, and the status is posted having confirmed
|
|
NOTHING about the head or the base — while the script's whole purpose at that point is to refuse
|
|
unless it can confirm. The transport failure one line above was already fatal, which is exactly
|
|
what made this shape easy to miss: the loud case was handled and the quiet one was not.
|
|
|
|
Asserted on the OBSERVABLE outcome — no status written — rather than on message text, so it
|
|
still holds if the wording changes.
|
|
"""
|
|
gitea.set_head_sequence(SHA_A, *head_seq)
|
|
gitea.set_base_sequence(*base_seq)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0, (
|
|
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed nothing about it"
|
|
)
|
|
assert not gitea.statuses(), (
|
|
f"a status was written despite the re-read carrying no {field} — this is the fail-open the -n conjunct created"
|
|
)
|
|
|
|
|
|
# --- ersatztv#792: no path may write no status and report success ------------------------------
|
|
#
|
|
# The issue was filed on an observed "printed the refusal AND exited 0". Re-measured on the tree
|
|
# that fixed the re-read fence: every refusal already exits NON-ZERO, and the exit-0 came from the
|
|
# caller's pipeline, not from the script. That is worth an executed contract rather than a second
|
|
# reading of the source — `die` is one line away from being edited into a `return`, and this file is
|
|
# where that would be caught. The parametrisation covers each refusal REASON, not one
|
|
# representative, because those paths were added at four different times and only the shared helper
|
|
# makes them agree today.
|
|
#
|
|
# SCOPE, since "every path" would overclaim: these are the eight refusal MODES reachable through the
|
|
# real entry point. The source also exits non-zero for a usage error (2), a failing `jq` (its own
|
|
# status, under `set -e` + `pipefail`), and a signal (128+n); none of those is a refusal DECISION,
|
|
# and only the non-zero-ness is common to all of them.
|
|
|
|
|
|
def _drive(gitea, mode):
|
|
if mode == "head-moved":
|
|
gitea.set_head_sequence(SHA_A, SHA_B)
|
|
elif mode == "reread-failed":
|
|
gitea.set_head_sequence(SHA_A, "GONE")
|
|
elif mode == "reread-lost-head":
|
|
gitea.set_head_sequence(SHA_A, "NOHEAD")
|
|
elif mode == "base-retargeted":
|
|
gitea.set_base_sequence("main", "some-feature-branch")
|
|
elif mode == "reread-lost-base":
|
|
gitea.set_base_sequence("main", "MISSING")
|
|
elif mode == "first-read-failed":
|
|
gitea.set_head_sequence("GONE")
|
|
elif mode == "pr-closed":
|
|
gitea.set_pr_state("closed")
|
|
elif mode == "status-post-failed":
|
|
gitea.fail_posts_to("/statuses/")
|
|
else: # pragma: no cover - a typo in the parametrisation must not pass silently
|
|
raise AssertionError(f"unknown mode {mode}")
|
|
return gitea.run("42", "MERGEABLE")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"mode",
|
|
[
|
|
"head-moved",
|
|
"reread-failed",
|
|
"reread-lost-head",
|
|
"base-retargeted",
|
|
"reread-lost-base",
|
|
"first-read-failed",
|
|
"pr-closed",
|
|
"status-post-failed",
|
|
],
|
|
)
|
|
def test_every_REFUSAL_MODE_that_writes_no_status_exits_non_zero(gitea, mode):
|
|
result = _drive(gitea, mode)
|
|
assert gitea.statuses() == [], f"{mode} wrote a status it had no business writing"
|
|
assert result.returncode != 0, (
|
|
f"{mode} wrote no status and reported SUCCESS — anything checking $? concludes the verdict "
|
|
f"posted. stdout={result.stdout!r} stderr={result.stderr!r}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"mode",
|
|
[
|
|
"head-moved",
|
|
"reread-failed",
|
|
"reread-lost-head",
|
|
"base-retargeted",
|
|
"reread-lost-base",
|
|
"status-post-failed",
|
|
],
|
|
)
|
|
def test_no_refusal_leaves_a_VERDICT_COMMENT_standing_in_for_the_status(gitea, mode):
|
|
"""The half-state, which is the part of #792 that was really broken.
|
|
|
|
The comment is not the gate — `review-verdict/h10` is — but `Review-verdict: MERGEABLE @ <head>`
|
|
sitting on a PR reads exactly like consent. Every mode here is one where the status is refused
|
|
after the head has been resolved, i.e. every mode that could once have left that comment behind.
|
|
"""
|
|
_drive(gitea, mode)
|
|
assert gitea.comments() == [], f"{mode} left an orphaned verdict comment: {gitea.comments()}"
|
|
|
|
|
|
def test_the_status_is_written_BEFORE_the_comment(gitea):
|
|
"""Ordering is the mechanism, so it is asserted rather than described.
|
|
|
|
Status-then-comment makes the only reachable half-state the safe one: a status with no comment
|
|
leaves the merge hook's condition (c) with nothing to classify, which is an `ask`. The reverse
|
|
order manufactures the appearance of a granted verdict.
|
|
"""
|
|
assert gitea.run("42", "MERGEABLE").returncode == 0
|
|
urls = [p["url"] for p in gitea.posts()]
|
|
assert len(urls) == 2, urls
|
|
assert "/statuses/" in urls[0], f"the status must be written first, got {urls}"
|
|
assert "/comments" in urls[1], f"the comment must be written second, got {urls}"
|
|
|
|
|
|
def test_a_failed_COMMENT_after_a_written_status_is_still_an_error(gitea):
|
|
"""The surviving half-state is safe, not silent: the operator is told to re-run."""
|
|
gitea.fail_posts_to("/comments")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert len(gitea.statuses()) == 1, "the status was already written and must not be rolled back"
|
|
assert gitea.comments() == []
|
|
assert "COMMENT could not be posted" in result.stderr
|
|
assert "ask" in result.stderr.lower(), "it must say what the gate will do, not just that a call failed"
|
|
|
|
|
|
# ==================================================================================================
|
|
# Is the verdict this tool writes one the GATE will actually inherit? (ersatztv#845)
|
|
# ==================================================================================================
|
|
#
|
|
# `.gitea/workflows/review-verdict.yml` inherits an existing `review-verdict/h10=success` only from a
|
|
# status whose `.creator.login` is on its `H10_REVIEWERS` allow-list (ersatztv#742). This script
|
|
# posts with whatever account owns the credential in the environment. Those two values were coupled
|
|
# with nothing asserting the coupling, and the failure was silent in the worst way: the status is
|
|
# written, THIS TOOL REPORTS SUCCESS, and the next PR event re-derives it and posts over it. Forever.
|
|
#
|
|
# The account below is written as the literal `timothy` on purpose, and it is the same deliberate
|
|
# hardcode `test_pr_changed_files.py::test_the_allowlisted_reviewer_in_the_TESTS_is_the_one_the_
|
|
# WORKFLOW_ships` exists to pin. Deriving it here would make these tests vacuous: the mutation proof
|
|
# works precisely BECAUSE changing the workflow literal moves the allow-list out from under a fixed
|
|
# creator. Derive both and they would move together and nothing would ever redden.
|
|
ALLOWLISTED = "timothy"
|
|
OFF_LIST = "renovate"
|
|
|
|
|
|
def test_a_verdict_posted_by_an_ALLOWLISTED_account_is_accepted(gitea):
|
|
"""The accept path — and the mutation proof for the whole derivation.
|
|
|
|
This is the test `mutation_manifest.py` reddens by rewriting `H10_REVIEWERS` in the SHIPPED
|
|
workflow. That single mutation proves two things at once, which is why it is the declared one:
|
|
the script must be reading the allow-list LIVE from the gate (a hand-copied list would not
|
|
move), and the membership comparison must actually gate the outcome (a no-op comparison would
|
|
not care that the list moved).
|
|
"""
|
|
gitea.set_status_creator(ALLOWLISTED)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode == 0, f"the verdict writer REFUSED an allow-listed account: {result.stderr}"
|
|
assert len(gitea.statuses()) == 1
|
|
assert len(gitea.comments()) == 1, "an accepted verdict must still post its comment"
|
|
|
|
|
|
def test_a_verdict_posted_by_an_account_OFF_the_allowlist_is_refused_before_the_comment(gitea):
|
|
"""The whole point of ersatztv#845: this used to report success and deadlock the PR later."""
|
|
gitea.set_status_creator(OFF_LIST)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0, "a verdict the gate will not inherit must not report success"
|
|
assert "H10_REVIEWERS" in result.stderr, (
|
|
"the diagnostic must name the allow-list, because the repair is to edit it or to re-post "
|
|
"with another credential — an operator cannot act on 'refused'"
|
|
)
|
|
assert OFF_LIST in result.stderr, "the diagnostic must name the account that actually posted"
|
|
assert gitea.comments() == [], (
|
|
"the refusal must land BEFORE the comment: a verdict comment with no honoured status is the "
|
|
"half-state ersatztv#792 removed, and it reads to a human as consent that was never granted"
|
|
)
|
|
# Stated rather than fixed. The status IS written by the time the author can be measured, and it
|
|
# is left standing deliberately — see the residual note in the script.
|
|
assert len(gitea.statuses()) == 1
|
|
|
|
|
|
def test_a_status_that_reads_back_with_NO_creator_is_refused(gitea):
|
|
"""`creator: null` is how a status POSTed by an ACTIONS token appears, and the gate never
|
|
inherits one of those as a reviewer verdict. Distinct from the off-list case: there is no
|
|
account to name, so 'add them to the allow-list' would be the wrong advice."""
|
|
gitea.set_status_creator("NULL")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "ACTIONS token" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_status_OVERWRITTEN_between_the_write_and_the_read_is_not_blamed_on_its_author(gitea):
|
|
"""`/commits/{sha}/status` returns the LATEST status per context. If the gate overwrote ours
|
|
between the POST and the read-back, judging THAT status's author would accuse the wrong party —
|
|
and an Actions write carries no creator, so it would surface as a credential problem that does
|
|
not exist. Identify our own write first, then judge it."""
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 1,
|
|
"statuses": [
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"status": "pending",
|
|
"description": "Awaiting review verdict",
|
|
"creator": None,
|
|
}
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "it was overwritten" in result.stderr
|
|
assert "ACTIONS token" not in result.stderr, (
|
|
"an overwritten status must not be reported as a wrong-credential problem"
|
|
)
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_an_unreadable_readback_refuses_rather_than_reporting_the_verdict_posted(gitea):
|
|
"""UNKNOWN must fail. A read that cannot be performed leaves it unproven that the verdict is in
|
|
force, and reporting success on unproven is exactly the ersatztv#845 shape."""
|
|
gitea.fail_status_readback()
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
# Same reason as above: `set -e` plus the absent-entry check already refuse, so without naming
|
|
# the message this test passed with the explicit `|| die` removed.
|
|
assert "could not be read back" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_readback_whose_statuses_array_is_NULL_is_refused(gitea):
|
|
"""Gitea serialises a nil slice as `null`, not `[]` (ersatztv#751). The script tests the array
|
|
TYPE rather than assuming one; a body that merely lost its array must refuse, not be read as
|
|
'no verdict, carry on'."""
|
|
gitea.set_status_body(json.dumps({"total_count": 0, "statuses": None}))
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
# The DIAGNOSTIC, not merely a non-zero exit. Asserting only the exit code let this pass on the
|
|
# downstream absent-entry check, so the array-TYPE clause it names could be disarmed with the
|
|
# suite still green — an outcome the script produces anyway is not evidence about a clause.
|
|
assert "carries no `.statuses` ARRAY" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
# --- The allow-list is DERIVED from the gate, not restated beside it. ----------------------------
|
|
|
|
|
|
def _sandbox(tmp_path, reviewers):
|
|
"""A throwaway tree carrying the real `scripts/` and the real gate workflow, with the workflow's
|
|
`H10_REVIEWERS` literal rewritten. Returns a runner over the SANDBOX copy of the script.
|
|
|
|
The whole `scripts/` directory is copied rather than the three files in play, because the script
|
|
resolves both of its libraries relative to `${BASH_SOURCE[0]}` — copying selectively would leave
|
|
it sourcing the real repo's copies and the rewrite would prove nothing.
|
|
"""
|
|
root = tmp_path / "sandbox"
|
|
shutil.copytree(REPO_ROOT / "scripts", root / "scripts")
|
|
workflow = root / ".gitea" / "workflows" / "review-verdict.yml"
|
|
workflow.parent.mkdir(parents=True)
|
|
source = (REPO_ROOT / ".gitea" / "workflows" / "review-verdict.yml").read_text()
|
|
rewritten, n = re.subn(
|
|
r'^(\s*)H10_REVIEWERS="[^"]*"$',
|
|
lambda m: f'{m.group(1)}H10_REVIEWERS="{reviewers}"',
|
|
source,
|
|
flags=re.MULTILINE,
|
|
)
|
|
assert n == 1, f"expected exactly one H10_REVIEWERS assignment to rewrite, substituted {n}"
|
|
workflow.write_text(rewritten)
|
|
return root / "scripts" / "post-review-verdict.sh"
|
|
|
|
|
|
def test_the_allowlist_MOVES_when_the_workflow_moves(gitea, tmp_path):
|
|
"""The single-declaration property, measured end to end rather than argued.
|
|
|
|
Rewrite the gate's own literal to an account this repo does not have, drive the REAL script, and
|
|
the accounts swap roles: the one the shipped workflow accepts is refused, and the invented one is
|
|
accepted. A copy of the list living beside the script could not produce that.
|
|
"""
|
|
invented = "not-a-real-account"
|
|
script = _sandbox(tmp_path, invented)
|
|
|
|
def run_with(creator):
|
|
gitea.set_status_creator(creator)
|
|
(gitea.state / "posts.jsonl").unlink(missing_ok=True)
|
|
(gitea.state / "pr_get_count").unlink(missing_ok=True)
|
|
return subprocess.run(
|
|
["bash", str(script), "42", "MERGEABLE"],
|
|
env=gitea.env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
|
|
accepted = run_with(invented)
|
|
assert accepted.returncode == 0, f"the rewritten allow-list was not the one used: {accepted.stderr}"
|
|
|
|
refused = run_with(ALLOWLISTED)
|
|
assert refused.returncode != 0, (
|
|
f"{ALLOWLISTED} was accepted against a workflow that no longer lists them, so the script is "
|
|
"not deriving the allow-list from the gate"
|
|
)
|
|
assert invented in refused.stderr
|
|
|
|
|
|
def test_a_workflow_whose_allowlist_cannot_be_DERIVED_refuses_before_any_write(gitea, tmp_path):
|
|
"""Fail-closed, and early. Two assignments means the shell would run the last one executed, so
|
|
which value the gate uses is unknowable from here — and a writer that cannot tell whether its
|
|
verdict is inheritable must not post one. Nothing is written at all, so there is no half-state
|
|
to clean up.
|
|
"""
|
|
script = _sandbox(tmp_path, ALLOWLISTED)
|
|
text = script.parent.parent.joinpath(".gitea/workflows/review-verdict.yml")
|
|
body = text.read_text()
|
|
marker = f'H10_REVIEWERS="{ALLOWLISTED}"'
|
|
assert body.count(marker) == 1
|
|
text.write_text(body.replace(marker, f'{marker}\n H10_REVIEWERS="someone-else"'))
|
|
|
|
result = subprocess.run(
|
|
["bash", str(script), "42", "MERGEABLE"],
|
|
env=gitea.env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
assert result.returncode != 0
|
|
assert gitea.statuses() == [], "a derivation it cannot trust must refuse before writing anything"
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_VERBATIM_live_gitea_status_body_is_accepted(gitea):
|
|
"""Pins the real response shape, independently of the shim that serves it.
|
|
|
|
The shim replays what was POSTed, so it can only ever disagree with the parser about a field it
|
|
is told to rename — which is a fixture agreeing with the code by construction. This body is the
|
|
shape `GET /repos/{o}/{r}/commits/{sha}/status` actually returned on Gitea 1.27.1 (measured
|
|
2026-08-29 against a real `review-verdict/h10` row), trimmed of fields nothing reads. The key
|
|
that matters is `status`: a row inside `.statuses[]` carries its state under that name, while
|
|
`state` is only the top-level AGGREGATE. Parsing `.state` per row yields "" for every
|
|
well-formed response, refuses every verdict, and deadlocks the repo — no PR merges without one.
|
|
"""
|
|
short = SHA_A[:7]
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"sha": SHA_A,
|
|
"state": "success", # the AGGREGATE, deliberately present and deliberately not read
|
|
"total_count": 2,
|
|
"statuses": [
|
|
{
|
|
"id": 32,
|
|
"status": "success",
|
|
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
|
|
"context": "review-verdict/h10",
|
|
"creator": {"id": 1, "login": ALLOWLISTED, "username": ALLOWLISTED},
|
|
"created_at": "2026-08-29T21:40:22+02:00",
|
|
},
|
|
{
|
|
"id": 31,
|
|
"status": "success",
|
|
"description": "Build & test",
|
|
"context": "ci/build",
|
|
"creator": None,
|
|
},
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode == 0, f"the verdict writer REFUSED a verbatim live Gitea response: {result.stderr}"
|
|
assert len(gitea.comments()) == 1
|
|
|
|
|
|
def test_a_status_row_carrying_NO_state_field_says_so_instead_of_blaming_a_race(gitea):
|
|
"""A shape it cannot read is not an overwrite. Stating a cause that did not happen is its own
|
|
defect class here (#859) — 'something overwrote it' would send the reader hunting a race that
|
|
never occurred, on what is really a Gitea-version problem."""
|
|
short = SHA_A[:7]
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 1,
|
|
"statuses": [
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
|
|
"creator": {"login": ALLOWLISTED},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "no state field" in result.stderr
|
|
assert "overwrote it" not in result.stderr, (
|
|
"an unreadable response shape must not be reported as a race that did not happen"
|
|
)
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_readback_MISSING_our_context_reports_the_row_count_and_names_no_cause(gitea):
|
|
"""The row this run wrote is the newest on the head and sorts first, so its absence means
|
|
something removed or replaced it — the diagnostic reports how many rows it read and does not
|
|
guess at which. Paging is deliberately NOT offered as a cause: `total_count` reports the page
|
|
rather than the total, which suggests it, but the selection ORDER — max id per context,
|
|
descending — makes that truncation unproducible for this row.
|
|
This also witnesses the row count, which the empty-body case cannot reach."""
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 2,
|
|
"statuses": [
|
|
{"context": "ci/build", "status": "success", "creator": None},
|
|
{"context": "ci/lint", "status": "success", "creator": None},
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "not among the 2 status rows" in result.stderr, (
|
|
"the row count it actually read must be in the message, or a paging overflow is unreadable"
|
|
)
|
|
assert gitea.comments() == []
|
|
|
|
|
|
# --- The derivation library itself. --------------------------------------------------------------
|
|
#
|
|
# Driven directly rather than through the script, because three of its clauses cannot be reached from
|
|
# the script at all: the script only ever runs against the real workflow, which is well-formed. Each
|
|
# of these was measured to survive a disarm of its clause when only the end-to-end tests existed.
|
|
|
|
LIB = REPO_ROOT / "scripts" / "lib" / "h10-reviewers.sh"
|
|
|
|
|
|
def _lib(snippet, cwd, workflow_body=None):
|
|
"""Source the real library and run `snippet`. Returns the CompletedProcess.
|
|
|
|
`cwd` matters: pathname expansion is resolved against it, which is the whole point of the glob
|
|
test below. A workflow body, when given, is written to `cwd/wf.yml`.
|
|
"""
|
|
if workflow_body is not None:
|
|
(cwd / "wf.yml").write_text(workflow_body)
|
|
script = f'set -euo pipefail\n. "{LIB}"\n{snippet}\n'
|
|
return subprocess.run(["bash", "-c", script], cwd=str(cwd), capture_output=True, text=True)
|
|
|
|
|
|
def test_a_GLOB_entry_is_rejected_rather_than_expanded_against_the_working_directory(tmp_path):
|
|
"""The documented fail-open, one file over from where it is recorded.
|
|
|
|
`for x in $list` and `$(printf %s "$list")` both perform PATHNAME EXPANSION, so a `*` entry
|
|
expands to the FILENAMES around it — and any of those matching the plain-login class would
|
|
validate cleanly and enter the allow-list as a real reviewer. `review-verdict-vocabulary.sh`
|
|
records this happening to its own word list. The cwd below is stocked with names that would
|
|
validate, so an expanding implementation loads a non-empty list instead of refusing.
|
|
"""
|
|
for name in ("timothy", "renovate", "attacker"):
|
|
(tmp_path / name).write_text("")
|
|
result = _lib(
|
|
'etv_h10_reviewers_load "$PWD/wf.yml" && echo "LOADED=[$ETV_H10_REVIEWERS]"',
|
|
tmp_path,
|
|
' H10_REVIEWERS="*"\n',
|
|
)
|
|
assert result.returncode != 0, (
|
|
f"a `*` entry was accepted; it expanded to {result.stdout.strip()!r} in a directory "
|
|
"containing plausible login names"
|
|
)
|
|
assert "LOADED=" not in result.stdout
|
|
assert "glob metacharacters" in result.stderr
|
|
|
|
|
|
def test_an_EMPTY_or_whitespace_only_allowlist_is_rejected(tmp_path):
|
|
"""`" "` is a non-empty STRING that splits to nothing, so emptiness is checked on the split
|
|
count. An empty allow-list is not a harmless default at either end: the gate's `for rv in ""`
|
|
iterates zero times, and here every account would be judged not-a-member."""
|
|
for body in (' H10_REVIEWERS=""\n', ' H10_REVIEWERS=" "\n'):
|
|
result = _lib('etv_h10_reviewers_load "$PWD/wf.yml"', tmp_path, body)
|
|
assert result.returncode != 0, f"accepted an empty allow-list from {body!r}"
|
|
assert "EMPTY" in result.stderr
|
|
|
|
|
|
def test_membership_asked_before_a_successful_load_answers_CANNOT_TELL_not_NOT_A_MEMBER(tmp_path):
|
|
"""Three outcomes, not two. Folding "cannot tell" into "not a member" would make a broken
|
|
checkout indistinguishable from a wrong credential, and the script would then tell the operator
|
|
to fix their credential — a cause that did not happen. Exit 2 is what keeps them apart."""
|
|
result = _lib('set +e; etv_h10_reviewers_contains timothy; echo "RC=$?"', tmp_path)
|
|
assert "RC=2" in result.stdout, f"expected cannot-tell (2), got {result.stdout.strip()!r}"
|
|
|
|
# And after a FAILED load, not merely before any load at all.
|
|
result = _lib(
|
|
'set +e; etv_h10_reviewers_load "$PWD/nonexistent.yml"; etv_h10_reviewers_contains timothy; echo "RC=$?"',
|
|
tmp_path,
|
|
)
|
|
assert "RC=2" in result.stdout, f"a failed load must not leave the list queryable: {result.stdout!r}"
|
|
|
|
|
|
def test_a_successful_load_answers_member_and_non_member_apart(tmp_path):
|
|
"""The positive control for the three-outcome contract above: 0 and 1 must both be reachable,
|
|
or the exit-2 assertion could be passing because the function only ever returns 2."""
|
|
result = _lib(
|
|
'etv_h10_reviewers_load "$PWD/wf.yml"; set +e; '
|
|
'etv_h10_reviewers_contains alice; echo "ALICE=$?"; '
|
|
'etv_h10_reviewers_contains carol; echo "CAROL=$?"',
|
|
tmp_path,
|
|
' H10_REVIEWERS="alice bob"\n',
|
|
)
|
|
assert "ALICE=0" in result.stdout, result.stdout
|
|
assert "CAROL=1" in result.stdout, result.stdout
|
|
|
|
|
|
def test_sourcing_the_library_does_not_disturb_the_callers_globbing_setting(tmp_path):
|
|
"""The membership split turns pathname expansion off and must put it back exactly as found —
|
|
`post-review-verdict.sh` runs under `set -euo pipefail` with globbing ON and interpolates API
|
|
values into messages afterwards."""
|
|
(tmp_path / "wf.yml").write_text(' H10_REVIEWERS="alice"\n')
|
|
on = _lib(
|
|
'etv_h10_reviewers_load "$PWD/wf.yml"; set +e; etv_h10_reviewers_contains alice; '
|
|
'case "$-" in *f*) echo "GLOB=off" ;; *) echo "GLOB=on" ;; esac',
|
|
tmp_path,
|
|
)
|
|
assert "GLOB=on" in on.stdout, on.stdout
|
|
off = _lib(
|
|
'set -f; etv_h10_reviewers_load "$PWD/wf.yml"; set +e; etv_h10_reviewers_contains alice; '
|
|
'case "$-" in *f*) echo "GLOB=off" ;; *) echo "GLOB=on" ;; esac',
|
|
tmp_path,
|
|
)
|
|
assert "GLOB=off" in off.stdout, off.stdout
|
|
|
|
|
|
def test_the_verdict_row_is_found_however_many_other_contexts_the_head_carries(gitea):
|
|
"""A POSITIVE CONTROL over a full page, not a property pin — said plainly because the obvious
|
|
reading is wrong.
|
|
|
|
The shim places our row at index 0 unconditionally (it models Gitea's real ordering: max id per
|
|
context, DESCENDING, and our just-POSTed row is the newest). So this cannot fail for the reason
|
|
its name suggests — deleting `?limit=100` leaves it green, and it would stay green against a
|
|
naive `.statuses[0]` implementation too. What it does check is that `map(select(...))` still
|
|
finds our context when the page is full of other rows.
|
|
|
|
It replaces a test that asserted the OPPOSITE — that with 40 other contexts the row was reachable
|
|
only because of `?limit=100` — which was an artifact of a shim that appended the new row LAST and
|
|
sliced from the front, manufacturing a truncation the server cannot produce. `?limit=100` has no
|
|
observable effect under the real ordering and is declared unwitnessed insurance in the script.
|
|
"""
|
|
gitea.set_status_filler(80)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode == 0, (
|
|
f"the verdict row was not found behind 80 other contexts, though it is the newest on the "
|
|
f"head and should sort first: {result.stderr}"
|
|
)
|
|
assert len(gitea.comments()) == 1
|
|
|
|
|
|
def test_TWO_rows_for_our_context_are_ambiguous_rather_than_resolved_by_taking_the_first(gitea):
|
|
"""`[0]` would pick a row and judge ITS author while a different row for the same context is the
|
|
one actually standing. Unreachable against Gitea as measured (the combined endpoint returns the
|
|
latest row per context), so this guards the property rather than an observed shape — which is
|
|
why the message says so instead of blaming a race."""
|
|
short = SHA_A[:7]
|
|
desc = f"Review-verdict: MERGEABLE @ {short} (base: main)"
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 2,
|
|
"statuses": [
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"status": "success",
|
|
"description": desc,
|
|
"creator": {"login": ALLOWLISTED},
|
|
},
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"status": "success",
|
|
"description": desc,
|
|
"creator": {"login": OFF_LIST},
|
|
},
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0, "an allow-listed FIRST row let a second row with an off-list creator through"
|
|
assert "2 separate rows" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_an_EMPTY_response_body_is_refused_as_a_SHAPE_problem(gitea):
|
|
"""An empty body yields no `.statuses` array, so it must refuse on the shape branch.
|
|
|
|
This does NOT witness the row-count fallback: an empty body exits
|
|
at the array-TYPE branch and never reaches the count. The count IS witnessed, by
|
|
`test_a_readback_MISSING_our_context_...`, which supplies a real array with no matching row. That
|
|
fallback was dead code and is gone — `.statuses` is
|
|
known to be an array by the time the count runs.
|
|
"""
|
|
gitea.set_status_body("")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "carries no `.statuses` ARRAY" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_the_split_pins_IFS_so_a_callers_separator_cannot_merge_or_split_entries(tmp_path):
|
|
"""Globbing is only half of what an unquoted expansion depends on; the other half is the ambient
|
|
`IFS`. With `IFS=-` a caller made `renovate` read as a member of the single entry `renovate-bot`
|
|
— measured against the real predecessor, not a hand-written mutant.
|
|
|
|
The mechanism is the split inside `etv_h10_reviewers_contains`, NOT the one in
|
|
`etv_h10_reviewers_load`: `load` stores the RAW string, so its own split can only affect
|
|
validation, never membership. Both are pinned, but only the `contains` one is witnessed by this
|
|
test, and that is the honest split of the two. Not reachable from the script today (bash resets
|
|
IFS at startup and the script never assigns it) — this pins the hardening the library's header
|
|
invites by inviting reuse."""
|
|
(tmp_path / "wf.yml").write_text(' H10_REVIEWERS="renovate-bot"\n')
|
|
result = _lib(
|
|
'IFS="-"; etv_h10_reviewers_load "$PWD/wf.yml"; set +e; '
|
|
'etv_h10_reviewers_contains renovate; echo "SPLIT=$?"; '
|
|
'etv_h10_reviewers_contains renovate-bot; echo "WHOLE=$?"',
|
|
tmp_path,
|
|
)
|
|
assert "SPLIT=1" in result.stdout, f"`renovate` read as a member under IFS=-: {result.stdout!r}"
|
|
assert "WHOLE=0" in result.stdout, result.stdout
|
|
|
|
|
|
def _standing_status(gitea, *, state, description):
|
|
"""A read-back carrying one `review-verdict/h10` row with exactly these two fields."""
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 1,
|
|
"statuses": [
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"status": state,
|
|
"description": description,
|
|
"creator": {"login": ALLOWLISTED},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
)
|
|
|
|
|
|
def test_a_row_differing_ONLY_in_state_is_refused(gitea):
|
|
"""Witnesses the STATE arm on its own.
|
|
|
|
`test_a_status_OVERWRITTEN_...` changes state AND description, so either arm could be deleted
|
|
with the suite green — the two overlap and mask each other (#685). Each arm therefore gets a
|
|
fixture that differs in exactly one field. The creator here is ALLOW-LISTED on purpose: the
|
|
refusal must come from identifying the row as not-ours, not from its author.
|
|
"""
|
|
short = SHA_A[:7]
|
|
_standing_status(gitea, state="pending", description=f"Review-verdict: MERGEABLE @ {short} (base: main)")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0, "a row whose STATE is not the one just written was accepted"
|
|
assert "it was overwritten" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_row_differing_ONLY_in_description_is_refused(gitea):
|
|
"""Witnesses the DESCRIPTION arm on its own — the one that actually identifies our write, since
|
|
it carries the verdict word, the short sha and the base."""
|
|
_standing_status(gitea, state="success", description="Review-verdict: MERGEABLE @ deadbee (base: main)")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0, "a row whose DESCRIPTION names a different head was accepted as this run's write"
|
|
assert "it was overwritten" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_an_off_list_account_may_still_record_a_REJECTION(gitea):
|
|
"""The allow-list applies to `success` ONLY, because that is what the gate does.
|
|
|
|
`review-verdict.yml` short-circuits on an existing `success` only for an allow-listed creator,
|
|
but on an existing `failure` for any ATTRIBUTABLE one — inheriting a rejection can only withhold
|
|
an exemption, while re-deriving one can turn it green. Enforcing membership on a `failure` here
|
|
refused a verdict the gate honours, told the reviewer their rejection would be re-derived when it
|
|
would not, and left a real reviewer no supported way to record a rejection at all.
|
|
"""
|
|
gitea.set_status_creator(OFF_LIST)
|
|
result = gitea.run("42", "BLOCKED", "two findings outstanding")
|
|
|
|
assert result.returncode == 0, (
|
|
f"an off-list account could not record a BLOCKED verdict, which the gate honours: {result.stderr}"
|
|
)
|
|
assert gitea.statuses()[0]["payload"]["state"] == "failure"
|
|
assert len(gitea.comments()) == 1, "a rejection must still get its human-readable comment"
|
|
assert "two findings outstanding" in gitea.comments()[0]["payload"]["body"]
|
|
|
|
|
|
def test_an_off_list_account_still_cannot_record_an_APPROVAL(gitea):
|
|
"""The other side of the same asymmetry — the positive direction still requires membership, so
|
|
the fix above did not widen the check into a no-op."""
|
|
gitea.set_status_creator(OFF_LIST)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "H10_REVIEWERS" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_row_carrying_ONLY_the_top_level_state_key_is_refused(gitea):
|
|
"""No `.state` fallback. The gate reads `.status` and nothing else, so tolerating a shape it
|
|
cannot read would make this tool report success on a verdict the gate re-derives — ersatztv#845,
|
|
recreated by the check meant to prevent it. On a shape neither understands, refusing is correct.
|
|
"""
|
|
short = SHA_A[:7]
|
|
gitea.set_status_body(
|
|
json.dumps(
|
|
{
|
|
"total_count": 1,
|
|
"statuses": [
|
|
{
|
|
"context": "review-verdict/h10",
|
|
"state": "success", # the key the GATE does not read
|
|
"description": f"Review-verdict: MERGEABLE @ {short} (base: main)",
|
|
"creator": {"login": ALLOWLISTED},
|
|
}
|
|
],
|
|
}
|
|
)
|
|
)
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0, "a row the gate cannot read was accepted, which is the ersatztv#845 stall re-created"
|
|
assert "no state field" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_an_ENVIRONMENT_variable_cannot_steer_which_workflow_the_allowlist_comes_from(gitea, tmp_path):
|
|
"""A declared security property that had no guard: `scripts/lib/h10-reviewers.sh` resolves the
|
|
gate definition from its OWN location and offers no `ETV_*_WORKFLOW` override, because an env var
|
|
choosing which tree answers a security question is an unsound input (ersatztv#858 is the same
|
|
shape one file over).
|
|
|
|
It held, but nothing pinned it — and restoring the override the way a refactor naturally would
|
|
(`ETV_H10_REVIEWERS_WORKFLOW="${ETV_H10_REVIEWERS_WORKFLOW:-}"`) left the whole suite green while
|
|
an environment variable installed an arbitrary allow-list and an off-list account was accepted as
|
|
a reviewer. That is the same class this branch removed elsewhere, in a clause labelled as a
|
|
deliberate decision rather than as untested.
|
|
"""
|
|
hostile = tmp_path / "hostile.yml"
|
|
hostile.write_text(f' H10_REVIEWERS="{OFF_LIST}"\n')
|
|
|
|
env = dict(gitea.env)
|
|
env["ETV_H10_REVIEWERS_WORKFLOW"] = str(hostile)
|
|
gitea.set_status_creator(OFF_LIST)
|
|
result = subprocess.run(["bash", str(SCRIPT), "42", "MERGEABLE"], env=env, capture_output=True, text=True)
|
|
|
|
assert result.returncode != 0, (
|
|
f"an environment variable steered the allow-list: {OFF_LIST} was accepted as a reviewer"
|
|
)
|
|
assert OFF_LIST in result.stderr and "H10_REVIEWERS" in result.stderr
|
|
assert gitea.comments() == []
|
|
|
|
|
|
def test_a_REJECTION_survives_a_workflow_whose_allowlist_cannot_be_derived(gitea, tmp_path):
|
|
"""Membership never applies to a `failure`, so a broken `H10_REVIEWERS` declaration must not
|
|
block one. It used to: the derivation was loaded unconditionally, so a workflow with two
|
|
assignments refused REJECTIONS as well — the exact outcome the success-only rule exists to avoid
|
|
("no supported way to record a rejection"), one condition earlier, and on the branch most likely
|
|
to have broken that declaration: the one editing it.
|
|
"""
|
|
script = _sandbox(tmp_path, ALLOWLISTED)
|
|
workflow = script.parent.parent / ".gitea" / "workflows" / "review-verdict.yml"
|
|
body = workflow.read_text()
|
|
marker = f'H10_REVIEWERS="{ALLOWLISTED}"'
|
|
assert body.count(marker) == 1
|
|
workflow.write_text(body.replace(marker, f'{marker}\n H10_REVIEWERS="someone-else"'))
|
|
|
|
gitea.set_status_creator(OFF_LIST)
|
|
result = subprocess.run(
|
|
["bash", str(script), "42", "BLOCKED", "findings outstanding"],
|
|
env=gitea.env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
|
|
assert result.returncode == 0, f"a rejection was blocked by an allow-list it does not depend on: {result.stderr}"
|
|
assert len(gitea.comments()) == 1
|
|
|
|
# And the positive direction still refuses on the same tree, so the guard was narrowed, not removed.
|
|
(gitea.state / "posts.jsonl").unlink(missing_ok=True)
|
|
(gitea.state / "pr_get_count").unlink(missing_ok=True)
|
|
approval = subprocess.run(["bash", str(script), "42", "MERGEABLE"], env=gitea.env, capture_output=True, text=True)
|
|
assert approval.returncode != 0
|
|
assert gitea.statuses() == [], "an underivable allow-list must refuse an APPROVAL before writing"
|