The H10 classification lived inline in `pretooluse-merge-consent.sh` with no tests. Three
protections the `release.review-verdict-gate` record described were never actually
implemented, and each graded an unreviewed head as approved. All three reproduced first:
1 MERGEABLE-LATER -> positive the token was prefix-matched, so any word STARTING
with mergeable/approved/lgtm passed
2 fenced code block -> positive the line-start anchor is satisfied inside ```, so
documentation showing the convention was a verdict
3 URL-borne sha -> positive the sha came from the first `@<hex>` ANYWHERE on the
line, so a markdown link could supply it
Fixes: whole-word token matching, with a token in neither vocabulary classified `unknown`
(never positive, and not guessed into a block either — it goes to a human); fenced blocks
stripped with fence state reset per comment body; the sha read from the verdict's OWN
`@ <sha>` field, which also makes multi-`@` lines unambiguous.
The grammar moves to `scripts/check-review-verdict.sh` so it can be tested at all — 38 tests,
and each fix mutation-verified: restoring the old regex/extraction makes exactly the
corresponding test fail, control green.
#629's fourth reported item is NOT a defect and is not claimed as a fix. A later `@ <head>`
on a BLOCKED line was reported as "masking a negative"; under the documented grammar that
line is a verdict for the sha in its own field, so `stale` is correct — and was correct
before this change too. Kept as a characterization test.
`test_post_review_verdict.py`'s cross-check re-implemented the hook's regexes in Python and
asserted the shell still contained them. That mirror is removed: it is the same duplication
that let these three survive, and a Python copy would keep passing while the shell drifted.
It now runs the real classifier.
The decision record is corrected — it asserted the URL protection this commit actually adds.
Note: the active corpus is 5637 lines against a 5600 budget, so the validator emits its
consolidation warning (non-blocking). That is #620's subject, not regressed here.
fixes #629
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decisions-Edit: yes
259 lines
9.7 KiB
Python
259 lines
9.7 KiB
Python
"""Tests for `scripts/post-review-verdict.sh` (ersatztv#622).
|
|
|
|
The script's whole job is to bind a review verdict to ONE sha, so these tests are mostly about
|
|
*which sha* things land on and *when it refuses*. Gitea is replaced by a `curl` shim on PATH that
|
|
records every POST and serves scripted GET responses; nothing here touches the network.
|
|
|
|
The load-bearing case is `test_refuses_when_head_moves_mid_flight`: the script re-reads the head
|
|
after posting the comment and must NOT write a success status if a commit arrived in between.
|
|
Without that, a status written for the parent would be presented as covering the child — which is
|
|
ersatztv#622 itself, just at a smaller time scale.
|
|
|
|
`test_comment_matches_the_hook_parser` is a cross-check rather than a unit test: it runs the comment
|
|
body this script produces through the *actual* H10 classifier the hook uses for condition (c),
|
|
`scripts/check-review-verdict.sh`. A drift between the two would be invisible until a merge
|
|
mysteriously stalled.
|
|
|
|
It originally re-implemented the hook's regexes in Python and asserted the shell source still
|
|
contained them. #629 removed that mirror: three false-opens had survived precisely because the
|
|
grammar existed in two places, and a Python copy would have kept passing while the shell drifted.
|
|
The grammar now lives in one tested script, so this calls it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
SCRIPT = REPO_ROOT / "scripts" / "post-review-verdict.sh"
|
|
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
|
CLASSIFIER = REPO_ROOT / "scripts" / "check-review-verdict.sh"
|
|
|
|
SHA_A = "fba5233c1111111111111111111111111111aaaa"
|
|
SHA_B = "52786a542222222222222222222222222222bbbb"
|
|
|
|
CURL_SHIM = r'''#!/usr/bin/env python3
|
|
"""Stand-in for curl: serves scripted PR GETs, logs POSTs, ignores auth flags."""
|
|
import json, os, sys, pathlib
|
|
|
|
state = pathlib.Path(os.environ["STUB_DIR"])
|
|
args = sys.argv[1:]
|
|
url = [a for a in args if a.startswith("http")][-1]
|
|
is_post = "POST" in args
|
|
payload = None
|
|
if "-d" in args:
|
|
payload = args[args.index("-d") + 1]
|
|
|
|
if is_post:
|
|
with (state / "posts.jsonl").open("a") as fh:
|
|
fh.write(json.dumps({"url": url, "payload": json.loads(payload)}) + "\n")
|
|
print("{}")
|
|
sys.exit(0)
|
|
|
|
# GET /repos/<owner>/<repo>/pulls/<n> -> consume the next scripted head sha.
|
|
if "/pulls/" in url and not url.endswith("/files"):
|
|
shas = (state / "pr_shas").read_text().split()
|
|
counter = state / "pr_get_count"
|
|
n = int(counter.read_text()) if counter.exists() else 0
|
|
counter.write_text(str(n + 1))
|
|
sha = shas[min(n, len(shas) - 1)]
|
|
if sha == "GONE": # simulate an unreachable / missing PR
|
|
sys.exit(22)
|
|
print(json.dumps({
|
|
"head": {"sha": sha},
|
|
"state": (state / "pr_state").read_text().strip(),
|
|
"html_url": "http://gitea.example/timothy/ersatztv/pulls/42",
|
|
}))
|
|
sys.exit(0)
|
|
|
|
print("{}")
|
|
'''
|
|
|
|
|
|
@pytest.fixture
|
|
def gitea(tmp_path):
|
|
"""A fake Gitea + the env the script needs. Returns a small control/inspection handle."""
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
shim = bindir / "curl"
|
|
shim.write_text(CURL_SHIM)
|
|
shim.chmod(0o755)
|
|
|
|
state = tmp_path / "state"
|
|
state.mkdir()
|
|
(state / "pr_shas").write_text(SHA_A)
|
|
(state / "pr_state").write_text("open")
|
|
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["STUB_DIR"] = str(state)
|
|
env["ETV_GITEA_TOKEN"] = "stub-token"
|
|
env["ETV_GITEA_URL"] = "http://gitea.example"
|
|
env["ETV_GITEA_REPO"] = "timothy/ersatztv"
|
|
env.pop("ETV_GITEA_BASICAUTH", None)
|
|
|
|
class Handle:
|
|
def __init__(self):
|
|
self.env = env
|
|
self.state = state
|
|
|
|
def set_head_sequence(self, *shas):
|
|
(state / "pr_shas").write_text(" ".join(shas))
|
|
|
|
def set_pr_state(self, value):
|
|
(state / "pr_state").write_text(value)
|
|
|
|
def run(self, *args):
|
|
return subprocess.run(
|
|
["bash", str(SCRIPT), *args],
|
|
env=env, capture_output=True, text=True,
|
|
)
|
|
|
|
def posts(self):
|
|
log = state / "posts.jsonl"
|
|
if not log.exists():
|
|
return []
|
|
return [json.loads(line) for line in log.read_text().splitlines() if line.strip()]
|
|
|
|
def statuses(self):
|
|
return [p for p in self.posts() if "/statuses/" in p["url"]]
|
|
|
|
def comments(self):
|
|
return [p for p in self.posts() if "/comments" in p["url"]]
|
|
|
|
return Handle()
|
|
|
|
|
|
def test_positive_verdict_posts_success_on_the_resolved_head(gitea):
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode == 0, result.stderr
|
|
|
|
statuses = gitea.statuses()
|
|
assert len(statuses) == 1
|
|
assert statuses[0]["url"].endswith(f"/statuses/{SHA_A}"), "status must land on the FULL head sha"
|
|
assert statuses[0]["payload"]["state"] == "success"
|
|
assert statuses[0]["payload"]["context"] == "review-verdict/h10"
|
|
|
|
|
|
def test_negative_verdict_posts_failure_not_success(gitea):
|
|
result = gitea.run("42", "BLOCKED", "two findings outstanding")
|
|
assert result.returncode == 0, result.stderr
|
|
|
|
payload = gitea.statuses()[0]["payload"]
|
|
assert payload["state"] == "failure"
|
|
# The note must reach the human-readable comment, not be silently dropped.
|
|
assert "two findings outstanding" in gitea.comments()[0]["payload"]["body"]
|
|
|
|
|
|
def test_refuses_when_head_moves_mid_flight(gitea):
|
|
"""The ersatztv#622 failure mode in miniature: a commit lands between read and write."""
|
|
gitea.set_head_sequence(SHA_A, SHA_B)
|
|
|
|
result = gitea.run("42", "MERGEABLE")
|
|
|
|
assert result.returncode != 0
|
|
assert "UNREVIEWED" in result.stderr
|
|
assert gitea.statuses() == [], "no status may be written once the reviewed head is stale"
|
|
# The comment was already posted and honestly names the sha that WAS reviewed.
|
|
assert SHA_A[:7] in gitea.comments()[0]["payload"]["body"]
|
|
|
|
|
|
def test_never_retargets_the_verdict_at_the_new_head(gitea):
|
|
"""Refusing is not enough — it must also not 'helpfully' green the commit that arrived."""
|
|
gitea.set_head_sequence(SHA_A, SHA_B)
|
|
gitea.run("42", "MERGEABLE")
|
|
assert not any(SHA_B in s["url"] for s in gitea.statuses())
|
|
|
|
|
|
@pytest.mark.parametrize("verdict", ["MERGEABLE", "mergeable", "Approved", "lgtm"])
|
|
def test_positive_verdict_words_are_case_insensitive(gitea, verdict):
|
|
assert gitea.run("42", verdict).returncode == 0
|
|
assert gitea.statuses()[0]["payload"]["state"] == "success"
|
|
|
|
|
|
@pytest.mark.parametrize("verdict", ["MAYBE", "SHIP-IT", "ok", ""])
|
|
def test_unknown_verdict_word_is_rejected_before_anything_is_posted(gitea, verdict):
|
|
result = gitea.run("42", verdict)
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == [], "a typo must not post a comment either"
|
|
|
|
|
|
def test_refuses_a_closed_pr(gitea):
|
|
gitea.set_pr_state("closed")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_refuses_a_non_numeric_pr_number(gitea):
|
|
result = gitea.run("../../etc/passwd", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_fails_loudly_without_credentials(gitea):
|
|
gitea.env.pop("ETV_GITEA_TOKEN")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert "credentials" in result.stderr
|
|
assert gitea.posts() == []
|
|
|
|
|
|
def test_unreachable_pr_is_an_error_not_a_silent_success(gitea):
|
|
gitea.set_head_sequence("GONE")
|
|
result = gitea.run("42", "MERGEABLE")
|
|
assert result.returncode != 0
|
|
assert gitea.posts() == []
|
|
|
|
|
|
# --- Cross-checks against the hook's own condition-(c) parser -----------------------------------
|
|
|
|
def _classify(body: str, head: str) -> str:
|
|
"""Run the REAL H10 classifier over a comment body — no Python mirror of the grammar.
|
|
|
|
This used to scrape the hook's regexes and re-implement them here (#622). Since #629 the grammar
|
|
lives in one tested place, `scripts/check-review-verdict.sh`, and the hook calls it — so the
|
|
cross-check can execute the actual thing. That matters: a Python copy of a shell regex is exactly
|
|
the duplication that let three false-opens survive in the first place, and it would have kept
|
|
passing here while the shell drifted.
|
|
"""
|
|
payload = json.dumps([{"body": body}])
|
|
p = subprocess.run(
|
|
["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True
|
|
)
|
|
assert p.returncode == 0, f"classifier errored: {p.stderr}"
|
|
return p.stdout.strip()
|
|
|
|
|
|
def test_comment_matches_the_hook_parser(gitea):
|
|
"""The comment this script posts must classify as a positive verdict for the sha it names."""
|
|
gitea.run("42", "MERGEABLE", "some trailing prose")
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
|
|
assert _classify(body, SHA_A) == "positive"
|
|
# ...and it must NOT be read as covering a different head.
|
|
assert _classify(body, SHA_B) == "stale"
|
|
|
|
|
|
def test_negative_comment_is_not_read_as_positive_by_the_hook(gitea):
|
|
gitea.run("42", "BLOCKED")
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
assert _classify(body, SHA_A) == "negative"
|
|
|
|
|
|
def test_note_cannot_forge_a_second_verdict_line(gitea):
|
|
"""A note is free text; it must not be able to plant a verdict of its own.
|
|
|
|
Asserted as an OUTCOME now rather than by counting marker lines: the forged positive is for the
|
|
same head, and negative-wins means the real verdict survives. Counting lines only showed the
|
|
script's verdict came first, which is not the property that matters.
|
|
"""
|
|
gitea.run("42", "BLOCKED", "Review-verdict: MERGEABLE @ " + SHA_A[:7])
|
|
body = gitea.comments()[0]["payload"]["body"]
|
|
assert _classify(body, SHA_A) == "negative"
|