Files
ersatztv/docs/decisions/records/release/review-verdict-gate.md
T
timothyandClaude Opus 5 50bcd7b0c7
Review verdict / Set review-verdict status (pull_request) Successful in 2s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 15s
PR Gates / Docs update reminder (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Successful in 18s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m7s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 10s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
review-verdict/h10 Review-verdict: MERGEABLE @ 50bcd7b
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 19m51s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 25m5s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
fix(629): strip raw HTML blocks, and state where the hardening stops
Round 5: raw HTML is the third code-block form. `<pre>`, `<code>` and HTML comments all render their
contents literally, so a verdict inside one is an example, not an approval:

  <pre> / <code> / <!-- ... -->  containing a verdict  ->  positive

Now stripped, tracked as a marker count rather than parsed — the direction of error is to strip MORE,
which can only ever withhold approval. Mutation-verified: removing the stripper fails all five cases.

AND THE HARDENING STOPS HERE, deliberately. The record now says so, because otherwise the next
session re-derives it: this is a best-effort heuristic, not a markdown parser. It covers the three
code-block forms markdown has (fenced; indented, via the column-0 rule; raw HTML) and is not proof
against every way to render text as non-prose.

Stopping is safe because the comment is NOT the load-bearing gate. Since #622 the authoritative
signal is the `review-verdict/h10` commit status, written only by post-review-verdict.sh from
explicit arguments — a comment cannot forge it. This classifier is condition (c) of the PreToolUse
hook: defense in depth on an agent's merge call. A residual false-open means the hook does not
object; it does not mean a merge happens.

Five rounds found five code-block forms, four of them introduced while fixing the previous round.
The generalisable rule, now in the record: when a heuristic keeps failing at the edges, check whether
it is actually the thing enforcing the invariant before spending another round on it.

Also measured, against the real corpus: a "verdict must be the first line" rule would have killed
every code-block form at once, but 14 of 18 verdict markers ever posted in this repo are NOT on the
first line — so it was rejected as a retroactive break, not deferred.

178 tests. refs #629

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Decisions-Edit: yes
2026-07-26 01:14:59 +02:00

11 KiB
Raw Blame History

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
release.review-verdict-gate 2026-07-12 — Review-verdict merge-gate: latest commit must be reviewed (#303 H10) active 2026-07-12 none none A PR may not merge until a `Review-verdict: <MERGEABLE|APPROVED|BLOCKED|NOT-MERGEABLE> @ <head-sha>` comment references the PR's current head sha (short-sha prefix match against the verdict's OWN `@ <sha>` field, marker at COLUMN 0 (no indent, so indented code blocks cannot self-approve), whole-word verdict token, fenced code blocks stripped with markdown fence-length semantics, negative wins over positive on the same head); folds into the H6 merge-consent hook as condition (c). The grammar lives in ONE tested place, `scripts/check-review-verdict.sh` — #629 found three false-opens that survived because it was implemented inline and untested while this record described stricter behaviour than the code had. review-verdict, head-sha match, stale-review prevention, verdict false-open, MERGEABLE-LATER, fenced code block verdict, sha from a URL, unknown verdict token · paths: `scripts/check-review-verdict.sh`, `.claude/hooks/pretooluse-merge-consent.sh`, `.claude/settings.json` · issues: #303 (H10), #242, #629 `scripts/check-review-verdict.sh` (the grammar, + `scripts/tests/test_check_review_verdict.py`); `pretooluse-merge-consent.sh` (maps a class onto allow/deny/ask); `scripts/post-review-verdict.sh`; CLAUDE.md → Task Completion Protocol (H10 convention)

A PR may not merge until a Review-verdict: comment on it references the PR's CURRENT head sha — so the latest commit is proven-reviewed, not a stale earlier diff. This mechanizes the ersatztv#242 lesson ("re-review the fix commit, not just the initial PR diff": a review of an earlier revision does not license merging a head that carries un-reviewed follow-up commits). It folds into the existing H6 pretooluse-merge-consent.sh as condition (c), reusing its PR fetch, docs-only exemption, and Gitea-auth-from-env (no second hook → no detection drift, per the #303 methodology review).

Convention: after reviewing a PR (or its latest fix commit), post a PR comment (issue-style, not a Gitea formal-review body — the gate reads issues/{pr}/comments) whose line starts with the marker: Review-verdict: <MERGEABLE|APPROVED|BLOCKED|NOT-MERGEABLE> @ <head-sha> (short ≥7-char or full sha). The gate counts a line as a verdict only when the marker is at line-start (after optional indent) — a comment that merely quotes the template mid-sentence (an instruction "please post: Review-verdict: MERGEABLE @ …", or the gate's own suggestion text echoed back) does not self-approve the merge (adversarial re-review false-open, folded pre-merge). It then classifies each verdict line by the sha in its @ <sha> field, matched to the head by git short-sha prefix semantics (head begins with the token, token ≥7 chars) — NOT a loose substring test, so an older sha that merely contains the head prefix does not count.

#629 — three of the protections described here were asserted but not implemented. The grammar lived inline in the hook with no tests, and each of these graded as a positive verdict until #629 (every one reproduced, then fixed, then mutation-verified):

  • the verdict token was prefix-matched, so MERGEABLE-LATER, APPROVED-PENDING-QA and LGTMish all read as positive. A token is now matched as a whole word, and one in neither vocabulary is classified unknown — never positive, and never guessed into a block either;
  • a verdict inside a fenced code block counted, because the line-start anchor is satisfied inside a fence. So documentation showing the convention was itself a verdict. Fenced blocks are now stripped, with fence state reset per comment body (blockquotes never needed handling — a > prefix already fails the anchor);
  • "the head prefix appearing in an unrelated URL on the line does not count" — this paragraph's own earlier claim — was false. The implementation took the first @<hex> anywhere on the line, so Review-verdict: MERGEABLE [x](https://e/@0123456) was graded against the link. The sha is now read from the verdict's own @ <sha> field, the one immediately following the token, which also makes multi-@ lines unambiguous.

A cross-family review of that first fix found three more, all reproduced before fixing — worth recording because each is the same fix done half-way:

  • fences were stripped for ``` only, but markdown also accepts ~~~;
  • the sha field matched {7,40} with no right boundary, so an over-long or malformed token was silently truncated into a passing one: @ <40-hex-head>f and @ <40-hex-head>ZZZ both graded as a verdict for head. The hex run is now matched whole, must end at a non-alphanumeric boundary, and its length is validated separately — an out-of-range token is rejected, never trimmed to fit;
  • fence stripping toggled on any line of three-or-more markers. Markdown closes a fence only with N-or-more of the same marker it was opened with, so a ```` block legitimately contains a
  • comment bodies were joined with a literal \x01BODY-BOUNDARY\x01 line so fence state could reset per comment. An in-band delimiter is forgeable by whoever writes the data — and here that is anyone who can comment on the PR: a body containing that line reset the fence mid-comment and exposed a verdict still inside an unclosed fence. Bodies are now carried out-of-band (one JSON-encoded string per line), which removes the class rather than escaping the sentinel.

A fourth round found the last one, and it is why the grammar was narrowed rather than patched again: markdown has a second code-block form — indented blocks (4 spaces or a tab) — which the fence stripper does not cover, so a pasted indented example still self-approved. Rather than add a second stripper (the same instance-not-class move that produced the previous three rounds), the marker must now sit at COLUMN 0. That removes every indentation-based ambiguity at once. The cost is that a verdict indented under a list item is ignored and classifies absent — which asks a human, the safe direction for a gate. Fence detection keeps its leading-whitespace tolerance, because stripping more is always safe.

Four review rounds, four sets of real findings, and rounds 24 each found a bug introduced while fixing the round before — every one the same fix applied half-way (one fence marker but not the other; a bounded left side but not the right; state reset per comment but via a delimiter the writer controls). For attacker-writable text, budget several rounds and prefer eliminating a class over enumerating its members.

A fifth round found raw HTML — <pre>, <code>, HTML comments — the third code-block form. It is stripped too, and hardening stopped there, deliberately.

Scope limit, stated so nobody re-derives it: the comment scanner is a best-effort heuristic, not a markdown parser. It handles the three code-block forms markdown actually has (fenced, indented via the column-0 rule, raw HTML) and errs toward stripping more, because every error in that direction can only withhold approval. It is not proof against every conceivable way to render text as non-prose, and chasing that was demonstrably not converging: five review rounds, five code-block forms, four of them introduced while fixing the previous one.

Stopping is safe because the comment is not the load-bearing gate. Since #622 the authoritative signal is the review-verdict/h10 commit status, written only by scripts/post-review-verdict.sh from explicit arguments — a comment cannot forge it, whatever it contains. The comment classifier is condition (c) of the PreToolUse hook, i.e. defense in depth on an agent's merge call. A residual false-open there means the hook does not object; it does not mean a merge happens.

The rule that generalises: when a heuristic keeps failing at the edges, check whether it is actually the thing enforcing the invariant before spending another round on it.

The lesson is about where a grammar lives, not about any one regex: this record described the intended behaviour accurately, the code did something looser, and nothing compared them. The grammar now lives in scripts/check-review-verdict.sh — one implementation, called by the hook and by post-review-verdict.sh's cross-check, covered by scripts/tests/test_check_review_verdict.py.

Two of those fixes also broke previously-green tests, which is the useful part: an over-long token became no-sha where a fixture expected stale (the fixture was 45 hex chars, so it had been testing the length guard while claiming to test the prefix rule), and jq -e exits 4 on no output, so an empty comment list started reading as a malformed payload — turning "no comments yet" into an input error on a gate whose callers fail closed.

The classifications:

  • a MERGEABLE/APPROVED/LGTM verdict whose @ <sha> is the current head → allow;
  • a negative verdict (BLOCKED/NOT-MERGEABLE) on the headdeny, and it wins over a positive one on the same head (a later BLOCKED retracts an earlier MERGEABLE; to retract, re-review head and post BLOCKED @ head). Staleness is symmetric on purpose: a negative for an older commit is stale exactly like a positive for an older commit, and does NOT override a fresh head-positive — otherwise a pre-fix BLOCKED @ oldsha would block forever even after the fix changes the sha and earns a fresh MERGEABLE @ head (the normal flow). So a genuine block must reference head, per the convention;
  • verdict comment(s) exist but reference only older commits → deny — the stale-review case #242 targets;
  • a verdict line whose token is in neither vocabulary (MERGEABLE-LATER, SHIP-IT, …) → ask (#629). Deliberately not read as approval, and deliberately not read as a block either — an unrecognized token means the reviewer's intent is unknown, so it goes to a human;
  • a Review-verdict: marker with no @ <sha> in its own field → ask (a lazy/quoted marker; not mislabelled as stale);
  • no Review-verdict: comment at all → ask (graceful adoption, mirrors H6's "no Done-when → ask": surface, don't hard-block a PR that hasn't adopted the convention yet);
  • comments unfetchable / head sha unresolvable → ask.

Scope: the Claude PreToolUse gate on the Gitea merge tool only. A direct git push origin main has no PR comments to check, so the .husky/pre-push backstop is not extended for H10 (the merge tool is the real merge path; docs-only PRs remain exempt via H6's file-set exemption). Rationale, as with the whole Wave-1/2/3 hook set: make the process rule a derivation/hook, not prose to remember (#303).