test(742): pin the annotation, each triple clause, and multi-entry membership

Round-3 cold tests/prose review returned BLOCKED. Its findings were all of one shape: the
clauses are correct, but the suite could not tell.

Measured green with the mechanism DELETED, now each reddens on exactly one test:
* deleting the ::warning:: entirely — 166 green. It is not decoration: docs/ci-cd.md heads a
  paragraph with 'If a reviewer's verdicts stop being honoured, this is why', and
  ci.exemption-provenance uses its existence to soften the #845 residual. Asserted present
  in the off-list case, and ABSENT for an allow-listed reviewer, which also pins the
  ex_human != yes conjunct.
* dropping any ONE clause of the changed-row triple, or reducing the triple to a single
  comparison. Both mid-run tests moved all three fields at once, so 'compare the whole row'
  was argued, not measured. STUB_MIDRUN_ROW plus one parametrised case per field.
* a membership loop that can only ever match its LAST entry. Every behavioural test ran the
  singleton list, under which that is indistinguishable from a correct loop — and the
  annotation this change adds tells an operator to ADD a second entry, so the two-entry path
  is the first one exercised in anger. A reviewers= override substitutes a two-entry literal
  into the real step body; first-entry, last-entry control and absent-account control.

Prose, all measured by the reviewer rather than argued:
* 'was inherited as a human verdict' described an event nobody observed — renovate has never
  posted a commit status in this repo (every row on all six Renovate-authored PR heads carries
  creator: null). The asymmetry underneath IS measured; the ROUTE is an inference. Conditional
  voice in all three places, with the distinction stated.
* the ::warning:: does not fire for a row recording a DIFFERENT base — the base binding
  re-derives it before the annotation is reached. 'whenever' was false.
* a credential drift re-derives to an exemption success on an exempt PR, not only to pending —
  the quieter and worse half.
* the live account-set measurement now carries its two curl paths.
* CLAUDE.md states the H10 convention and now carries the constraint: the credential you post
  with must be an allow-listed account, or the verdict is written, reported as success, and
  silently re-derived (#845).
* residual session chronology trimmed from the record.

STUB_PRE_ROW now emits the measured shape (total_count, a decoy context) rather than a
minimal one.

scripts/tests: 976 passed, 2 skipped.

fixes #742

Decisions-Edit: yes
This commit is contained in:
2026-08-27 00:07:58 +02:00
parent 2ed3e1895a
commit 51338b3cb3
5 changed files with 176 additions and 22 deletions
+8 -1
View File
@@ -240,7 +240,14 @@ jobs:
# CANNOT be scoped down the way `#697` scoped the registry credential (Renovate genuinely
# needs repo write to push branches and open PRs), and secrets are a per-repo STORE rather
# than a per-workflow grant — so a status POSTed with it reads back `creator: renovate`,
# non-null, and was inherited as a human verdict.
# non-null, and would be inherited as a human verdict.
#
# CONDITIONAL VOICE ON PURPOSE. The asymmetry it rests on IS measured (a user credential
# posts with a real creator, an Actions job with `creator: null`), but it has not been
# measured FOR THIS TOKEN: every status row on all six Renovate-authored PR heads in this
# repo carries `creator: null` — `renovate` has never posted a commit status here. So the
# route is a sound inference from a measurement, not an observed event, and an earlier
# draft's past tense ("was inherited") described something nobody saw.
#
# Membership keeps the POSITIVE direction the description test already uses: short-circuit
# only on something identified as a REVIEWER's verdict; every other creator, recognised or
+1 -1
View File
@@ -83,7 +83,7 @@ main in) and re-run the local gate whenever the fetch shows movement.
Every task that closes a Gitea issue MUST complete ALL of these before it is considered done. Use `/done <issue>` to run through this automatically.
**Merge-consent is derived from state, not asserted (`## Done-when` convention — ersatztv#303 H6 + H10).** Any issue whose PR will merge to `main` should carry a `## Done-when` section in its **issue body** — a checklist of completion criteria (always include an "adversarial review passed" box; add per-issue criteria like tests-green, docs-updated, live-E2E). Two hooks derive merge-consent from it so a premature merge is blocked *by construction*, not by memory:
- `pretooluse-merge-consent.sh` (Claude PreToolUse on the Gitea merge tool) — **auto-grants** a merge (emits `permissionDecision: allow`, so **no** redundant mechanical prompt fires) only when the PR's CI is green **and** every `## Done-when` box on the linked issue (`fixes #N`) is ticked **and** a `Review-verdict:` comment references the PR's *current head sha* (**H10**); **denies** on an unticked box, red CI, or a stale/negative review verdict; **asks** (falls back to a human prompt) when it can't derive state (no linked issue, no `## Done-when` section, no `Review-verdict:` comment yet, no creds, Gitea down). On the auto-grant (satisfied) path the derived state **is** the consent — do not also ask conversationally to merge; a separate human confirmation is warranted only when the gate **asks** (ersatztv#314). **The H10 review-verdict convention**: after an adversarial/Codex review of a PR (or its latest fix commit), run **`scripts/post-review-verdict.sh <pr> <MERGEABLE|APPROVED|LGTM|BLOCKED|NOT-MERGEABLE> [note]`** — it posts both the `Review-verdict: … @ <head-sha>` comment and the sha-bound `review-verdict/h10` commit status, proving the *latest* commit was reviewed rather than a stale earlier diff (ersatztv#242). Do not hand-write the comment: the **status** is the required check branch protection enforces, and a comment alone leaves it absent.
- `pretooluse-merge-consent.sh` (Claude PreToolUse on the Gitea merge tool) — **auto-grants** a merge (emits `permissionDecision: allow`, so **no** redundant mechanical prompt fires) only when the PR's CI is green **and** every `## Done-when` box on the linked issue (`fixes #N`) is ticked **and** a `Review-verdict:` comment references the PR's *current head sha* (**H10**); **denies** on an unticked box, red CI, or a stale/negative review verdict; **asks** (falls back to a human prompt) when it can't derive state (no linked issue, no `## Done-when` section, no `Review-verdict:` comment yet, no creds, Gitea down). On the auto-grant (satisfied) path the derived state **is** the consent — do not also ask conversationally to merge; a separate human confirmation is warranted only when the gate **asks** (ersatztv#314). **The H10 review-verdict convention**: after an adversarial/Codex review of a PR (or its latest fix commit), run **`scripts/post-review-verdict.sh <pr> <MERGEABLE|APPROVED|LGTM|BLOCKED|NOT-MERGEABLE> [note]`** — it posts both the `Review-verdict: … @ <head-sha>` comment and the sha-bound `review-verdict/h10` commit status, proving the *latest* commit was reviewed rather than a stale earlier diff (ersatztv#242). Do not hand-write the comment: the **status** is the required check branch protection enforces, and a comment alone leaves it absent. **The credential you post with must be an account on `H10_REVIEWERS` in `.gitea/workflows/review-verdict.yml`** (`timothy` today) — since ersatztv#742 the gate inherits an existing `success` only from an allow-listed creator, so a verdict posted with any other account is written, reported as success by the script, and then silently re-derived on the next PR event (ersatztv#845).
- **The gate is enforced server-side, per sha (ersatztv#622).** `review-verdict/h10` is a required status check on `main`. Because a commit status belongs to one sha, a commit pushed *after* an auto-merge is scheduled clears it and blocks the merge — closing the hole where `merge_when_checks_succeed` froze consent at scheduling time and Gitea later merged an unreviewed head. Renovate-authored and docs-only PRs are auto-passed by `.gitea/workflows/review-verdict.yml`, **except** when they touch `.claude/`, `.codex/`, `.gitea/`, `.husky/`, `scripts/` or `docker/ci/`. See `docs/ci-cd.md` → Review-verdict gate.
- `.husky/pre-push` → `prepush-donewhen.sh` — a fail-open backstop that blocks a direct `git push origin main` whose commits `fix #N` an issue with unticked boxes. **Since ersatztv#743 that push can no longer happen at all** (see below), so this hook is now belt-and-braces for a path the server refuses.
+11 -5
View File
@@ -1308,7 +1308,10 @@ for accounts that are not reviewers. `RENOVATE_TOKEN` is a `write:repository` PA
bot; unlike the registry credential it **cannot be scoped down** (Renovate needs repo write to push
branches and open PRs), and secrets are a per-repo *store* rather than a per-workflow grant, so a
PR-added workflow can reference it and POST a `Review-verdict:` success that reads back
`creator: renovate` — non-null, and therefore inherited. Membership makes every non-reviewer account
`creator: renovate` — non-null, and therefore inheritable. (Inferred, not observed: `renovate` has
never actually posted a commit status in this repo — every row on all six Renovate-authored PR heads
carries `creator: null`. The user-credential-vs-Actions-token asymmetry underneath it IS measured.)
Membership makes every non-reviewer account
re-derive exactly as an Actions-token status does. The list is a **literal in the workflow**, not a
repo `vars:` entry or a secret: those are settable without a PR, i.e. without this gate, while
`.gitea/` is on the PROTECTED path list so the edit cannot ride a docs-only exemption either.
@@ -1345,13 +1348,16 @@ Pinned by `test_the_MID_RUN_abstain_still_fires_for_a_NON_allowlisted_creator` a
`test_the_RACED_verdict_check_still_counts_a_NON_allowlisted_creator`.
**If a reviewer's verdicts stop being honoured, this is why.** The job emits a `::warning::` naming the
creator whenever it re-derives over a verdict-shaped status from an account that is not on the list;
creator whenever it re-derives over a verdict-shaped status **for this base** from an account not on
the list — a row recording a *different* base is re-derived by the base binding before the annotation
is reached, and gets no warning;
add them to `H10_REVIEWERS` in `.gitea/workflows/review-verdict.yml`. `post-review-verdict.sh` posts
with whatever account owns `ETV_GITEA_TOKEN`/`ETV_GITEA_BASICAUTH` and does **not** check that it is an
allow-listed one (ersatztv#845). A changed writer credential does not fail loudly — it reports success,
and each *subsequent* PR event re-derives the verdict to `pending`; a verdict posted after the last
event stands until the next one, so the symptom is a PR that keeps losing its verdict rather than an
immediate refusal.
and each *subsequent* PR event re-derives the status: to `pending` on a PR that needs a verdict, or to
an exemption **`success`** on a docs-only or Renovate-manifest one, which is the quieter and worse
half. A verdict posted after the last event stands until the next one, so the symptom is a PR that
keeps losing its verdict rather than an immediate refusal.
This remains a *provenance* check, not an authentication one — an **allow-listed** collaborator's own
write-scoped token can still POST a verdict they did not form, branch protection binds the context
@@ -122,7 +122,11 @@ POSTed with a user credential", and *any* account's credential answered yes. `RE
`write:repository` PAT of the `renovate` bot: unlike `#697`'s registry credential it **cannot** be
scoped off status-write, because Renovate genuinely needs repo write to push branches and open PRs,
and secrets are a per-repo *store* rather than a per-workflow grant — so a workflow can reference it
and POST a `Review-verdict:` success that reads back `creator: renovate`, non-null, and was inherited.
and POST a `Review-verdict:` success that reads back `creator: renovate` — non-null, and therefore
inheritable. **Inferred from a measurement, not an observed event:** the user-credential-vs-Actions-token
asymmetry is measured, but `renovate` has never actually posted a commit status in this repo — every
status row on all six Renovate-authored PR heads carries `creator: null`. An earlier draft's past tense
("was inherited") described something nobody saw.
Membership in `H10_REVIEWERS` restores the intended set: every other account, recognised or not,
re-derives exactly as an Actions-token status already does. The list is a literal in this
base-resolved definition rather than a repo `vars:` entry, because a `vars:` entry is settable without
@@ -151,8 +155,7 @@ comparison made that load-bearing where presence alone had not: two POSTs of con
`probe742/combined-newest` to one sha (`pending`, then `success`) came back as ONE row on
`/commits/{sha}/status`, the newest, and as BOTH rows on `/statuses/{sha}`.
**The allow-list governs `success` ONLY. Treating the two states alike is a fail-open, and this branch
carried it for a round.** Trace an off-list `failure` under a symmetric rule: a real reviewer left off
**The allow-list governs `success` ONLY. Treating the two states alike is a fail-open.** Trace an off-list `failure` under a symmetric rule: a real reviewer left off
the literal posts `BLOCKED`; `ex_human=no`, so the short-circuit declines; the PR is docs-only, so the
classifier exempts it; the re-read triple never changed, so the mid-run abstain declines; the row
predates the high-water mark, so the post-write repair cannot see it — and the job posts `success` over
@@ -164,8 +167,7 @@ which uses a DOCS-ONLY file list on purpose — with a code file the job posts `
test would pass measuring nothing.
**Why the post-write raced check stays broad — and NOT for the reason first written.** The first draft
justified it as "narrowing it would leave an exemption success standing over a rejection". Cold review
showed that is not a checked reason: a genuine reviewer is on `H10_REVIEWERS` by construction, so
justified it as "narrowing it would leave an exemption success standing over a rejection". That is not a checked reason: a genuine reviewer is on `H10_REVIEWERS` by construction, so
membership there would still count every real verdict, and an off-list row is one the job deliberately
overwrote a few lines earlier. What is left once `failure` is handled at the short-circuit is a
NARROWER net than an earlier draft claimed: it covers a misconfigured reviewer's verdict landing inside
@@ -189,9 +191,9 @@ would falsify it silently.
whatever account owns `ETV_GITEA_TOKEN`/`ETV_GITEA_BASICAUTH` and never checks whose it is. If that
ever stops being an allow-listed account, every verdict silently stops being inheritable and each
later event re-posts `pending` over it — a deadlocked PR whose only diagnostic is a line in a job log.
Correct today (verified live 2026-08-26: the token's `/user` login is `timothy`, and the instance has
exactly the accounts `timothy` and `renovate`), but nothing keeps it correct — #845. There IS a
diagnostic, contrary to an earlier draft of this paragraph: the job emits a `::warning::` naming the
Correct today — verified live 2026-08-26, reproduce with `GET /api/v1/user` (login `timothy`) and
`GET /api/v1/admin/users?limit=50` (exactly `renovate`, `timothy`) — but nothing keeps it correct
(#845). There IS a diagnostic: the job emits a `::warning::` naming the
off-list creator whenever it re-derives over a verdict-shaped status. What #845 is about is that the
warning fires in the WORKFLOW RUN, while the mistake is made at the terminal running
`post-review-verdict.sh`, which reports success either way.
+146 -7
View File
@@ -1153,20 +1153,39 @@ if "/status" in url:
pre = os.environ.get("STUB_PRE_ROW", "")
if pre:
pre_creator, pre_state, pre_desc = pre.split("|", 2)
print(json.dumps({"statuses": [
# DECOY CONTEXT + `total_count`, matching the shape the `existing:` modes model and the
# server actually returns. Without a decoy a dropped `select(.context == $c)` is
# invisible; without `total_count` the emptiness cross-check is never exercised.
rows = [
{"context": "ci/decoy", "status": "pending", "creator": None,
"description": "unrelated"},
{"context": "review-verdict/h10", "status": pre_state,
"creator": ({"login": pre_creator} if pre_creator else None),
"description": pre_desc}]}))
"description": pre_desc},
]
print(json.dumps({"state": "pending", "total_count": len(rows), "statuses": rows}))
sys.exit(0)
print(empty_statuses())
sys.exit(0)
# Configurable creator (ersatztv#742). This site asks "did anything human land while we were
# classifying", NOT "may this be inherited", so it must stay broad: an off-allow-list account
# posting BLOCKED here still has to make the job abstain.
print(json.dumps({"statuses": [
{"context": "review-verdict/h10", "status": "failure",
"creator": {"login": os.environ.get("STUB_MIDRUN_CREATOR", "timothy")},
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}]}))
# STUB_MIDRUN_ROW overrides the whole row, so a test can make the mid-run row differ from
# STUB_PRE_ROW in exactly ONE field of the state/creator/description triple. Without that,
# every mid-run test moves all three at once and no single clause of the comparison is pinned.
mid = os.environ.get("STUB_MIDRUN_ROW", "")
if mid:
mid_creator, mid_state, mid_desc = mid.split("|", 2)
else:
mid_creator = os.environ.get("STUB_MIDRUN_CREATOR", "timothy")
mid_state, mid_desc = "failure", "Review-verdict: BLOCKED @ a9e3e23 (base: main)"
mid_rows = [
{"context": "ci/decoy", "status": "pending", "creator": None, "description": "unrelated"},
{"context": "review-verdict/h10", "status": mid_state,
"creator": ({"login": mid_creator} if mid_creator else None),
"description": mid_desc},
]
print(json.dumps({"state": "pending", "total_count": len(mid_rows), "statuses": mid_rows}))
sys.exit(0)
if mode.startswith("sentinel-appears-on-read:"):
# A repair sentinel written by ANOTHER, overlapping run between this job's first read and its
@@ -1252,6 +1271,8 @@ def _run_classify(
history_creator: str = "timothy",
midrun_creator: str = "timothy",
pre_row: str = "",
midrun_row: str = "",
reviewers: str | None = None,
timeline_terminator: str = "null",
status_empty_shape: str = "null",
):
@@ -1292,6 +1313,7 @@ def _run_classify(
env["STUB_HISTORY_CREATOR"] = history_creator
env["STUB_MIDRUN_CREATOR"] = midrun_creator
env["STUB_PRE_ROW"] = pre_row
env["STUB_MIDRUN_ROW"] = midrun_row
env.update(
{
"GITEA_TOKEN": "stub",
@@ -1311,7 +1333,16 @@ def _run_classify(
)
script = tmp_path / "step.sh"
script.write_text(_classify_step()["run"])
body = _classify_step()["run"]
if reviewers is not None:
# Substitute the allow-list into the REAL step body so the shipped membership loop runs
# against a multi-entry value. Every other test runs the singleton `timothy`, under which a
# loop that can only ever match its LAST entry is indistinguishable from a correct one — and
# the workflow's own diagnostic tells operators to add a second entry.
body, n = re.subn(r'^(\s*)H10_REVIEWERS="[^"]*"$', lambda m: f'{m.group(1)}H10_REVIEWERS="{reviewers}"',
body, flags=re.MULTILINE)
assert n == 1, f"expected exactly one H10_REVIEWERS assignment to override, substituted {n}"
script.write_text(body)
r = subprocess.run(["bash", str(script)], cwd=tmp_path, env=env, capture_output=True, text=True)
# Assert the WIRING, not only the classification. The stub accepts every POST, so a status aimed
# at the wrong endpoint, sha, host or repo would otherwise leave these tests green while the real
@@ -1725,6 +1756,15 @@ def test_a_verdict_shaped_SUCCESS_from_a_NON_allowlisted_account_is_RE_DERIVED(t
f"read as 'a reviewer wrote this'. Log:\n{r.stdout[-900:]}"
)
assert posted["state"] == "pending"
# THE ANNOTATION IS PART OF THE CONTRACT, not decoration. `docs/ci-cd.md` heads a paragraph with
# "If a reviewer's verdicts stop being honoured, this is why", and `ci.exemption-provenance` uses
# its existence to soften the #845 residual — so a tidy-up that deletes it must redden something.
# Cold review measured that deleting it left the whole suite green.
assert "::warning::" in r.stdout and "NOT in H10_REVIEWERS" in r.stdout, (
"no ::warning:: was emitted while re-deriving over a verdict-shaped status from an off-list "
f"account — the only signal a real reviewer's reviews are being discarded. Log:\n{r.stdout[-1200:]}"
)
assert creator in r.stdout, "the annotation does not name the account whose verdict was discarded"
@pytest.mark.parametrize("creator", ["renovate", "someone-else"])
@@ -1896,6 +1936,98 @@ def test_every_test_name_cited_by_the_exemption_provenance_record_EXISTS():
)
_PRE = "renovate|success|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"
@pytest.mark.parametrize(
"clause,midrun_row",
[
("description", "renovate|success|Review-verdict: BLOCKED @ a9e3e23 (base: main)"),
("creator", "timothy|success|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"),
("state", "renovate|failure|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"),
],
)
def test_EACH_clause_of_the_changed_row_comparison_is_load_bearing(tmp_path, clause, midrun_row):
"""One case per field, because a triple whose cases all move all three fields pins none of them.
The mid-run row here differs from the pre-existing one in EXACTLY the named field, so dropping
that field's comparison makes the job miss the change, skip the abstain, and post its docs-only
exemption over the row. Reviewed cold: with only the two "everything moves at once" cases, three
single-clause deletions and a reduction of the whole triple to one comparison all shipped green.
The pre-existing row is an OFF-LIST `success` so the first read does not short-circuit — which is
also the realistic setup, since an off-list row is exactly why the job would be re-deriving.
"""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="appears-on-read:2",
pre_row=_PRE,
midrun_row=midrun_row,
)
assert r.returncode == 0, r.stderr
assert posted is None, (
f"the mid-run row differed from the pre-existing one only in its {clause}, and the job did "
f"not notice — that clause of the changed-row comparison is not being evaluated. "
f"Posted: {posted}. Log:\n{r.stdout[-1200:]}"
)
def test_an_allow_list_with_SEVERAL_entries_honours_the_FIRST_one(tmp_path):
"""Multi-entry semantics, which every other test leaves unexercised.
All the behavioural cases run the singleton `H10_REVIEWERS="timothy"`, under which a loop that
can only ever match its LAST entry — drop the `break`, add an `else ex_human=no` — behaves
identically to a correct one. That matters more than it looks: the `::warning::` this change adds
tells an operator whose reviews are being discarded to ADD THEMSELVES to the list, so a
two-entry list is the first thing the fix produces, and the first entry is the one that breaks.
Docs-only on purpose: if the verdict is inherited the job posts nothing, and if it is not, the
exemption `success` is posted. So `posted is None` IS the inheritance.
"""
posted, r = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:success",
status_creator="alice",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
reviewers="alice timothy",
)
assert r.returncode == 0, r.stderr
assert posted is None, (
"a verdict from the FIRST entry of a two-entry H10_REVIEWERS was not inherited — membership "
f"is not scanning the whole list. Posted: {posted}. Log:\n{r.stdout[-900:]}"
)
def test_the_multi_entry_control_the_LAST_entry_is_honoured_too(tmp_path):
"""Without this, the test above could pass for any reason unrelated to list position."""
posted, _ = _run_classify(
tmp_path,
_emitting("docs/a.md"),
status_mode="existing:success",
status_creator="timothy",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
reviewers="alice timothy",
)
assert posted is None, "the positive control failed, so the first-entry test proves nothing"
def test_an_account_on_NEITHER_entry_is_still_re_derived(tmp_path):
"""The negative control for the two above: widening the list must not widen it to everyone."""
posted, _ = _run_classify(
tmp_path,
_emitting("ErsatzTV/Program.cs"),
status_mode="existing:success",
status_creator="mallory",
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
reviewers="alice timothy",
)
assert posted is not None and posted["state"] == "pending", (
"an account absent from a two-entry allow-list was inherited"
)
def _h10_reviewers_literal():
"""The allow-list as the workflow actually declares it — read from the shipped file, never
restated here, so a test cannot pass against a value the gate does not use."""
@@ -1985,6 +2117,13 @@ def test_a_REAL_human_verdict_is_still_NEVER_overwritten(tmp_path, existing):
f"overwrote a human '{existing}' verdict written by timothy — the never-overwrite property "
"has been lost while fixing route 3"
)
# The MIRROR of the annotation assertion in the off-list test: it must NOT fire for an
# allow-listed reviewer. This is what pins the `ex_human != yes` conjunct — without it the
# annotation reads "from 'timothy', which is NOT in H10_REVIEWERS ('timothy')", a diagnostic that
# contradicts itself and trains the reader to ignore it.
assert "NOT in H10_REVIEWERS" not in r.stdout, (
f"the off-list annotation fired for an ALLOW-LISTED creator. Log:\n{r.stdout[-1200:]}"
)
def test_a_PENDING_status_from_a_previous_run_is_replaced_normally(tmp_path):