review-verdict/h10 writes are not serialized — a stale run can overwrite a fresher verdict (ABA retarget + human-rejection race) #706

Closed
opened 2026-07-29 20:36:01 +02:00 by timothy · 2 comments
Owner

Found by cross-family adversarial review of PR #705 (#698). Not introduced by that PR — it is the residual that PR narrows but cannot close, and #705's decision record now says so explicitly rather than claiming otherwise.

The common root cause: the gate's classification is read-then-write with no serialization and no compare-and-set. A commit status POST is last-writer-wins, and nothing orders concurrent workflow runs.

Race 1 — the retarget ABA

  1. Run A reads base main.
  2. PR is retargeted to a scratch base; A's /pulls/{n}/files sees a docs-only diff.
  3. PR is retargeted back to main; A's post-enumeration re-read sees main again, so both of A's base checks pass.
  4. The retarget-back queues Run B, but A can post success before B posts pending — an already-scheduled auto-merge can fire in that green window.
  5. Runs are not serialized, so A can equally finish after B and restore success permanently, with no further event left to correct it.

#705 added edited to types: and re-derivation of machine-written successes. That gives detection, not atomicity — it does not order the writes. My first draft of the record claimed this made the residual "non-durable"; that was wrong and is retracted in the record.

Race 2 — a human rejection can be overwritten

  1. The workflow reads the existing statuses and finds no verdict.
  2. A reviewer posts a human failure (BLOCKED) via scripts/post-review-verdict.sh.
  3. The workflow finishes classifying a docs-only PR and posts success.

The latest review-verdict/h10 is now green despite an explicit human rejection. #705's provenance check does not help: it runs before the human status exists. The existing tests preload a status before the step executes, so none of them exercise this window.

Why this is not simply "fix the ordering"

Gitea's status API has no conditional write (no ETag / If-Match / expected-previous-state), so a plain re-read-before-POST only narrows the window again. Candidate directions, none yet evaluated:

  • Re-read the combined status immediately before the POST and refuse to overwrite a human verdict discovered in the interim — narrows race 2 substantially, does not close it.
  • Have the workflow never write success at all, only pending/failure, and move exemption-granting to a separately-issued credential — changes the trust model rather than the timing.
  • Serialize with a Gitea Actions concurrency group per PR. Note the existing header comment in review-verdict.yml explains why this workflow deliberately has no cancelling concurrency group (a cancelled run leaves an exempt PR statusless forever); a non-cancelling group may be viable.

Done-when

  • Both races reproduced on scratch PRs (probe-named contexts only, never a real forged h10)
  • A direction chosen with its cost stated, including whether it interacts with the no-cancel constraint
  • Race 2 in particular addressed or explicitly accepted — silently greening a human BLOCKED is the worst outcome in this class
  • ci.exemption-provenance residual updated to the resolved state
  • Adversarial review passed

Related

#698 / PR #705 (the change that surfaced and documented this), #697 (credentials that can write the status), #663 (status is repo-global per sha), #622 (why the status is per-sha at all).

Found by cross-family adversarial review of PR #705 (#698). **Not introduced by that PR** — it is the residual that PR narrows but cannot close, and #705's decision record now says so explicitly rather than claiming otherwise. The common root cause: **the gate's classification is read-then-write with no serialization and no compare-and-set.** A commit status POST is last-writer-wins, and nothing orders concurrent workflow runs. ## Race 1 — the retarget ABA 1. Run A reads base `main`. 2. PR is retargeted to a scratch base; A's `/pulls/{n}/files` sees a docs-only diff. 3. PR is retargeted **back** to `main`; A's post-enumeration re-read sees `main` again, so **both** of A's base checks pass. 4. The retarget-back queues Run B, but A can post `success` before B posts `pending` — an already-scheduled auto-merge can fire in that green window. 5. Runs are not serialized, so A can equally finish *after* B and restore `success` **permanently**, with no further event left to correct it. #705 added `edited` to `types:` and re-derivation of machine-written successes. That gives **detection, not atomicity** — it does not order the writes. My first draft of the record claimed this made the residual "non-durable"; that was wrong and is retracted in the record. ## Race 2 — a human rejection can be overwritten 1. The workflow reads the existing statuses and finds no verdict. 2. A reviewer posts a human `failure` (BLOCKED) via `scripts/post-review-verdict.sh`. 3. The workflow finishes classifying a docs-only PR and posts `success`. The latest `review-verdict/h10` is now green despite an explicit human rejection. #705's provenance check does not help: it runs *before* the human status exists. The existing tests preload a status before the step executes, so none of them exercise this window. ## Why this is not simply "fix the ordering" Gitea's status API has no conditional write (no ETag / If-Match / expected-previous-state), so a plain re-read-before-POST only narrows the window again. Candidate directions, none yet evaluated: - Re-read the combined status **immediately before** the POST and refuse to overwrite a human verdict discovered in the interim — narrows race 2 substantially, does not close it. - Have the workflow never write `success` at all, only `pending`/`failure`, and move exemption-granting to a separately-issued credential — changes the trust model rather than the timing. - Serialize with a Gitea Actions `concurrency` group per PR. Note the existing header comment in `review-verdict.yml` explains why this workflow deliberately has **no cancelling** concurrency group (a cancelled run leaves an exempt PR statusless forever); a non-cancelling group may be viable. ## Done-when - [x] Both races reproduced on scratch PRs (probe-named contexts only, never a real forged `h10`) - [x] A direction chosen with its cost stated, including whether it interacts with the no-cancel constraint - [x] Race 2 in particular addressed or explicitly accepted — silently greening a human BLOCKED is the worst outcome in this class - [x] `ci.exemption-provenance` residual updated to the resolved state - [x] Adversarial review passed ## Related #698 / PR #705 (the change that surfaced and documented this), #697 (credentials that can write the status), #663 (status is repo-global per sha), #622 (why the status is per-sha at all).
timothy added the ci-cdpriority: highsecurity labels 2026-07-29 20:37:28 +02:00
timothy added the in-progress label 2026-08-03 20:17:39 +02:00
Author
Owner

Claiming this (Claude Code session, 2026-08-03). Four-way claim check clear: no open PR references #706, no remote branch names it, no prior comments, fresh git fetch origin main at 9dc360c9f.

Plan: reproduce both races on scratch PRs with probe-named contexts (never a real review-verdict/h10), then choose a direction. Will surface bundle-able ci-cd/security siblings (#697, #707, #711) separately rather than folding them in silently.

Claiming this (Claude Code session, 2026-08-03). Four-way claim check clear: no open PR references #706, no remote branch names it, no prior comments, fresh `git fetch origin main` at 9dc360c9f. Plan: reproduce both races on scratch PRs with probe-named contexts (never a real `review-verdict/h10`), then choose a direction. Will surface bundle-able ci-cd/security siblings (#697, #707, #711) separately rather than folding them in silently.
Author
Owner

Closing record

Outcome: Fixed and merged in PR #723 (merge aa79ec59c). Race 1 is fenced; race 2 is narrowed, repaired after the fact, and the repair is durable.

Root cause: The gate's classification was read-then-write with no serialization and no compare-and-set. Confirmed live rather than reasoned: pull_request_target runs for one PR genuinely overlap and the OLDER can finish LAST (probe PR #722 — run 7520 completed 20s after run 7521 started).

Decisions/conventions changed: Added ci.verdict-write-retarget-fence. Narrowed ci.exemption-provenance (its #706 residual is now resolved, not merely tracked) and release.verdict-status-check / ci.shared-pr-file-enumeration for the sibling fixes.

Reusable knowledge — three things worth carrying:

  1. This issue's own proposed fix was wrong, and measurement is the only reason we know. A non-cancelling concurrency group does NOT serialize pull_request_target runs on Gitea 1.25.4 — with the group active, runs 7528/7529 still overlapped and 7528 ended 36s after 7529 began.
  2. A negative control changed the answer. A first probe with a group showed cancellations and looked like confirmation. The identical workflow with no concurrency: key at all cancelled the same way — Gitea auto-cancels superseded push runs by itself, and that does not extend to pull_request_target. Without the control, a no-op would have shipped as a solution.
  3. The ABA needs a monotonic key. main → S → main reads main at both ends, so every name-based check passes. The timeline's change_target_branch COUNT cannot alias (verified on PR #703: exactly 2 events; PR #717: 0). Abstaining is safe only because every retarget fires edited, which guarantees a successor run — updated_at was rejected as the key precisely because it moves for comments/labels, which queue nothing.

Verification: 411 script tests green (local + CI Script tests 90s, a real run). Every new guard mutation-checked individually — each neutered in turn, each producing the intended red while its positive control stayed green. Required CI contexts green on fe3d292: Build & test (.NET) (535s, full run), EF migration integrity, review-verdict/h10. Both races reproduced live on scratch PRs with probe-only contexts — race 1 as above; race 2 on probe PR #724, where the per-POST history shows a human failure (id 2, creator timothy) then a machine success (id 3, creator null), leaving the combined status branch protection reads as SUCCESS over an explicit human BLOCKED. All probe PRs closed, all probe branches deleted, no real review-verdict/h10 ever written by a probe.

Deferred: Four residuals recorded in the decision record, none failing toward silent success — the sub-round-trip window (no compare-and-set exists on Gitea's status API), the repair being itself a read-then-write, the absent vocabulary tripwire if change_target_branch is ever renamed, and permanent exemption denial for a timeline over the 20-page cap. A human re-posting a BASE-MISMATCHED verdict after a repair buries the sentinel — that needs a user credential, so it is #697's.

Docs updated: docs/decisions/records/ci/verdict-write-retarget-fence.md (new), ci/exemption-provenance.md, ci/shared-pr-file-enumeration.md, release/verdict-status-check.md, docs/decisions/README.md (regenerated), docs/ci-cd.md → Review-verdict gate, CLAUDE.md.

Review: Independent cold adversarial review (worktree-isolated, review-only brief), six rounds, five real defects, all fixed: mark ordered after the re-read (blind multi-round-trip window); repair undone by the next event; sentinel self-clobbering via the generic description; a stale overlapping run overwriting a mid-run sentinel (ex_repair was dead state after the re-read — the one path failing toward SUCCESS); the same clobber via the pending path; and the post-write filter blind to sentinel rows. Final verdict MERGEABLE @ fe3d292.

## Closing record **Outcome:** Fixed and merged in PR #723 (merge aa79ec59c). Race 1 is fenced; race 2 is narrowed, repaired after the fact, and the repair is durable. **Root cause:** The gate's classification was read-then-write with no serialization and no compare-and-set. Confirmed live rather than reasoned: `pull_request_target` runs for one PR genuinely overlap and the OLDER can finish LAST (probe PR #722 — run 7520 completed 20s after run 7521 started). **Decisions/conventions changed:** Added `ci.verdict-write-retarget-fence`. Narrowed `ci.exemption-provenance` (its #706 residual is now resolved, not merely tracked) and `release.verdict-status-check` / `ci.shared-pr-file-enumeration` for the sibling fixes. **Reusable knowledge — three things worth carrying:** 1. **This issue's own proposed fix was wrong, and measurement is the only reason we know.** A non-cancelling concurrency group does NOT serialize `pull_request_target` runs on Gitea 1.25.4 — with the group active, runs 7528/7529 still overlapped and 7528 ended 36s after 7529 began. 2. **A negative control changed the answer.** A first probe *with* a group showed cancellations and looked like confirmation. The identical workflow with **no `concurrency:` key at all** cancelled the same way — Gitea auto-cancels superseded **`push`** runs by itself, and that does not extend to `pull_request_target`. Without the control, a no-op would have shipped as a solution. 3. **The ABA needs a monotonic key.** `main → S → main` reads `main` at both ends, so every name-based check passes. The timeline's `change_target_branch` COUNT cannot alias (verified on PR #703: exactly 2 events; PR #717: 0). Abstaining is safe *only* because every retarget fires `edited`, which guarantees a successor run — `updated_at` was rejected as the key precisely because it moves for comments/labels, which queue nothing. **Verification:** 411 script tests green (local + CI `Script tests` 90s, a real run). Every new guard mutation-checked individually — each neutered in turn, each producing the intended red while its positive control stayed green. Required CI contexts green on fe3d292: `Build & test (.NET)` (535s, full run), `EF migration integrity`, `review-verdict/h10`. **Both races reproduced live on scratch PRs with probe-only contexts** — race 1 as above; race 2 on probe PR #724, where the per-POST history shows a human `failure` (id 2, creator `timothy`) then a machine `success` (id 3, creator `null`), leaving the combined status branch protection reads as SUCCESS over an explicit human BLOCKED. All probe PRs closed, all probe branches deleted, no real `review-verdict/h10` ever written by a probe. **Deferred:** Four residuals recorded in the decision record, none failing toward silent success — the sub-round-trip window (no compare-and-set exists on Gitea's status API), the repair being itself a read-then-write, the absent vocabulary tripwire if `change_target_branch` is ever renamed, and permanent exemption denial for a timeline over the 20-page cap. A human re-posting a BASE-MISMATCHED verdict after a repair buries the sentinel — that needs a user credential, so it is #697's. **Docs updated:** `docs/decisions/records/ci/verdict-write-retarget-fence.md` (new), `ci/exemption-provenance.md`, `ci/shared-pr-file-enumeration.md`, `release/verdict-status-check.md`, `docs/decisions/README.md` (regenerated), `docs/ci-cd.md` → Review-verdict gate, `CLAUDE.md`. **Review:** Independent cold adversarial review (worktree-isolated, review-only brief), **six rounds, five real defects**, all fixed: mark ordered after the re-read (blind multi-round-trip window); repair undone by the next event; sentinel self-clobbering via the generic description; a stale overlapping run overwriting a mid-run sentinel (`ex_repair` was dead state after the re-read — the one path failing toward SUCCESS); the same clobber via the pending path; and the post-write filter blind to sentinel rows. Final verdict MERGEABLE @ fe3d292.
timothy removed the in-progress label 2026-08-03 23:02:25 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#706