pr-changed-files.sh paging can drop a path when the base branch advances mid-enumeration #707

Closed
opened 2026-07-29 20:36:55 +02:00 by timothy · 1 comment
Owner

Found by cross-family adversarial review of PR #705 (#698). Not introduced there; the base-ref binding that PR adds does not address it.

/pulls/{n}/files is paged by offset, and Gitea recomputes the diff against the live base branch for each page independently, then applies that page's offset. So if main advances between page requests, the underlying result set can shrink or shift and a path can move into an offset range that was already consumed.

Concrete scenario:

  1. Page 1 (offset 0, limit 50) returns 50 documentation changes.
  2. A commit merges to main that already contains 20 of those changes, so they drop out of the PR's diff.
  3. A code path previously at position ~55 shifts into positions 31–50.
  4. Page 2 requests offset 50 and starts past it. The code path is never enumerated.

The result is a complete-looking docs-only list that omits a code file. The existing guards do not catch it:

  • the head-sha binding passes — the head never moved;
  • the .base.ref binding added in #705 passes — the base branch never changed, it only advanced;
  • termination is on a validated empty page, which still occurs normally.

Why .base.ref is the right comparator anyway

Worth stating so this isn't "fixed" by breaking something else: post-review-verdict.sh and pr-changed-files.sh deliberately compare the base branch name, never its tip sha (ci.exemption-provenance, #632). Comparing tips would invalidate on every unrelated merge to main — a self-inflicted deadlock.

The fix here is narrower: capture .base.sha before and after the enumeration and fail closed if it moved during it. That rejects only the actual race window, not a base that advanced before the run started. It is a different question from "is this PR still targeting the branch I was told about", which is what .base.ref answers.

Done-when

  • Reproduced deterministically (a stub or a live scratch PR with a merge to main interleaved between pages)
  • Fixed by binding .base.sha across the paging round-trips only, with the .base.ref comparator left intact and a test pinning that both exist for different reasons
  • A test that a base which merely advances before enumeration still succeeds — the deadlock this must not cause
  • ci.shared-pr-file-enumeration updated
  • Adversarial review passed

Related

#698 / PR #705, #649 (the shared enumeration), #632 (base-ref vs base-sha comparator).

Found by cross-family adversarial review of PR #705 (#698). Not introduced there; the base-ref binding that PR adds does not address it. `/pulls/{n}/files` is paged by **offset**, and Gitea recomputes the diff against the **live base branch** for each page independently, then applies that page's offset. So if `main` advances between page requests, the underlying result set can shrink or shift and a path can move into an offset range that was already consumed. Concrete scenario: 1. Page 1 (offset 0, limit 50) returns 50 documentation changes. 2. A commit merges to `main` that already contains 20 of those changes, so they drop out of the PR's diff. 3. A code path previously at position ~55 shifts into positions 31–50. 4. Page 2 requests offset 50 and starts *past* it. The code path is never enumerated. The result is a **complete-looking docs-only list** that omits a code file. The existing guards do not catch it: - the head-sha binding passes — the head never moved; - the `.base.ref` binding added in #705 passes — the base *branch* never changed, it only advanced; - termination is on a validated empty page, which still occurs normally. ## Why `.base.ref` is the right comparator anyway Worth stating so this isn't "fixed" by breaking something else: `post-review-verdict.sh` and `pr-changed-files.sh` deliberately compare the base **branch name**, never its tip sha (`ci.exemption-provenance`, #632). Comparing tips would invalidate on every unrelated merge to `main` — a self-inflicted deadlock. The fix here is narrower: capture `.base.sha` **before and after** the enumeration and fail closed if it moved *during* it. That rejects only the actual race window, not a base that advanced before the run started. It is a different question from "is this PR still targeting the branch I was told about", which is what `.base.ref` answers. ## Done-when - [x] Reproduced deterministically (a stub or a live scratch PR with a merge to `main` interleaved between pages) - [x] Fixed by binding `.base.sha` across the paging round-trips only, with the `.base.ref` comparator left intact and a test pinning that both exist for different reasons - [x] A test that a base which merely advances *before* enumeration still succeeds — the deadlock this must not cause - [x] `ci.shared-pr-file-enumeration` updated - [x] Adversarial review passed ## Related #698 / PR #705, #649 (the shared enumeration), #632 (base-ref vs base-sha comparator).
timothy added the ci-cdpriority: mediumsecurity labels 2026-07-29 20:37:31 +02:00
Author
Owner

Closing record

Outcome: Fixed and merged in PR #723 (merge aa79ec59c), bundled with #706/#711 because all three touch the same two files.

Root cause: scripts/pr-changed-files.sh bound .base.ref (branch NAME) and .head.sha across the enumeration window but never .base.sha. /pulls/{n}/files diffs each offset-paged page against the base's LIVE tip, so an ordinary advance of main mid-paging can drop rows and shift later ones into already-consumed offset ranges — producing a complete-looking docs-only list that omits a code file, with every existing guard silent.

Decisions/conventions changed: ci.shared-pr-file-enumeration updated — the script now owns base-TIP binding as well as base-REF binding, and the record states why they answer different questions.

Reusable knowledge: The ref check and the tip check are NOT interchangeable and both are needed. Comparing tips for the retarget question would self-deadlock (main advances on every unrelated merge); comparing names for the advance question is blind. An advance BEFORE the window is deliberately not an error — pinned by a positive control, because a guard that failed on that would take every open PR down with it.

Verification: 411 script tests green (local + CI Script tests 90s, a real run). Every new guard mutation-checked individually — each neutered in turn, each producing the intended red while its positive control stayed green. Required CI contexts green on fe3d292: Build & test (.NET) (535s, full run), EF migration integrity, review-verdict/h10. Centerpiece test is the exact window (base ref unchanged, tip advanced). The full-suite run also caught a regression a scoped run missed: this guard broke 9 tests in test_merge_consent_exemption.py whose shim served no .base.sha — all positive controls, failing closed. The shim now models the field, and its comment records this as a standing property (it serves what the SHARED enumeration reads, so each new binding must be modelled there too).

Deferred: None. Accepted cost stated in the record: a PR whose enumeration loses the race against a main merge gets no exemption until its next event — a seconds-wide window at this repo's merge frequency.

Docs updated: docs/decisions/records/ci/shared-pr-file-enumeration.md.

## Closing record **Outcome:** Fixed and merged in PR #723 (merge aa79ec59c), bundled with #706/#711 because all three touch the same two files. **Root cause:** `scripts/pr-changed-files.sh` bound `.base.ref` (branch NAME) and `.head.sha` across the enumeration window but never `.base.sha`. `/pulls/{n}/files` diffs each offset-paged page against the base's LIVE tip, so an ordinary advance of `main` mid-paging can drop rows and shift later ones into already-consumed offset ranges — producing a complete-looking docs-only list that omits a code file, with every existing guard silent. **Decisions/conventions changed:** `ci.shared-pr-file-enumeration` updated — the script now owns base-TIP binding as well as base-REF binding, and the record states why they answer different questions. **Reusable knowledge:** The ref check and the tip check are NOT interchangeable and both are needed. Comparing tips for the *retarget* question would self-deadlock (`main` advances on every unrelated merge); comparing names for the *advance* question is blind. An advance BEFORE the window is deliberately not an error — pinned by a positive control, because a guard that failed on that would take every open PR down with it. **Verification:** 411 script tests green (local + CI `Script tests` 90s, a real run). Every new guard mutation-checked individually — each neutered in turn, each producing the intended red while its positive control stayed green. Required CI contexts green on fe3d292: `Build & test (.NET)` (535s, full run), `EF migration integrity`, `review-verdict/h10`. Centerpiece test is the exact window (base ref unchanged, tip advanced). The full-suite run also caught a regression a scoped run missed: this guard broke 9 tests in `test_merge_consent_exemption.py` whose shim served no `.base.sha` — all positive controls, failing closed. The shim now models the field, and its comment records this as a standing property (it serves what the SHARED enumeration reads, so each new binding must be modelled there too). **Deferred:** None. Accepted cost stated in the record: a PR whose enumeration loses the race against a `main` merge gets no exemption until its next event — a seconds-wide window at this repo's merge frequency. **Docs updated:** `docs/decisions/records/ci/shared-pr-file-enumeration.md`.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#707