Release-tag builds: small-lane PR-only gate jobs run + fail on v* tag pushes #535

Closed
opened 2026-07-21 17:36:02 +02:00 by timothy · 2 comments
Owner

Observed (v26.12.0 tag, run 952 / b52c9388)

Three gate jobs reported failure on the release-tag build, turning the tag's overall commit status red even though the release built fine:

  • CI image pin matches docker/ci (job id 5289)
  • Docs update reminder (job id 5290)
  • decisions lifecycle (job id 5291)

All three: started ~14:51–14:52, completed at the identical timestamp 15:05:55, and their job logs do not exist (dbfs open … file does not exist) — the killed-job signature (runner restart mid-run), not a code failure.

Root pattern

All five PR-only jobs carry the same if: github.event_name == 'pull_request'. On the tag push:

  • Skipped correctly (instant): api-docs, format — both runs-on: ubuntu-latest
  • Ran, then killed: ci-image-pin, docs-reminder, decisions-guard — all runs-on: small

The only difference is the runner label. The small lane did not honour the if: skip on a tag ref (the jobs dispatched to a runner and ran ~13 min instead of skipping). #508 recently reworked the small lane, which fits the timing — v26.11.0's tag (601d855b) showed none of these jobs at all.

Even absent the kill, docs-reminder/decisions-guard would fail on a tag anyway: they diff against origin/${{ github.base_ref }}, and base_ref is empty on a tag push.

Impact

  • Non-blocking for releases: build (the image push) needs: [test, migrations] only, so :prod + :<version> still ship. Confirmed: v26.12.0 built, scanned clean, and deployed to prod.
  • Cosmetic/operational: every release tag now shows a red overall status + three spurious failed jobs, which will confuse each release session (it cost this one a forensic detour).

Fix directions

  1. Exclude tag refs from these jobs explicitly, e.g. if: github.event_name == 'pull_request' → keep, but investigate why the small lane ignores it (likely the small-runner config after #508). Root-cause the small-lane if: handling first.
  2. Or gate the small-lane gate jobs with an additional && !startsWith(github.ref, 'refs/tags/').
  3. Make decisions-guard/docs-reminder no-op safely when base_ref is empty.

Start at (1): the discriminator is the runner lane, not the job logic.

Done-when

  • .gitea/workflows/pr-checks.yml created (on: pull_request); ci-image-pin, docs-reminder, decisions-guard removed from docker-build.yml (byte-identical relocation)
  • Those 3 gates run and pass under the new PR Gates / status context on this PR (live proof the split works)
  • docs/ci-cd.md updated in the same PR (new "PR gates workflow" section + cross-refs)
  • CI green on the PR head sha
  • Adversarial review passed

Post-merge verification (not a merge gate — unsatisfiable pre-merge): the next v* release tag shows no spurious ci-image-pin/docs-reminder/decisions lifecycle failures on its overall commit status.

## Observed (v26.12.0 tag, run 952 / b52c9388) Three gate jobs reported **failure** on the release-tag build, turning the tag's overall commit status red even though the release built fine: - `CI image pin matches docker/ci` (job id 5289) - `Docs update reminder` (job id 5290) - `decisions lifecycle` (job id 5291) All three: started ~14:51–14:52, **completed at the identical timestamp 15:05:55**, and their **job logs do not exist** (`dbfs open … file does not exist`) — the killed-job signature (runner restart mid-run), not a code failure. ## Root pattern All five PR-only jobs carry the same `if: github.event_name == 'pull_request'`. On the tag push: - **Skipped correctly (instant):** `api-docs`, `format` — both `runs-on: ubuntu-latest` - **Ran, then killed:** `ci-image-pin`, `docs-reminder`, `decisions-guard` — all **`runs-on: small`** The only difference is the runner label. The `small` lane did not honour the `if:` skip on a tag ref (the jobs dispatched to a runner and ran ~13 min instead of skipping). #508 recently reworked the small lane, which fits the timing — v26.11.0's tag (601d855b) showed none of these jobs at all. Even absent the kill, `docs-reminder`/`decisions-guard` would fail on a tag anyway: they diff against `origin/${{ github.base_ref }}`, and `base_ref` is empty on a tag push. ## Impact - **Non-blocking for releases**: `build` (the image push) `needs: [test, migrations]` only, so `:prod` + `:<version>` still ship. Confirmed: v26.12.0 built, scanned clean, and deployed to prod. - **Cosmetic/operational**: every release tag now shows a red overall status + three spurious failed jobs, which will confuse each release session (it cost this one a forensic detour). ## Fix directions 1. Exclude tag refs from these jobs explicitly, e.g. `if: github.event_name == 'pull_request'` → keep, but investigate why the `small` lane ignores it (likely the small-runner config after #508). Root-cause the small-lane `if:` handling first. 2. Or gate the `small`-lane gate jobs with an additional `&& !startsWith(github.ref, 'refs/tags/')`. 3. Make `decisions-guard`/`docs-reminder` no-op safely when `base_ref` is empty. Start at (1): the discriminator is the runner lane, not the job logic. ## Done-when - [x] `.gitea/workflows/pr-checks.yml` created (`on: pull_request`); `ci-image-pin`, `docs-reminder`, `decisions-guard` removed from `docker-build.yml` (byte-identical relocation) - [x] Those 3 gates run **and pass** under the new `PR Gates /` status context on this PR (live proof the split works) - [x] `docs/ci-cd.md` updated in the same PR (new "PR gates workflow" section + cross-refs) - [x] CI green on the PR head sha - [x] Adversarial review passed Post-merge verification (not a merge gate — unsatisfiable pre-merge): the next `v*` release tag shows no spurious `ci-image-pin`/`docs-reminder`/`decisions lifecycle` failures on its overall commit status.
timothy added the priority: medium label 2026-07-21 17:36:17 +02:00
timothy added the in-progress label 2026-07-21 17:56:35 +02:00
Author
Owner

Claiming. Run-952 forensics confirm the mechanism: all 5 PR-only jobs get dispatched on the tag push; api-docs/format (ubuntu-latest, identical if:) resolved to skipped cleanly, while the 3 small-lane git-diff gates (ci-image-pin, docs-reminder, decisions-guard) wedged in act setup and were killed by a runner restart → spurious failure. So the job logic is fine; the kill happens in the dispatch window before any if:/step runs, and #508 (already in the tag) did not prevent it.

Fix: relocate exactly those 3 pure git-diff gates into a dedicated on: pull_request workflow. Gitea evaluates workflow triggers before creating any jobs, so on a tag/main push they are never created → zero dispatch → zero spurious red, permanently. They carry no container:/image pin, so ci-image-pin’s grep of docker-build.yml (which keeps all pin-bearing jobs) stays correct. None are required checks, so branch protection is unaffected.

Claiming. Run-952 forensics confirm the mechanism: all 5 PR-only jobs get *dispatched* on the tag push; `api-docs`/`format` (ubuntu-latest, identical `if:`) resolved to `skipped` cleanly, while the 3 `small`-lane git-diff gates (`ci-image-pin`, `docs-reminder`, `decisions-guard`) wedged in act setup and were killed by a runner restart → spurious `failure`. So the job *logic* is fine; the kill happens in the dispatch window before any `if:`/step runs, and #508 (already in the tag) did not prevent it. Fix: relocate exactly those 3 pure git-diff gates into a dedicated `on: pull_request` workflow. Gitea evaluates workflow triggers *before* creating any jobs, so on a tag/main push they are never created → zero dispatch → zero spurious red, permanently. They carry no `container:`/image pin, so `ci-image-pin`’s grep of `docker-build.yml` (which keeps all pin-bearing jobs) stays correct. None are required checks, so branch protection is unaffected.
timothy removed the in-progress label 2026-07-21 18:32:50 +02:00
Author
Owner

Done — fixed in #537 (merged to main as c21d2271).

Root cause. The three gates are pure checkout + git diff jobs on the small lane with if: github.event_name == 'pull_request'. They lived in docker-build.yml, which also triggers on push to main and v* tags. Gitea dispatches a job as a runner task even when its if skips it (documented small-lane behavior in docs/ci-cd.md), evaluating the skip only after dispatch. On the v26.12.0 tag those dispatched skip-tasks wedged in act's setup phase and were killed by a runner restart mid-setup → failure with no logs → red overall commit status on a release that built, scanned, and deployed fine. The two PR-only jobs on ubuntu-latest (api-docs, format) carry the identical if: and skipped cleanly — the job logic was never the problem, so the issue's options 2/3 (tweak if:/step logic) could not have fixed it; the kill lands in the dispatch window before any step or skip runs. #508 (already in the tag) widened the lane but did not close that window.

Fix. Relocated exactly those three gates verbatim into a new .gitea/workflows/pr-checks.yml that triggers on: pull_request only. Gitea evaluates a workflow's trigger before creating any job, so on a tag/main push the workflow produces zero jobs — no dispatch, no kill, no spurious red, for the whole class permanently.

Live proof. PR #537's own CI ran all three under the new context and passed: PR Gates / CI image pin matches docker/ci, PR Gates / Docs update reminder, PR Gates / decisions lifecycle — all green, alongside the six docker-build.yml jobs.

Files changed.

  • .gitea/workflows/pr-checks.yml (new) — the three relocated gates + defaults: run: shell: bash (ci-image-pin uses mapfile).
  • .gitea/workflows/docker-build.yml — removed the three jobs; added a pointer comment.
  • docs/ci-cd.md — new "PR gates workflow" section + cross-references (same PR).

Why it's safe. ci-image-pin carries no image pin and still greps docker-build.yml, where all five pin-bearing jobs remain → validation unchanged. None of the three are required checks (only Build & test (.NET) + EF migration integrity are), and the merge-consent hook reads the prefix-agnostic combined status, so the context-prefix change (Build ErsatzTV Image / … → PR Gates / …) doesn't affect merges.

Deferred / not done. api-docs/format were intentionally left in docker-build.yml: they carry the shared container: image pin (ci-image-pin's grep target) and run on the healthy ubuntu-latest lane where they skipped correctly — moving them would split the pin across two files for no observed benefit. Residual (low-probability) risk that they could be killed on a future tag if the ubuntu-latest runner restarts during their skip window; not worth the entanglement now.

Review. Cold adversarial review over the diff: MERGEABLE, zero Blocker/High/Medium/Low findings.

Post-merge verification (tracked in Done-when, not a merge gate): the next v* release tag should show no spurious ci-image-pin/docs-reminder/decisions lifecycle failures on its overall commit status.

**Done** — fixed in #537 (merged to `main` as c21d2271). **Root cause.** The three gates are pure `checkout + git diff` jobs on the `small` lane with `if: github.event_name == 'pull_request'`. They lived in `docker-build.yml`, which *also* triggers on push to `main` and `v*` tags. Gitea **dispatches a job as a runner task even when its `if` skips it** (documented small-lane behavior in `docs/ci-cd.md`), evaluating the skip only after dispatch. On the v26.12.0 tag those dispatched skip-tasks wedged in act's setup phase and were killed by a runner restart mid-setup → `failure` with no logs → red overall commit status on a release that built, scanned, and deployed fine. The two PR-only jobs on `ubuntu-latest` (`api-docs`, `format`) carry the **identical** `if:` and skipped cleanly — the job logic was never the problem, so the issue's options 2/3 (tweak `if:`/step logic) could not have fixed it; the kill lands in the dispatch window before any step or skip runs. #508 (already in the tag) widened the lane but did not close that window. **Fix.** Relocated exactly those three gates **verbatim** into a new `.gitea/workflows/pr-checks.yml` that triggers `on: pull_request` only. Gitea evaluates a workflow's trigger *before* creating any job, so on a tag/main push the workflow produces **zero** jobs — no dispatch, no kill, no spurious red, for the whole class permanently. **Live proof.** PR #537's own CI ran all three under the new context and passed: `PR Gates / CI image pin matches docker/ci`, `PR Gates / Docs update reminder`, `PR Gates / decisions lifecycle` — all green, alongside the six `docker-build.yml` jobs. **Files changed.** - `.gitea/workflows/pr-checks.yml` (new) — the three relocated gates + `defaults: run: shell: bash` (ci-image-pin uses `mapfile`). - `.gitea/workflows/docker-build.yml` — removed the three jobs; added a pointer comment. - `docs/ci-cd.md` — new "PR gates workflow" section + cross-references (same PR). **Why it's safe.** `ci-image-pin` carries no image pin and still greps `docker-build.yml`, where all five pin-bearing jobs remain → validation unchanged. None of the three are required checks (only `Build & test (.NET)` + `EF migration integrity` are), and the merge-consent hook reads the prefix-agnostic combined status, so the context-prefix change (`Build ErsatzTV Image / …` → `PR Gates / …`) doesn't affect merges. **Deferred / not done.** `api-docs`/`format` were intentionally **left** in `docker-build.yml`: they carry the shared `container:` image pin (`ci-image-pin`'s grep target) and run on the healthy `ubuntu-latest` lane where they skipped correctly — moving them would split the pin across two files for no observed benefit. Residual (low-probability) risk that they could be killed on a future tag if the ubuntu-latest runner restarts during their skip window; not worth the entanglement now. **Review.** Cold adversarial review over the diff: **MERGEABLE**, zero Blocker/High/Medium/Low findings. **Post-merge verification (tracked in Done-when, not a merge gate):** the next `v*` release tag should show no spurious `ci-image-pin`/`docs-reminder`/`decisions lifecycle` failures on its overall commit status.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#535