timothy/ersatztv
1
0
Fork 0
Code Issues 57 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

Move the two docker build jobs off the small runner lane #508

New Issue
Closed
opened 2026-07-20 22:23:32 +02:00 by timothy · 2 comments
timothy commented 2026-07-20 22:23:32 +02:00
Owner
Copy Link
Copy Source

Problem

runs-on: small currently carries two jobs that are not small:

  • docker-build.yml → build (Build & push image) — a real buildx on main/tag pushes.
  • ci-image.yml → build (Build & push CI image) — a full buildx of the .NET toolchain image. It reads as lightweight because it is "docker-only, no toolchain needed — it builds the toolchain", but it is the heaviest job in the lane.

A lane's per-job memory cap is set by its worst member, so these two pinned small at --memory=10g. On bumblebee's 25 GiB (also the prod media host) that permits exactly one slot — and four jobs shared it.

The visible symptom is not queue wait. A dispatched job wedges in act's setup phase: >10 min in_progress, no log file written at all (OpenLogs … .log.zst: file does not exist), then failure, before Checkout runs. That is the origin of the "decisions.md append-only is a known flake, just rerun it" folklore — the rerun only works because it lands after load clears. See server-management#639 for the evidence (run 910 / internal 1179).

Change

Move both build jobs to runs-on: ubuntu-latest. small then holds only git-only jobs (decisions-guard, ci-image-pin, docs-reminder), so server-management can cap it at 1 GiB and widen it to 4 slots across two hosts — committing less RAM to CI than the single 10 GiB slot did.

server-management#639 owns the runner-side half (bumblebee retune + a new capped runner on jazz).

Why this doesn't re-create #574

#574 moved build to small because Gitea dispatches a job as a runner task even when its if skips it, and that skip-task queued ~31 min behind long builds. It cannot recur: build has needs: [test, migrations], so it is not dispatched until the jobs it would queue behind have already finished. ci-image.yml's build has no needs but triggers only on docker/ci/** pushes and a weekly cron.

Done-when

  • Both build jobs are runs-on: ubuntu-latest
  • docs/ci-cd.md runner/lane section reflects the four-runner topology and the git-only rule for small
  • docs/decisions.md entry recorded (append-only)
  • CI green on the PR
  • adversarial review passed
## Problem `runs-on: small` currently carries two jobs that are not small: - `docker-build.yml` → `build` (Build & push image) — a real buildx on main/tag pushes. - `ci-image.yml` → `build` (Build & push CI image) — a full buildx of the .NET toolchain image. It reads as lightweight because it is "docker-only, no toolchain needed — it *builds* the toolchain", but it is the heaviest job in the lane. A lane's per-job memory cap is set by its **worst** member, so these two pinned `small` at `--memory=10g`. On bumblebee's 25 GiB (also the prod media host) that permits exactly **one slot** — and four jobs shared it. The visible symptom is not queue wait. A dispatched job wedges in act's **setup** phase: >10 min `in_progress`, **no log file written at all** (`OpenLogs … .log.zst: file does not exist`), then failure, before Checkout runs. That is the origin of the "`decisions.md` append-only is a known flake, just rerun it" folklore — the rerun only works because it lands after load clears. See server-management#639 for the evidence (run 910 / internal 1179). ## Change Move both `build` jobs to `runs-on: ubuntu-latest`. `small` then holds only git-only jobs (`decisions-guard`, `ci-image-pin`, `docs-reminder`), so server-management can cap it at 1 GiB and widen it to 4 slots across two hosts — committing *less* RAM to CI than the single 10 GiB slot did. server-management#639 owns the runner-side half (bumblebee retune + a new capped runner on jazz). ### Why this doesn't re-create #574 #574 moved `build` to `small` because Gitea dispatches a job as a runner task even when its `if` skips it, and that skip-task queued ~31 min behind long builds. It cannot recur: `build` has `needs: [test, migrations]`, so it is not dispatched until the jobs it would queue behind have already finished. `ci-image.yml`'s build has no `needs` but triggers only on `docker/ci/**` pushes and a weekly cron. ## Done-when - [x] Both `build` jobs are `runs-on: ubuntu-latest` - [x] `docs/ci-cd.md` runner/lane section reflects the four-runner topology and the git-only rule for `small` - [x] `docs/decisions.md` entry recorded (append-only) - [x] CI green on the PR - [x] adversarial review passed
timothy commented 2026-07-20 22:41:55 +02:00
Author
Owner
Copy Link
Copy Source

Status — PR #509, one flaky-red away from mergeable

Diff done & reviewed. Both heavy jobs (docker-build.yml's image build + ci-image.yml's toolchain build) are on ubuntu-latest; small is now git-only (ci-image-pin, docs-reminder, decisions-guard); docs updated (ci-cd.md four-runner table + git-only rule; decisions.md entry). Cold adversarial review passed — Review-verdict: MERGEABLE @ 5709bf5a on the PR. 4/5 Done-when boxes ticked.

Why there are two commits. Editing ci-image.yml makes the ci-image-pin guard require the pin to point at the newly-published toolchain image — commit 4263cf7 (stale pin) failed that guard on run 917, and 5709bf5a bumped all pins to 4263cf7 (guard green on run 918). Verified: 1 distinct pin, resolves to the guard's expected, image present in the registry.

The only red is an unrelated CI-capacity flake. Build & test (.NET) on run 918 failed a single SPA test (LibrariesScreen.test.tsx:292, waitFor timeout). It cannot be caused by this diff (CI-config + docs only, 0 lines of app/SPA code). Root cause is a starved CI VM — run summary shows environment 49.30s / import 16.42s, ~200× the local ~0.2s — amplified because this PR's own ci-image.yml edit triggered a concurrent 2 GB toolchain rebuild (run 917) on the same ubuntu-latest lane. Locally: 3/3 for that test, 889/889 full suite. Every other run-918 job passed (EF migrations included).

Action needed (operator). I can't clear this from the agent side: a push would re-trigger the ci-image rebuild and re-amplify the load, and PR-job rerun/cancel is UI-only on Gitea 1.25.4. Please re-run just Build & test (.NET) on run 918 from the Actions UI once the superseded run 917 drains — same pattern as the decisions.md-flake rerun. It should land green (no new ci-image rebuild, lane no longer double-loaded), which ticks the last Done-when box and lets the H6/H10 gate auto-grant the merge.

Follow-up worth filing: LibrariesScreen.test.tsx's scan-complete test still depends on real-microtask timing under a waitFor budget, which is what makes it CI-load-flaky despite prior hardening — a candidate for a deterministic rewrite (drive the poll chain's promises to completion rather than racing them).

### Status — PR #509, one flaky-red away from mergeable **Diff done & reviewed.** Both heavy jobs (`docker-build.yml`'s image `build` + `ci-image.yml`'s toolchain `build`) are on `ubuntu-latest`; `small` is now git-only (`ci-image-pin`, `docs-reminder`, `decisions-guard`); docs updated (`ci-cd.md` four-runner table + git-only rule; `decisions.md` entry). Cold adversarial review passed — `Review-verdict: MERGEABLE @ 5709bf5a` on the PR. 4/5 Done-when boxes ticked. **Why there are two commits.** Editing `ci-image.yml` makes the `ci-image-pin` guard require the pin to point at the newly-published toolchain image — commit `4263cf7` (stale pin) failed that guard on run 917, and `5709bf5a` bumped all pins to `4263cf7` (guard green on run 918). Verified: 1 distinct pin, resolves to the guard's `expected`, image present in the registry. **The only red is an unrelated CI-capacity flake.** `Build & test (.NET)` on run 918 failed a single SPA test (`LibrariesScreen.test.tsx:292`, `waitFor` timeout). It cannot be caused by this diff (CI-config + docs only, 0 lines of app/SPA code). Root cause is a starved CI VM — run summary shows `environment 49.30s` / `import 16.42s`, ~200× the local ~0.2s — amplified because this PR's own `ci-image.yml` edit triggered a concurrent 2 GB toolchain rebuild (run 917) on the same `ubuntu-latest` lane. Locally: **3/3 for that test, 889/889 full suite.** Every other run-918 job passed (EF migrations included). **Action needed (operator).** I can't clear this from the agent side: a push would re-trigger the ci-image rebuild and re-amplify the load, and PR-job rerun/cancel is UI-only on Gitea 1.25.4. Please **re-run just `Build & test (.NET)` on run 918** from the Actions UI once the superseded run 917 drains — same pattern as the `decisions.md`-flake rerun. It should land green (no new ci-image rebuild, lane no longer double-loaded), which ticks the last Done-when box and lets the H6/H10 gate auto-grant the merge. Follow-up worth filing: `LibrariesScreen.test.tsx`'s scan-complete test still depends on real-microtask timing under a `waitFor` budget, which is what makes it CI-load-flaky despite prior hardening — a candidate for a deterministic rewrite (drive the poll chain's promises to completion rather than racing them).
timothy commented 2026-07-20 22:57:04 +02:00
Author
Owner
Copy Link
Copy Source

Closed — merged as 64414be1 (PR #509)

What was done. Both heavy build jobs moved off runs-on: small → ubuntu-latest: docker-build.yml's image push and ci-image.yml's toolchain buildx. small is now git-only (ci-image-pin, docs-reminder, decisions-guard), so server-management#639 can cap it at 1 GiB and widen it to 4 slots across two hosts. #574's skip-task queueing can't recur because build keeps needs: [test, migrations].

Files changed. .gitea/workflows/docker-build.yml, .gitea/workflows/ci-image.yml, docs/ci-cd.md (four-runner table + git-only rule), docs/decisions.md (append-only entry). Two commits: the lane move (4263cf7) + a required CI-image pin bump (5709bf5a).

Why the pin bump was necessary (root cause of the two-commit shape). Editing ci-image.yml re-publishes the toolchain image at that commit's sha and makes the ci-image-pin guard require the pin to match it — so 4263cf7 alone failed ci-image-pin on run 917, and 5709bf5a bumped all six pins to 4263cf7 (guard green on run 918). This is inherent guard behavior, not a defect.

CI. Run 918 went green after a single Build & test (.NET) rerun. The first attempt failed one flaky SPA test (LibrariesScreen.test.tsx:292) purely from CI-VM starvation (environment 49.30s vs ~0.2s local), amplified by the concurrent toolchain rebuild this PR's own ci-image.yml edit triggered on the same lane — unrelated to the diff (0 lines of app/SPA code; 889/889 local). The rerun (no new rebuild, undivided lane) passed.

Deferred / follow-up. Filed a low-priority issue to make that scan-complete SPA test deterministic (it still races real microtasks under a waitFor budget). Note that #508 + server-management#639 attack that flake's root cause (lane starvation), so its frequency should drop regardless.

Docs updated. docs/ci-cd.md + docs/decisions.md in-PR (both required by the change). No operational Obsidian/MEMORY change needed — this is CI-lane topology already captured in ci-cd.md.

### Closed — merged as `64414be1` (PR #509) **What was done.** Both heavy `build` jobs moved off `runs-on: small` → `ubuntu-latest`: `docker-build.yml`'s image push and `ci-image.yml`'s toolchain buildx. `small` is now git-only (`ci-image-pin`, `docs-reminder`, `decisions-guard`), so server-management#639 can cap it at 1 GiB and widen it to 4 slots across two hosts. #574's skip-task queueing can't recur because `build` keeps `needs: [test, migrations]`. **Files changed.** `.gitea/workflows/docker-build.yml`, `.gitea/workflows/ci-image.yml`, `docs/ci-cd.md` (four-runner table + git-only rule), `docs/decisions.md` (append-only entry). Two commits: the lane move (`4263cf7`) + a required CI-image pin bump (`5709bf5a`). **Why the pin bump was necessary (root cause of the two-commit shape).** Editing `ci-image.yml` re-publishes the toolchain image at that commit's sha and makes the `ci-image-pin` guard require the pin to match it — so `4263cf7` alone failed `ci-image-pin` on run 917, and `5709bf5a` bumped all six pins to `4263cf7` (guard green on run 918). This is inherent guard behavior, not a defect. **CI.** Run 918 went green after a single `Build & test (.NET)` rerun. The first attempt failed one flaky SPA test (`LibrariesScreen.test.tsx:292`) purely from CI-VM starvation (`environment 49.30s` vs ~0.2s local), amplified by the concurrent toolchain rebuild this PR's own `ci-image.yml` edit triggered on the same lane — unrelated to the diff (0 lines of app/SPA code; 889/889 local). The rerun (no new rebuild, undivided lane) passed. **Deferred / follow-up.** Filed a low-priority issue to make that scan-complete SPA test deterministic (it still races real microtasks under a `waitFor` budget). Note that #508 + server-management#639 attack that flake's *root cause* (lane starvation), so its frequency should drop regardless. **Docs updated.** `docs/ci-cd.md` + `docs/decisions.md` in-PR (both required by the change). No operational Obsidian/MEMORY change needed — this is CI-lane topology already captured in `ci-cd.md`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#508