ci: move api-docs and format back to ubuntu-latest (refs #406) #407

Merged
timothy merged 1 commits from ci/604-lane-rebalance into main 2026-07-17 15:09:11 +02:00
Owner

Part 3 of #406, split out because it is order-coupled with server-management#604's runner re-sizing (which has already landed on the bumblebee side).

What

Two runs-on: small → runs-on: ubuntu-latest, plus updated rationale comments. No other change.

Why

#390 moved these onto the small lane to dodge a ~29 min queue in the contended ubuntu-latest lane. The queue was real — but the lane was the wrong place to fix it, and #390's own comment flagged exactly why:

Caveat: on an API-touching PR this job does a full dotnet build, so it is not always a "small" job; capacity 4 absorbs that.

"Capacity 4 absorbs that" held only because nothing enforces the sum of the runners' per-job memory caps. Each job container is correctly capped (--memory=10g, verified HostConfig.Memory=10737418240), but 6 slots × 10 GiB = 60 GiB on a 25 GiB host that also runs prod media (jellyfin / frigate / prod ersatztv). On 2026-07-17 that host hit load 713 with 21 GiB swapped, and telegraf could no longer get scheduled to report it.

server-management#604 fixes it at the source instead of by shuffling lanes:

Lane Before After #604
ubuntu-latest 4 slots 5 (ci-runner 48 GiB @ capacity 4 + 1 bumblebee overflow)
small 4 slots × 10g on the prod host sized for genuinely-tiny jobs

With 5 ubuntu-latest slots against this repo's 5 ubuntu-latest jobs, the queue #390 was dodging is gone — and these two dotnet jobs go back to a lane that can actually give them memory.

What this does NOT do

It does not let the small lane's caps shrink. build (Build & push image) still runs there, and it is the very job #570 measured pegging 5.999/6 GiB — so that lane keeps a 10g-capable slot either way. I'd assumed otherwise at first; #574's comment ("the build job … and docs-reminder") is correct and I was wrong.

The real unlock is #406 item 1 — disabling Roslyn's persistent VBCSCompiler in CI (7.8 GB RSS observed on the host, the single largest consumer). If that drops peak RSS to ~4 GB, every cap on both runners can come down and the lanes get materially wider. Every capacity number in #604 is downstream of that one measurement, so it's worth doing next.

Risk

Low — lane routing only. Worth watching that the first run's api-docs / format land on ubuntu-latest and that queue wait doesn't regress; if it does, that's #604's problem, not a reason to move them back here.

Refs #406, server-management#604. Supersedes the lane half of #390.

Part 3 of #406, split out because it is order-coupled with server-management#604's runner re-sizing (which has already landed on the bumblebee side). ## What Two `runs-on: small` → `runs-on: ubuntu-latest`, plus updated rationale comments. No other change. ## Why #390 moved these onto the `small` lane to dodge a ~29 min queue in the contended `ubuntu-latest` lane. The queue was real — but the lane was the wrong place to fix it, and **#390's own comment flagged exactly why**: > Caveat: on an API-touching PR this job does a full `dotnet build`, so it is not always a "small" job; capacity 4 absorbs that. "Capacity 4 absorbs that" held only because **nothing enforces the sum** of the runners' per-job memory caps. Each job container is correctly capped (`--memory=10g`, verified `HostConfig.Memory=10737418240`), but 6 slots × 10 GiB = **60 GiB on a 25 GiB host** that also runs prod media (jellyfin / frigate / prod ersatztv). On 2026-07-17 that host hit **load 713 with 21 GiB swapped**, and telegraf could no longer get scheduled to report it. server-management#604 fixes it at the source instead of by shuffling lanes: | Lane | Before | After #604 | |---|---|---| | `ubuntu-latest` | 4 slots | **5** (ci-runner 48 GiB @ capacity 4 + 1 bumblebee overflow) | | `small` | 4 slots × 10g on the prod host | sized for genuinely-tiny jobs | With 5 `ubuntu-latest` slots against this repo's 5 `ubuntu-latest` jobs, the queue #390 was dodging is gone — and these two dotnet jobs go back to a lane that can actually give them memory. ## What this does NOT do It does **not** let the `small` lane's caps shrink. `build` (Build & push image) still runs there, and it is the very job #570 measured *pegging 5.999/6 GiB* — so that lane keeps a 10g-capable slot either way. I'd assumed otherwise at first; #574's comment (*"the `build` job … and `docs-reminder`"*) is correct and I was wrong. **The real unlock is #406 item 1** — disabling Roslyn's persistent `VBCSCompiler` in CI (7.8 GB RSS observed on the host, the single largest consumer). If that drops peak RSS to ~4 GB, every cap on both runners can come down and the lanes get materially wider. Every capacity number in #604 is downstream of that one measurement, so it's worth doing next. ## Risk Low — lane routing only. Worth watching that the first run's `api-docs` / `format` land on `ubuntu-latest` and that queue wait doesn't regress; if it does, that's #604's problem, not a reason to move them back here. Refs #406, server-management#604. Supersedes the lane half of #390.
timothy added 1 commit 2026-07-17 14:18:07 +02:00
ci: move api-docs and format back to ubuntu-latest (refs #406)
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 6s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 5s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 5m29s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 11s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 11s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 5m11s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 10m14s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
17d65909b6
#390 moved these two onto the `small` lane to dodge a ~29 min queue in the
contended `ubuntu-latest` lane. The queue was real, but the lane was the
wrong place to fix it, and #390's own rationale flagged why:

    Caveat: on an API-touching PR this job does a full `dotnet build`, so it
    is not always a "small" job; capacity 4 absorbs that.

"Capacity 4 absorbs that" held only because nothing enforced the SUM of the
runners' per-job memory caps. Each job container is correctly capped at
--memory=10g, but 6 slots x 10g = 60 GiB sat on a 25 GiB host that also runs
prod media. On 2026-07-17 it went to load 713 with 21 GiB swapped, and
telegraf could no longer get scheduled to report it.

server-management#604 fixes that at the source rather than by shuffling
lanes: the `ubuntu-latest` lane now has 5 slots (ci-runner grown to 48 GiB at
capacity 4, plus a bumblebee overflow slot) against this repo's 5
ubuntu-latest jobs, and the `small` lane is sized for genuinely-tiny jobs.
So the queue #390 was dodging is gone, and these two dotnet jobs go back
where they can actually be given memory.

Note this does NOT by itself shrink the `small` lane's caps: `build`
(Build & push image) still runs there and is the very job #570 measured
pegging 5.999/6 GiB, so that lane keeps a 10g-capable slot either way. The
real unlock is disabling Roslyn's persistent VBCSCompiler in CI (#406) —
7.8 GB RSS observed — which would let every cap drop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
Owner

Independent review: exempted under the skip rubric (stated, not silent — per the handoff lore's skip rubric).

This diff is 2 lines of real change (runs-on: small → runs-on: ubuntu-latest ×2) plus rationale comments in a workflow file. No C#, no locks/concurrency, no auth, no API write path, no DB migration, well under the ~150-line threshold. Nothing an independent reviewer could catch that the run itself doesn't.

Verification is the PR's own CI run, which is stronger than a read-through here: pull_request events execute the head branch's workflow, so run 739 actually scheduled api-docs and format on ubuntu-latest — both green in 11s each. All 9 checks green (build correctly skipped on PR).

Merging as part 3 of #406; body says Refs #406, not fixes, so #406 stays open for items 1 (Roslyn VBCSCompiler) and 2 (uncapped services: mysql), which I'm picking up next in a fresh worktree off main.

Review-verdict: MERGEABLE @ 17d65909b6

**Independent review: exempted under the skip rubric** (stated, not silent — per the handoff lore's skip rubric). This diff is 2 lines of real change (`runs-on: small` → `runs-on: ubuntu-latest` ×2) plus rationale comments in a workflow file. No C#, no locks/concurrency, no auth, no API write path, no DB migration, well under the ~150-line threshold. Nothing an independent reviewer could catch that the run itself doesn't. **Verification is the PR's own CI run**, which is stronger than a read-through here: `pull_request` events execute the *head branch's* workflow, so run 739 actually scheduled `api-docs` and `format` on `ubuntu-latest` — both green in 11s each. All 9 checks green (`build` correctly skipped on PR). Merging as part 3 of #406; body says `Refs #406`, not `fixes`, so #406 stays open for items 1 (Roslyn `VBCSCompiler`) and 2 (uncapped `services:` mysql), which I'm picking up next in a fresh worktree off main. Review-verdict: MERGEABLE @ 17d65909b6228716b015b16850bb328a6e6ad0ba
timothy merged commit 76c710dfa5 into main 2026-07-17 15:09:11 +02:00
Sign in to join this conversation.