Part 3 of #406, split out because it is order-coupled with server-management#604's runner re-sizing (which has already landed on the bumblebee side).
What
Two runs-on: small → runs-on: ubuntu-latest, plus updated rationale comments. No other change.
Why
#390 moved these onto the small lane to dodge a ~29 min queue in the contended ubuntu-latest lane. The queue was real — but the lane was the wrong place to fix it, and #390's own comment flagged exactly why:
Caveat: on an API-touching PR this job does a full dotnet build, so it is not always a "small" job; capacity 4 absorbs that.
"Capacity 4 absorbs that" held only because nothing enforces the sum of the runners' per-job memory caps. Each job container is correctly capped (--memory=10g, verified HostConfig.Memory=10737418240), but 6 slots × 10 GiB = 60 GiB on a 25 GiB host that also runs prod media (jellyfin / frigate / prod ersatztv). On 2026-07-17 that host hit load 713 with 21 GiB swapped, and telegraf could no longer get scheduled to report it.
server-management#604 fixes it at the source instead of by shuffling lanes:
With 5 ubuntu-latest slots against this repo's 5 ubuntu-latest jobs, the queue #390 was dodging is gone — and these two dotnet jobs go back to a lane that can actually give them memory.
What this does NOT do
It does not let the small lane's caps shrink. build (Build & push image) still runs there, and it is the very job #570 measured pegging 5.999/6 GiB — so that lane keeps a 10g-capable slot either way. I'd assumed otherwise at first; #574's comment ("the build job … and docs-reminder") is correct and I was wrong.
The real unlock is #406 item 1 — disabling Roslyn's persistent VBCSCompiler in CI (7.8 GB RSS observed on the host, the single largest consumer). If that drops peak RSS to ~4 GB, every cap on both runners can come down and the lanes get materially wider. Every capacity number in #604 is downstream of that one measurement, so it's worth doing next.
Risk
Low — lane routing only. Worth watching that the first run's api-docs / format land on ubuntu-latest and that queue wait doesn't regress; if it does, that's #604's problem, not a reason to move them back here.
Refs #406, server-management#604. Supersedes the lane half of #390.
Part 3 of #406, split out because it is order-coupled with server-management#604's runner re-sizing (which has already landed on the bumblebee side).
## What
Two `runs-on: small` → `runs-on: ubuntu-latest`, plus updated rationale comments. No other change.
## Why
#390 moved these onto the `small` lane to dodge a ~29 min queue in the contended `ubuntu-latest` lane. The queue was real — but the lane was the wrong place to fix it, and **#390's own comment flagged exactly why**:
> Caveat: on an API-touching PR this job does a full `dotnet build`, so it is not always a "small" job; capacity 4 absorbs that.
"Capacity 4 absorbs that" held only because **nothing enforces the sum** of the runners' per-job memory caps. Each job container is correctly capped (`--memory=10g`, verified `HostConfig.Memory=10737418240`), but 6 slots × 10 GiB = **60 GiB on a 25 GiB host** that also runs prod media (jellyfin / frigate / prod ersatztv). On 2026-07-17 that host hit **load 713 with 21 GiB swapped**, and telegraf could no longer get scheduled to report it.
server-management#604 fixes it at the source instead of by shuffling lanes:
| Lane | Before | After #604 |
|---|---|---|
| `ubuntu-latest` | 4 slots | **5** (ci-runner 48 GiB @ capacity 4 + 1 bumblebee overflow) |
| `small` | 4 slots × 10g on the prod host | sized for genuinely-tiny jobs |
With 5 `ubuntu-latest` slots against this repo's 5 `ubuntu-latest` jobs, the queue #390 was dodging is gone — and these two dotnet jobs go back to a lane that can actually give them memory.
## What this does NOT do
It does **not** let the `small` lane's caps shrink. `build` (Build & push image) still runs there, and it is the very job #570 measured *pegging 5.999/6 GiB* — so that lane keeps a 10g-capable slot either way. I'd assumed otherwise at first; #574's comment (*"the `build` job … and `docs-reminder`"*) is correct and I was wrong.
**The real unlock is #406 item 1** — disabling Roslyn's persistent `VBCSCompiler` in CI (7.8 GB RSS observed on the host, the single largest consumer). If that drops peak RSS to ~4 GB, every cap on both runners can come down and the lanes get materially wider. Every capacity number in #604 is downstream of that one measurement, so it's worth doing next.
## Risk
Low — lane routing only. Worth watching that the first run's `api-docs` / `format` land on `ubuntu-latest` and that queue wait doesn't regress; if it does, that's #604's problem, not a reason to move them back here.
Refs #406, server-management#604. Supersedes the lane half of #390.
#390 moved these two onto the `small` lane to dodge a ~29 min queue in the
contended `ubuntu-latest` lane. The queue was real, but the lane was the
wrong place to fix it, and #390's own rationale flagged why:
Caveat: on an API-touching PR this job does a full `dotnet build`, so it
is not always a "small" job; capacity 4 absorbs that.
"Capacity 4 absorbs that" held only because nothing enforced the SUM of the
runners' per-job memory caps. Each job container is correctly capped at
--memory=10g, but 6 slots x 10g = 60 GiB sat on a 25 GiB host that also runs
prod media. On 2026-07-17 it went to load 713 with 21 GiB swapped, and
telegraf could no longer get scheduled to report it.
server-management#604 fixes that at the source rather than by shuffling
lanes: the `ubuntu-latest` lane now has 5 slots (ci-runner grown to 48 GiB at
capacity 4, plus a bumblebee overflow slot) against this repo's 5
ubuntu-latest jobs, and the `small` lane is sized for genuinely-tiny jobs.
So the queue #390 was dodging is gone, and these two dotnet jobs go back
where they can actually be given memory.
Note this does NOT by itself shrink the `small` lane's caps: `build`
(Build & push image) still runs there and is the very job #570 measured
pegging 5.999/6 GiB, so that lane keeps a 10g-capable slot either way. The
real unlock is disabling Roslyn's persistent VBCSCompiler in CI (#406) —
7.8 GB RSS observed — which would let every cap drop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Independent review: exempted under the skip rubric (stated, not silent — per the handoff lore's skip rubric).
This diff is 2 lines of real change (runs-on: small → runs-on: ubuntu-latest ×2) plus rationale comments in a workflow file. No C#, no locks/concurrency, no auth, no API write path, no DB migration, well under the ~150-line threshold. Nothing an independent reviewer could catch that the run itself doesn't.
Verification is the PR's own CI run, which is stronger than a read-through here: pull_request events execute the head branch's workflow, so run 739 actually scheduled api-docs and format on ubuntu-latest — both green in 11s each. All 9 checks green (build correctly skipped on PR).
Merging as part 3 of #406; body says Refs #406, not fixes, so #406 stays open for items 1 (Roslyn VBCSCompiler) and 2 (uncapped services: mysql), which I'm picking up next in a fresh worktree off main.
**Independent review: exempted under the skip rubric** (stated, not silent — per the handoff lore's skip rubric).
This diff is 2 lines of real change (`runs-on: small` → `runs-on: ubuntu-latest` ×2) plus rationale comments in a workflow file. No C#, no locks/concurrency, no auth, no API write path, no DB migration, well under the ~150-line threshold. Nothing an independent reviewer could catch that the run itself doesn't.
**Verification is the PR's own CI run**, which is stronger than a read-through here: `pull_request` events execute the *head branch's* workflow, so run 739 actually scheduled `api-docs` and `format` on `ubuntu-latest` — both green in 11s each. All 9 checks green (`build` correctly skipped on PR).
Merging as part 3 of #406; body says `Refs #406`, not `fixes`, so #406 stays open for items 1 (Roslyn `VBCSCompiler`) and 2 (uncapped `services:` mysql), which I'm picking up next in a fresh worktree off main.
Review-verdict: MERGEABLE @ 17d65909b6228716b015b16850bb328a6e6ad0ba
timothy
merged commit 76c710dfa5 into main2026-07-17 15:09:11 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part 3 of #406, split out because it is order-coupled with server-management#604's runner re-sizing (which has already landed on the bumblebee side).
What
Two
runs-on: small→runs-on: ubuntu-latest, plus updated rationale comments. No other change.Why
#390 moved these onto the
smalllane to dodge a ~29 min queue in the contendedubuntu-latestlane. The queue was real — but the lane was the wrong place to fix it, and #390's own comment flagged exactly why:"Capacity 4 absorbs that" held only because nothing enforces the sum of the runners' per-job memory caps. Each job container is correctly capped (
--memory=10g, verifiedHostConfig.Memory=10737418240), but 6 slots × 10 GiB = 60 GiB on a 25 GiB host that also runs prod media (jellyfin / frigate / prod ersatztv). On 2026-07-17 that host hit load 713 with 21 GiB swapped, and telegraf could no longer get scheduled to report it.server-management#604 fixes it at the source instead of by shuffling lanes:
ubuntu-latestsmallWith 5
ubuntu-latestslots against this repo's 5ubuntu-latestjobs, the queue #390 was dodging is gone — and these two dotnet jobs go back to a lane that can actually give them memory.What this does NOT do
It does not let the
smalllane's caps shrink.build(Build & push image) still runs there, and it is the very job #570 measured pegging 5.999/6 GiB — so that lane keeps a 10g-capable slot either way. I'd assumed otherwise at first; #574's comment ("thebuildjob … anddocs-reminder") is correct and I was wrong.The real unlock is #406 item 1 — disabling Roslyn's persistent
VBCSCompilerin CI (7.8 GB RSS observed on the host, the single largest consumer). If that drops peak RSS to ~4 GB, every cap on both runners can come down and the lanes get materially wider. Every capacity number in #604 is downstream of that one measurement, so it's worth doing next.Risk
Low — lane routing only. Worth watching that the first run's
api-docs/formatland onubuntu-latestand that queue wait doesn't regress; if it does, that's #604's problem, not a reason to move them back here.Refs #406, server-management#604. Supersedes the lane half of #390.
ubuntu-latest(refs #406)Independent review: exempted under the skip rubric (stated, not silent — per the handoff lore's skip rubric).
This diff is 2 lines of real change (
runs-on: small→runs-on: ubuntu-latest×2) plus rationale comments in a workflow file. No C#, no locks/concurrency, no auth, no API write path, no DB migration, well under the ~150-line threshold. Nothing an independent reviewer could catch that the run itself doesn't.Verification is the PR's own CI run, which is stronger than a read-through here:
pull_requestevents execute the head branch's workflow, so run 739 actually scheduledapi-docsandformatonubuntu-latest— both green in 11s each. All 9 checks green (buildcorrectly skipped on PR).Merging as part 3 of #406; body says
Refs #406, notfixes, so #406 stays open for items 1 (RoslynVBCSCompiler) and 2 (uncappedservices:mysql), which I'm picking up next in a fresh worktree off main.Review-verdict: MERGEABLE @
17d65909b6