---
key: ci.runner-placement
title: '2026-07-17 — No persistent compiler servers in CI; every `services:` container gets an explicit cap; #390''s small-lane move reversed (#406)'
status: active
since: '2026-07-17'
supersedes: none
superseded-by: none
stale-after: '2027-01-15'
rule: No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's).
signals: 'CI memory/swap thrash · paths: `.gitea/workflows/*.yml`, `docker/Dockerfile` · issues: #406, #390 (prose-reversed, no standalone record), server-management#604, server-management#570'
mechanics: '`docs/ci-cd.md` → CI build memory; `scripts/ci-peak-anon.sh`'
sources: 'the 2026-07-17 bumblebee incident — load 340, 21 GiB swapped, ~238 MiB free, prod down until reboot (server-management#604) · host sizing at the time: 25 GiB / 12 cores'
---

Three CI changes, all downstream of one incident: on 2026-07-17 bumblebee (the **prod media** Docker
host, 25 GiB / 12 cores) hit load **340** with **21 GiB swapped** and ~238 MiB free, taking prod
ersatztv/jellyfin down until reboot. Prod ersatztv itself was healthy at 168 MiB throughout — the
thrash was **CI-induced**, triggered by a burst of parallel merges to `main`. Infra sizing is
server-management#604's boundary; these three levers live in this repo and each does more than any
capacity knob.

**1. No persistent compiler servers.** `VBCSCompiler` is a *persistent* Roslyn server: it outlives
the `dotnet build` that started it and holds its heap for the next one. Measured at **7.8 GB RSS**
live on bumblebee — the single largest consumer on the box, and the actual reason each job needed a
10 GiB cap. In CI it buys **nothing**: each job container is torn down at the end of the run, so
there is never a "next build" to warm. The workflow's top-level `env:` now sets
`UseSharedCompilation=false`, `DOTNET_CLI_USE_MSBUILD_SERVER=0`, `MSBUILDDISABLENODEREUSE=1` —
MSBuild properties set as env vars so they apply to every `dotnet` call without touching each call
site (MSBuild surfaces env vars as properties; `UseSharedCompilation` only defaults to `true` when
empty, so the env var wins). Verified locally: a default build leaves 1 `VBCSCompiler` alive, the
same build under these vars leaves **0**, and `ErsatzTV.sln` still builds clean (0 errors).

**Why the Dockerfile also sets them** — and this is the part the issue's suggestion would have
missed: the workflow `env:` reaches the *runner-side* dotnet jobs only. The `build` job compiles
inside `docker build`, where it does not propagate, so the SDK stage of `docker/Dockerfile` sets the
same three as `ENV`. That is precisely the job server-management#570 measured pegging **5.999/6
GiB** — the one that most needs it. Build-stage only; the final image is `FROM runtime-base`, so
nothing lands in the shipped image or affects runtime.

**Trade-off accepted**: without the shared server each project's `csc` is a fresh process, which
costs some build time. Worth it — the memory spike is what takes prod down, and the cap sizing that
spike forces is what starves the lanes.

**2. `services:` containers do not inherit the runner's cap.** A runner's `container.options`
(`--cpus=4 --memory=10g`) applies to the **job container only**. Verified by inspecting a live
`migrations` job: the job container reported `HostConfig.Memory=10737418240`, its `mysql:8.4`
service reported `mem=0 nanocpus=0` — **unbounded**. Every migrations run was adding an uncapped
MySQL to an already-tight host. Now `--memory=2g --memory-swap=2g --cpus=2`.

**Why `--memory-swap` is not redundant** (cold-review catch, and the sharpest thing in this change):
Docker defaults an unset `--memory-swap` to **twice** `--memory`, so `--memory=2g` alone grants 2g
RAM **plus 2g of swap** — verified live: `--memory=2g` → `memory.max=2147483648` **and**
`memory.swap.max=2147483648`; `--memory=2g --memory-swap=2g` → `memory.swap.max=0`. Capping RAM
while silently permitting swap is close to the worst outcome **on the host whose swap thrash is the
entire reason for the cap**, and a swapping mysqld mid-DDL is exactly the pathology behind the known
`Command Timeout expired` migrations flake — i.e. the naive cap could have made that flake worse.
**Standing rule: prefer a loud OOM over silent swapping.** An OOM is an unambiguous "raise the cap"
signal; swapping just degrades everything and blames something else.

**The same 2× applies to the runners' 10g job slots** — each is really 10 GiB RAM *plus* 10 GiB
swap, so the "60 GiB promise on a 25 GiB host" understates by 2× and is a plausible direct mechanism
for the incident's 21 GiB swapped. That is server-management#604's boundary; reported there.

**On 2g, honestly**: 543 MiB is init+idle, **not** the 787-migration replay (which grows caches idle
never touches), so 2g is a measured *floor* plus headroom, not a measured ceiling — the `migrations`
job going green is what validates it. `--cpus=2` has no measurement behind it at all; 787 sequential
DDL statements on one connection are ~1-core-bound, so it is judgement. Recording that rather than
dressing a guess as measurement — the same failure this entry criticises below.

**Standing rule: any new `services:` container needs its own explicit cap** — it will not inherit
one, and `--memory` without `--memory-swap` silently grants 2× in swap.

**3. #390's `small`-lane move for `api-docs`/`format` reversed.** #390 moved them to dodge a ~29 min
`ubuntu-latest` queue. The queue was real, but the lane was the wrong fix, and **#390's own comment
flagged why**: *"on an API-touching PR this job does a full `dotnet build`, so it is not always a
'small' job; capacity 4 absorbs that."* "Capacity 4 absorbs that" held only because **nothing
enforces the sum** of the lanes' caps — 6 slots × 10 GiB on a 25 GiB host is a 60 GiB promise. These
are not small jobs: a live `docker stats` caught the `format` job container at **3.95 GiB**, which
#604's re-sized 2 GiB `small` lane would OOM-kill outright. #604 grows `ubuntu-latest` to 5 slots
(48 GiB ci-runner at capacity 4 + a bumblebee overflow slot) and fixes the queue at the source. This
one is **order-coupled with #604**: the small lane's caps can't tighten until it lands.

**Measurement is now continuous, not a one-off.** The `test` job's **last** step reports the
cgroup's `memory.peak` plus an `anon`/`file` breakdown (`continue-on-error`, tolerates absence).
#604 sizes both runners' caps on that number, and until now it was *inherited* rather than measured
— the 10g cap traces back to #570 observing a different job entirely. Two things that look like
details but are the whole point: it must run **last** (`memory.peak` read at step N reports the peak
only up to N, so an earlier placement silently excludes the job's later workload), and
`continue-on-error` — not `if: always()` — is what makes it advisory (`always()` controls whether a
step *runs*, not whether its failure fails the job, and `defaults.run.shell: bash` means `-e` is on).

**And then the instrument taught us the lesson twice, both times at our own expense.** The first
reading came back `peak 8305 MiB`. `memory.peak` is the high-water mark of `memory.current`, which
charges **page cache** as well as anonymous memory — proven: a container with `anon=0` that merely
reads an 800 MB file reports `memory.peak=826 MiB`, `file=800 MiB`. Page cache is *reclaimed* under
a tighter cap, not OOM-killed, so a large peak that is mostly `file` is **not** evidence a cap must
stay high. `anon` is what forces an OOM; size caps on it. The step now prints the split (end-of-job,
so indicative rather than peak-instant); a true peak-anon sample is **#412**.

**Then we made the same mistake in the opposite direction.** Having established that peak
*overstates*, the docs (and a report to #604) leaned to "so this number is probably mostly cache."
That was a guess about *magnitude* dressed in a verified fact about *mechanism* — and an independent
probe killed it: a full solution build in the CI image with shared compilation off measured `peak
9457 MiB` / **`anon 7134 MiB`** / `file 421 MiB`. **Anon dominated.** So a 6g cap looks *unsafe*,
#570's "6g proved too tight" is the rule rather than an outlier, and **#406's premise ("if this
brings peak RSS well under 6 GiB, the whole budget loosens") is looking dead** — the 7134 MiB was
measured *with* shared compilation already off. The switches are still right (no persistent 7.8 GB
server between builds); the looser budget they were supposed to buy is not.

**What is NOT established:** there is still no pre-change baseline from this instrument — the 7.8 GB
`VBCSCompiler` figure was measured host-wide across concurrent jobs, not inside one job container —
and the anon figure above is one probe, not the `test` job. #412 covers the real A/B. What *is*
established: no persistent compiler server survives a build, and `migrations` is green with mysql
capped at 2g with swap disabled.

The lesson generalizes, and note it bit *this* change twice — once in the issue's premise and once
in our own instrument: this repo's CI perf work keeps stating numbers from plausibility rather than
measurement (see #390's "2–4min" apt-ffmpeg estimate; real 110s, and not load-bearing). Measuring
the wrong quantity precisely is the same failure wearing a lab coat.

**What this does NOT shrink**: `format`. `dotnet format` loads Roslyn in-process via
MSBuildWorkspace and never spawns `csc`, so its measured 3.95 GiB is untouched by any of this. Do
not size the `small` lane expecting otherwise.

**Not addressed here**: the 12–35 min queue waits (server-management#604) and the redundant
triple-build (#398).
