--- key: ci.runner-placement title: '2026-07-17 — No persistent compiler servers in CI; every `services:` container gets an explicit cap; #390''s small-lane move reversed (#406)' status: active since: '2026-07-17' supersedes: none superseded-by: none stale-after: '2027-01-15' rule: No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's). signals: 'CI memory/swap thrash · paths: `.gitea/workflows/*.yml`, `docker/Dockerfile` · issues: #406, #390 (prose-reversed, no standalone record), server-management#604, server-management#570' mechanics: '`docs/ci-cd.md` → CI build memory; `scripts/ci-peak-anon.sh`' sources: 'the 2026-07-17 bumblebee incident — load 340, 21 GiB swapped, ~238 MiB free, prod down until reboot (server-management#604) · host sizing at the time: 25 GiB / 12 cores' --- Three CI changes, all downstream of one incident: on 2026-07-17 bumblebee (the **prod media** Docker host, 25 GiB / 12 cores) hit load **340** with **21 GiB swapped** and ~238 MiB free, taking prod ersatztv/jellyfin down until reboot. Prod ersatztv itself was healthy at 168 MiB throughout — the thrash was **CI-induced**, triggered by a burst of parallel merges to `main`. Infra sizing is server-management#604's boundary; these three levers live in this repo and each does more than any capacity knob. **1. No persistent compiler servers.** `VBCSCompiler` is a *persistent* Roslyn server: it outlives the `dotnet build` that started it and holds its heap for the next one. Measured at **7.8 GB RSS** live on bumblebee — the single largest consumer on the box, and the actual reason each job needed a 10 GiB cap. In CI it buys **nothing**: each job container is torn down at the end of the run, so there is never a "next build" to warm. The workflow's top-level `env:` now sets `UseSharedCompilation=false`, `DOTNET_CLI_USE_MSBUILD_SERVER=0`, `MSBUILDDISABLENODEREUSE=1` — MSBuild properties set as env vars so they apply to every `dotnet` call without touching each call site (MSBuild surfaces env vars as properties; `UseSharedCompilation` only defaults to `true` when empty, so the env var wins). Verified locally: a default build leaves 1 `VBCSCompiler` alive, the same build under these vars leaves **0**, and `ErsatzTV.sln` still builds clean (0 errors). **Why the Dockerfile also sets them** — and this is the part the issue's suggestion would have missed: the workflow `env:` reaches the *runner-side* dotnet jobs only. The `build` job compiles inside `docker build`, where it does not propagate, so the SDK stage of `docker/Dockerfile` sets the same three as `ENV`. That is precisely the job server-management#570 measured pegging **5.999/6 GiB** — the one that most needs it. Build-stage only; the final image is `FROM runtime-base`, so nothing lands in the shipped image or affects runtime. **Trade-off accepted**: without the shared server each project's `csc` is a fresh process, which costs some build time. Worth it — the memory spike is what takes prod down, and the cap sizing that spike forces is what starves the lanes. **2. `services:` containers do not inherit the runner's cap.** A runner's `container.options` (`--cpus=4 --memory=10g`) applies to the **job container only**. Verified by inspecting a live `migrations` job: the job container reported `HostConfig.Memory=10737418240`, its `mysql:8.4` service reported `mem=0 nanocpus=0` — **unbounded**. Every migrations run was adding an uncapped MySQL to an already-tight host. Now `--memory=2g --memory-swap=2g --cpus=2`. **Why `--memory-swap` is not redundant** (cold-review catch, and the sharpest thing in this change): Docker defaults an unset `--memory-swap` to **twice** `--memory`, so `--memory=2g` alone grants 2g RAM **plus 2g of swap** — verified live: `--memory=2g` → `memory.max=2147483648` **and** `memory.swap.max=2147483648`; `--memory=2g --memory-swap=2g` → `memory.swap.max=0`. Capping RAM while silently permitting swap is close to the worst outcome **on the host whose swap thrash is the entire reason for the cap**, and a swapping mysqld mid-DDL is exactly the pathology behind the known `Command Timeout expired` migrations flake — i.e. the naive cap could have made that flake worse. **Standing rule: prefer a loud OOM over silent swapping.** An OOM is an unambiguous "raise the cap" signal; swapping just degrades everything and blames something else. **The same 2× applies to the runners' 10g job slots** — each is really 10 GiB RAM *plus* 10 GiB swap, so the "60 GiB promise on a 25 GiB host" understates by 2× and is a plausible direct mechanism for the incident's 21 GiB swapped. That is server-management#604's boundary; reported there. **On 2g, honestly**: 543 MiB is init+idle, **not** the 787-migration replay (which grows caches idle never touches), so 2g is a measured *floor* plus headroom, not a measured ceiling — the `migrations` job going green is what validates it. `--cpus=2` has no measurement behind it at all; 787 sequential DDL statements on one connection are ~1-core-bound, so it is judgement. Recording that rather than dressing a guess as measurement — the same failure this entry criticises below. **Standing rule: any new `services:` container needs its own explicit cap** — it will not inherit one, and `--memory` without `--memory-swap` silently grants 2× in swap. **3. #390's `small`-lane move for `api-docs`/`format` reversed.** #390 moved them to dodge a ~29 min `ubuntu-latest` queue. The queue was real, but the lane was the wrong fix, and **#390's own comment flagged why**: *"on an API-touching PR this job does a full `dotnet build`, so it is not always a 'small' job; capacity 4 absorbs that."* "Capacity 4 absorbs that" held only because **nothing enforces the sum** of the lanes' caps — 6 slots × 10 GiB on a 25 GiB host is a 60 GiB promise. These are not small jobs: a live `docker stats` caught the `format` job container at **3.95 GiB**, which #604's re-sized 2 GiB `small` lane would OOM-kill outright. #604 grows `ubuntu-latest` to 5 slots (48 GiB ci-runner at capacity 4 + a bumblebee overflow slot) and fixes the queue at the source. This one is **order-coupled with #604**: the small lane's caps can't tighten until it lands. **Measurement is now continuous, not a one-off.** The `test` job's **last** step reports the cgroup's `memory.peak` plus an `anon`/`file` breakdown (`continue-on-error`, tolerates absence). #604 sizes both runners' caps on that number, and until now it was *inherited* rather than measured — the 10g cap traces back to #570 observing a different job entirely. Two things that look like details but are the whole point: it must run **last** (`memory.peak` read at step N reports the peak only up to N, so an earlier placement silently excludes the job's later workload), and `continue-on-error` — not `if: always()` — is what makes it advisory (`always()` controls whether a step *runs*, not whether its failure fails the job, and `defaults.run.shell: bash` means `-e` is on). **And then the instrument taught us the lesson twice, both times at our own expense.** The first reading came back `peak 8305 MiB`. `memory.peak` is the high-water mark of `memory.current`, which charges **page cache** as well as anonymous memory — proven: a container with `anon=0` that merely reads an 800 MB file reports `memory.peak=826 MiB`, `file=800 MiB`. Page cache is *reclaimed* under a tighter cap, not OOM-killed, so a large peak that is mostly `file` is **not** evidence a cap must stay high. `anon` is what forces an OOM; size caps on it. The step now prints the split (end-of-job, so indicative rather than peak-instant); a true peak-anon sample is **#412**. **Then we made the same mistake in the opposite direction.** Having established that peak *overstates*, the docs (and a report to #604) leaned to "so this number is probably mostly cache." That was a guess about *magnitude* dressed in a verified fact about *mechanism* — and an independent probe killed it: a full solution build in the CI image with shared compilation off measured `peak 9457 MiB` / **`anon 7134 MiB`** / `file 421 MiB`. **Anon dominated.** So a 6g cap looks *unsafe*, #570's "6g proved too tight" is the rule rather than an outlier, and **#406's premise ("if this brings peak RSS well under 6 GiB, the whole budget loosens") is looking dead** — the 7134 MiB was measured *with* shared compilation already off. The switches are still right (no persistent 7.8 GB server between builds); the looser budget they were supposed to buy is not. **What is NOT established:** there is still no pre-change baseline from this instrument — the 7.8 GB `VBCSCompiler` figure was measured host-wide across concurrent jobs, not inside one job container — and the anon figure above is one probe, not the `test` job. #412 covers the real A/B. What *is* established: no persistent compiler server survives a build, and `migrations` is green with mysql capped at 2g with swap disabled. The lesson generalizes, and note it bit *this* change twice — once in the issue's premise and once in our own instrument: this repo's CI perf work keeps stating numbers from plausibility rather than measurement (see #390's "2–4min" apt-ffmpeg estimate; real 110s, and not load-bearing). Measuring the wrong quantity precisely is the same failure wearing a lab coat. **What this does NOT shrink**: `format`. `dotnet format` loads Roslyn in-process via MSBuildWorkspace and never spawns `csc`, so its measured 3.95 GiB is untouched by any of this. Do not size the `small` lane expecting otherwise. **Not addressed here**: the 12–35 min queue waits (server-management#604) and the redundant triple-build (#398).