ersatztv/docs/decisions/records/ci/runner-placement.md

key, title, status, since, supersedes, superseded-by, stale-after, rule, signals, mechanics, sources

key	title	status	since	supersedes	superseded-by	stale-after	rule	signals	mechanics	sources
ci.runner-placement	2026-07-17 — No persistent compiler servers in CI; every `services:` container gets an explicit cap; #390's small-lane move reversed (#406)	active	2026-07-17	none	none	2027-01-15	No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's).	CI memory/swap thrash · paths: `.gitea/workflows/*.yml`, `docker/Dockerfile` · issues: #406, #390 (prose-reversed, no standalone record), server-management#604, server-management#570	`docs/ci-cd.md` → CI build memory; `scripts/ci-peak-anon.sh`	the 2026-07-17 bumblebee incident — load 340, 21 GiB swapped, ~238 MiB free, prod down until reboot (server-management#604) · host sizing at the time: 25 GiB / 12 cores

Three CI changes, all downstream of one incident: on 2026-07-17 bumblebee (the prod media Docker host, 25 GiB / 12 cores) hit load 340 with 21 GiB swapped and ~238 MiB free, taking prod ersatztv/jellyfin down until reboot. Prod ersatztv itself was healthy at 168 MiB throughout — the thrash was CI-induced, triggered by a burst of parallel merges to main. Infra sizing is server-management#604's boundary; these three levers live in this repo and each does more than any capacity knob.

1. No persistent compiler servers. VBCSCompiler is a persistent Roslyn server: it outlives the dotnet build that started it and holds its heap for the next one. Measured at 7.8 GB RSS live on bumblebee — the single largest consumer on the box, and the actual reason each job needed a 10 GiB cap. In CI it buys nothing: each job container is torn down at the end of the run, so there is never a "next build" to warm. The workflow's top-level env: now sets UseSharedCompilation=false, DOTNET_CLI_USE_MSBUILD_SERVER=0, MSBUILDDISABLENODEREUSE=1 — MSBuild properties set as env vars so they apply to every dotnet call without touching each call site (MSBuild surfaces env vars as properties; UseSharedCompilation only defaults to true when empty, so the env var wins). Verified locally: a default build leaves 1 VBCSCompiler alive, the same build under these vars leaves 0, and ErsatzTV.sln still builds clean (0 errors).

Why the Dockerfile also sets them — and this is the part the issue's suggestion would have missed: the workflow env: reaches the runner-side dotnet jobs only. The build job compiles inside docker build, where it does not propagate, so the SDK stage of docker/Dockerfile sets the same three as ENV. That is precisely the job server-management#570 measured pegging 5.999/6 GiB — the one that most needs it. Build-stage only; the final image is FROM runtime-base, so nothing lands in the shipped image or affects runtime.

Trade-off accepted: without the shared server each project's csc is a fresh process, which costs some build time. Worth it — the memory spike is what takes prod down, and the cap sizing that spike forces is what starves the lanes.

2. services: containers do not inherit the runner's cap. A runner's container.options (--cpus=4 --memory=10g) applies to the job container only. Verified by inspecting a live migrations job: the job container reported HostConfig.Memory=10737418240, its mysql:8.4 service reported mem=0 nanocpus=0 — unbounded. Every migrations run was adding an uncapped MySQL to an already-tight host. Now --memory=2g --memory-swap=2g --cpus=2.

Why --memory-swap is not redundant (cold-review catch, and the sharpest thing in this change): Docker defaults an unset --memory-swap to twice --memory, so --memory=2g alone grants 2g RAM plus 2g of swap — verified live: --memory=2g → memory.max=2147483648 and memory.swap.max=2147483648; --memory=2g --memory-swap=2g → memory.swap.max=0. Capping RAM while silently permitting swap is close to the worst outcome on the host whose swap thrash is the entire reason for the cap, and a swapping mysqld mid-DDL is exactly the pathology behind the known Command Timeout expired migrations flake — i.e. the naive cap could have made that flake worse. Standing rule: prefer a loud OOM over silent swapping. An OOM is an unambiguous "raise the cap" signal; swapping just degrades everything and blames something else.

The same 2× applies to the runners' 10g job slots — each is really 10 GiB RAM plus 10 GiB swap, so the "60 GiB promise on a 25 GiB host" understates by 2× and is a plausible direct mechanism for the incident's 21 GiB swapped. That is server-management#604's boundary; reported there.

On 2g, honestly: 543 MiB is init+idle, not the 787-migration replay (which grows caches idle never touches), so 2g is a measured floor plus headroom, not a measured ceiling — the migrations job going green is what validates it. --cpus=2 has no measurement behind it at all; 787 sequential DDL statements on one connection are ~1-core-bound, so it is judgement. Recording that rather than dressing a guess as measurement — the same failure this entry criticises below.

Standing rule: any new services: container needs its own explicit cap — it will not inherit one, and --memory without --memory-swap silently grants 2× in swap.

3. #390's small-lane move for api-docs/format reversed. #390 moved them to dodge a ~29 min ubuntu-latest queue. The queue was real, but the lane was the wrong fix, and #390's own comment flagged why: "on an API-touching PR this job does a full dotnet build, so it is not always a 'small' job; capacity 4 absorbs that." "Capacity 4 absorbs that" held only because nothing enforces the sum of the lanes' caps — 6 slots × 10 GiB on a 25 GiB host is a 60 GiB promise. These are not small jobs: a live docker stats caught the format job container at 3.95 GiB, which #604's re-sized 2 GiB small lane would OOM-kill outright. #604 grows ubuntu-latest to 5 slots (48 GiB ci-runner at capacity 4 + a bumblebee overflow slot) and fixes the queue at the source. This one is order-coupled with #604: the small lane's caps can't tighten until it lands.

Measurement is now continuous, not a one-off. The test job's last step reports the cgroup's memory.peak plus an anon/file breakdown (continue-on-error, tolerates absence). #604 sizes both runners' caps on that number, and until now it was inherited rather than measured — the 10g cap traces back to #570 observing a different job entirely. Two things that look like details but are the whole point: it must run last (memory.peak read at step N reports the peak only up to N, so an earlier placement silently excludes the job's later workload), and continue-on-error — not if: always() — is what makes it advisory (always() controls whether a step runs, not whether its failure fails the job, and defaults.run.shell: bash means -e is on).

And then the instrument taught us the lesson twice, both times at our own expense. The first reading came back peak 8305 MiB. memory.peak is the high-water mark of memory.current, which charges page cache as well as anonymous memory — proven: a container with anon=0 that merely reads an 800 MB file reports memory.peak=826 MiB, file=800 MiB. Page cache is reclaimed under a tighter cap, not OOM-killed, so a large peak that is mostly file is not evidence a cap must stay high. anon is what forces an OOM; size caps on it. The step now prints the split (end-of-job, so indicative rather than peak-instant); a true peak-anon sample is #412.

Then we made the same mistake in the opposite direction. Having established that peak overstates, the docs (and a report to #604) leaned to "so this number is probably mostly cache." That was a guess about magnitude dressed in a verified fact about mechanism — and an independent probe killed it: a full solution build in the CI image with shared compilation off measured peak 9457 MiB / anon 7134 MiB / file 421 MiB. Anon dominated. So a 6g cap looks unsafe, #570's "6g proved too tight" is the rule rather than an outlier, and #406's premise ("if this brings peak RSS well under 6 GiB, the whole budget loosens") is looking dead — the 7134 MiB was measured with shared compilation already off. The switches are still right (no persistent 7.8 GB server between builds); the looser budget they were supposed to buy is not.

What is NOT established: there is still no pre-change baseline from this instrument — the 7.8 GB VBCSCompiler figure was measured host-wide across concurrent jobs, not inside one job container — and the anon figure above is one probe, not the test job. #412 covers the real A/B. What is established: no persistent compiler server survives a build, and migrations is green with mysql capped at 2g with swap disabled.

The lesson generalizes, and note it bit this change twice — once in the issue's premise and once in our own instrument: this repo's CI perf work keeps stating numbers from plausibility rather than measurement (see #390's "2–4min" apt-ffmpeg estimate; real 110s, and not load-bearing). Measuring the wrong quantity precisely is the same failure wearing a lab coat.

What this does NOT shrink: format. dotnet format loads Roslyn in-process via MSBuildWorkspace and never spawns csc, so its measured 3.95 GiB is untouched by any of this. Do not size the small lane expecting otherwise.

Not addressed here: the 12–35 min queue waits (server-management#604) and the redundant triple-build (#398).