ci(406): disable persistent compiler servers, cap the mysql service, report peak RSS

Refs #406, server-management#604. Items 1 and 2 of #406 (item 3 landed in PR #407).

Root cause of the 2026-07-17 bumblebee incident (load 340, 21 GiB swapped, prod media
down until reboot) is CI-induced memory pressure, and the largest single consumer was
Roslyn's persistent VBCSCompiler at 7.8 GB RSS. It is a compiler *server*: it outlives
the build that started it to keep its heap warm for the next one. In CI there is no next
build -- each job container is torn down at the end of the run -- so it is pure cost, and
it is the actual reason each job needed a 10 GiB cap.

1. Disable the compiler servers via the workflow's top-level env (UseSharedCompilation,
   DOTNET_CLI_USE_MSBUILD_SERVER, MSBUILDDISABLENODEREUSE). MSBuild surfaces env vars as
   properties and only defaults UseSharedCompilation when empty, so the env var wins and
   every dotnet call is covered without touching each call site.

   Also set as ENV in the Dockerfile's SDK stage: the workflow env reaches runner-side
   jobs only, and the `build` job compiles inside `docker build` where it does not
   propagate -- that is the job server-management#570 measured pegging 5.999/6 GiB, so it
   is the one that most needs this. Build stage only; the final image is FROM
   runtime-base, so nothing lands in the shipped image.

   Verified locally: a default build leaves 1 VBCSCompiler alive, the same build under
   these vars leaves 0, and ErsatzTV.sln still builds clean (0 errors). Trade-off: csc
   runs per project instead of via the server, costing some build time.

2. Cap the mysql service (--memory=2g --cpus=2). A runner's container.options applies to
   the job container ONLY: a live migrations job reported HostConfig.Memory=10737418240
   for the job and mem=0 nanocpus=0 for its mysql service, i.e. unbounded. 2g is sized on
   measurement -- mysql:8.4 with this exact env peaked at 543 MiB during init, 481 MiB
   idle (probed on bumblebee) -- not on inheritance.

Also add a `Report peak container memory` step to the test job (cgroup memory.peak,
always(), never fails a build). server-management#604 sizes both runners' caps on that
number and it was inherited rather than measured; now every run reports it.

Docs: ci-cd.md gains the compiler-server + services-cap sections and the lane table is
corrected (PR #407 reverted the #390 lane move without updating it). decisions.md records
all three, incl. the standing rule that a services: container never inherits a cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitea/workflows/docker-build.yml

@@ -60,6 +60,26 @@ env:
   REGISTRY: 192.168.1.95:3000
   IMAGE: 192.168.1.95:3000/timothy/ersatztv
 
+  # --- CI build memory (ersatztv#406, server-management#604) ---
+  # Roslyn's `VBCSCompiler` is a *persistent* compiler server: it outlives the `dotnet build` that
+  # started it and keeps its managed heap warm for the next one. Locally that is a real speedup.
+  # In CI it buys nothing — each job container is torn down at the end of the run, so there is
+  # never a "next build" to warm — while costing a lot: 7.8 GB RSS was measured live on bumblebee,
+  # the single largest consumer on a 25 GiB host that also runs prod media. Several of those, one
+  # per concurrent job container, is what drove the host to load 340 with 21 GiB swapped.
+  #
+  # These are MSBuild properties/switches, set here as environment variables so they apply to every
+  # dotnet invocation in every job (restore/build/test/format/api-docs) without touching each call
+  # site. MSBuild surfaces environment variables as properties, and `UseSharedCompilation` is only
+  # defaulted to true when empty, so setting it here wins.
+  #
+  # NOTE: this reaches the *runner-side* dotnet jobs only. The `build` job compiles inside
+  # `docker build`, where these do not propagate — the same switches are set as ENV in the
+  # Dockerfile's SDK stage (docker/Dockerfile) to cover it.
+  UseSharedCompilation: "false"   # no persistent VBCSCompiler; csc runs per-project and exits
+  DOTNET_CLI_USE_MSBUILD_SERVER: "0"  # no persistent MSBuild server process
+  MSBUILDDISABLENODEREUSE: "1"    # MSBuild worker nodes exit with the build instead of lingering
+
 jobs:
   test:
     name: Build & test (.NET)
@@ -131,6 +151,30 @@ jobs:
           dotnet test --configuration Release --no-build --blame-hang-timeout "2m" --verbosity normal
           --collect:"XPlat Code Coverage" --settings coverlet.runsettings --results-directory ./coverage
 
+      # Peak memory of THIS job container, reported every run (ersatztv#406, server-management#604).
+      # #604 sizes the runners' per-job caps on this number, and until now it was inherited rather
+      # than measured: the 10g cap traces back to server-management#570 observing the image build
+      # peg 5.999/6 GiB, which is a different job entirely. `memory.peak` is the cgroup's
+      # high-water mark since the container started, so reading it once at the end of the job is
+      # exactly the peak — no polling needed. cgroup v2 first, v1 fallback; purely informational,
+      # so it must never fail a build (`always()` + tolerated absence).
+      - name: Report peak container memory
+        if: always()
+        run: |
+          for f in /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory/memory.max_usage_in_bytes; do
+            if [ -r "$f" ]; then
+              bytes=$(cat "$f")
+              printf 'peak container memory: %s MiB (%s bytes, from %s)\n' \
+                "$(( bytes / 1048576 ))" "$bytes" "$f"
+              if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
+                printf '**Peak container memory (test job):** %s MiB\n' \
+                  "$(( bytes / 1048576 ))" >> "$GITHUB_STEP_SUMMARY"
+              fi
+              exit 0
+            fi
+          done
+          echo "No cgroup peak-memory file readable in this container -- skipping."
+
       # Coverage reporting (ersatztv#15 scope item 4): coverlet.collector emits a Cobertura report
       # per test project (via --collect above); ReportGenerator merges them into a human-readable
       # summary printed to the log and the job step summary. No floor is enforced yet ("decide on a
@@ -180,7 +224,21 @@ jobs:
         # No host-port binding: the job reaches this service as mysql:3306 on the shared
         # runner network. Publishing 3306 made concurrent runs collide ("port is already
         # allocated") whenever two migrations jobs overlapped.
+        #
+        # `--memory`/`--cpus` here because the runner's `container.options` (`--memory=10g`)
+        # applies to the JOB container ONLY, not to `services:` — verified by inspecting a live
+        # migrations job: the job container reported HostConfig.Memory=10737418240, its mysql
+        # service reported `mem=0 nanocpus=0`, i.e. unbounded. So every migrations run was adding
+        # an uncapped MySQL to an already-tight host (ersatztv#406, server-management#604).
+        #
+        # 2g is sized on measurement, not inheritance: a mysql:8.4 container with this exact env
+        # peaked at 543 MiB during init and settled at 481 MiB idle (probed on bumblebee
+        # 2026-07-17). Applying the migrations is DDL-heavy rather than data-heavy and the InnoDB
+        # buffer pool defaults to 128M, so 2g leaves ~4x headroom. If this ever OOM-kills the
+        # service, raise it deliberately — do not remove the cap.
         options: >-
+          --memory=2g
+          --cpus=2
           --health-cmd="mysqladmin ping -h 127.0.0.1 -uroot -persatztv --silent"
           --health-interval=5s
           --health-timeout=5s

docker/Dockerfile

@@ -21,6 +21,18 @@ RUN apt-get update && \
 
 # https://hub.docker.com/_/microsoft-dotnet
 FROM mcr.microsoft.com/dotnet/sdk:10.0-noble-amd64 AS build
+
+# Disable the persistent Roslyn/MSBuild compiler servers for the in-image build (ersatztv#406,
+# server-management#604). The workflow sets these as env for the runner-side dotnet jobs, but this
+# stage compiles inside `docker build`, so the workflow's env does NOT reach it — the restore and
+# the two publishes below would otherwise spin up their own VBCSCompiler and hold its heap. This
+# is the `build` job that server-management#570 measured pegging 5.999/6 GiB.
+# Build-stage only: the final image is FROM runtime-base, so none of this lands in the shipped
+# image or affects runtime.
+ENV UseSharedCompilation=false \
+    DOTNET_CLI_USE_MSBUILD_SERVER=0 \
+    MSBUILDDISABLENODEREUSE=1
+
 RUN apt-get update && apt-get install -y ca-certificates gnupg default-jre-headless python3-pip
 WORKDIR /source
 

docs/ci-cd.md

@@ -99,20 +99,71 @@ saturated and the `small` lane idle — **queue wait exceeded every job's runtim
 | `test` | 354s | 1363s | ubuntu-latest |
 | `migrations` | 639s | 1428s | ubuntu-latest |
 | `functional-e2e` | 520s | 1447s | ubuntu-latest |
-| `api-docs` | 5s | **1722s** | ubuntu-latest → now `small` |
-| `format` | 37s | **1731s** | ubuntu-latest → now `small` |
+| `api-docs` | 5s | **1722s** | ubuntu-latest → `small` → **reverted to ubuntu-latest** (#406) |
+| `format` | 37s | **1731s** | ubuntu-latest → `small` → **reverted to ubuntu-latest** (#406) |
 | `docs-reminder` / `decisions-guard` | 10s | **5s** | small |
 
-So `api-docs` and `format` — genuinely small jobs — moved to `small`, which also drops the
-contended lane from 5 jobs to 3 and shortens the queue for the three heavy jobs. Both lanes run
-the *identical* `runner-images:ubuntu-latest` base, so `small` is a label with spare capacity, not
-a different capability. This move is only possible because those jobs now run in the **CI toolchain
-image** (below) and no longer need the runner image to supply .NET/Node. Caveat: on an
-API-touching PR `api-docs` does a full `dotnet build`, so it is not *always* small.
+`api-docs` and `format` moved to `small` because the queue wait dwarfed their runtime. Both lanes
+run the *identical* `runner-images:ubuntu-latest` base, so `small` was a label with spare capacity,
+not a different capability — a move only possible because those jobs now run in the **CI toolchain
+image** (below) and no longer need the runner image to supply .NET/Node.
 
-Queue wait is still the dominant cost and capacity is server-management's boundary — tracked in
+**Reverted 2026-07-17 (ersatztv#406 / server-management#604).** #390's own caveat — *"on an
+API-touching PR `api-docs` does a full `dotnet build`, so it is not always small"* — turned out to
+be the deciding factor, and "capacity 4 absorbs that" held only because **nothing enforces the sum**
+of the lanes' per-job caps. Each job container is correctly capped (`--memory=10g`), but 6 slots ×
+10 GiB = **60 GiB on a 25 GiB host** that also runs prod media; on 2026-07-17 bumblebee hit load
+340 with 21 GiB swapped. These are not small jobs — a live `docker stats` caught the `format` job
+container at **3.95 GiB**, which the re-sized 2 GiB `small` lane would OOM-kill outright. #604 fixes
+the queue at the source instead (`ubuntu-latest` grown to 5 slots: a 48 GiB ci-runner at capacity 4
+plus a bumblebee overflow slot), so the `small` lane can be reserved for genuinely-tiny shell jobs.
+
+Queue wait is still a dominant cost and capacity is server-management's boundary — tracked in
 **server-management#604**. The redundant triple-build behind those runtimes is **ersatztv#398**.
 
+### CI build memory: no persistent compiler servers (ersatztv#406)
+
+Roslyn's `VBCSCompiler` is a **persistent** compiler server — it outlives the `dotnet build` that
+started it and keeps its heap warm for the next one. Locally that is a genuine speedup; in CI it
+buys nothing, because each job container is torn down at the end of the run and there is never a
+"next build" to warm. It was measured at **7.8 GB RSS** on bumblebee — the single largest consumer
+on the host, and the reason each job needed a 10 GiB cap in the first place.
+
+So the workflow's top-level `env:` disables the servers for every runner-side dotnet job:
+
+| Variable | Effect |
+|---|---|
+| `UseSharedCompilation=false` | no persistent `VBCSCompiler`; `csc` runs per project and exits |
+| `DOTNET_CLI_USE_MSBUILD_SERVER=0` | no persistent MSBuild server process |
+| `MSBUILDDISABLENODEREUSE=1` | MSBuild worker nodes exit with the build instead of lingering |
+
+These are MSBuild properties set as environment variables so they apply to every `dotnet`
+invocation without touching each call site (MSBuild surfaces env vars as properties, and
+`UseSharedCompilation` is only defaulted to `true` when empty, so the env var wins).
+
+**The workflow `env:` does not reach the `build` job's compilation**, which happens inside
+`docker build` — the same three are set as `ENV` in the SDK stage of `docker/Dockerfile`. That is
+the job server-management#570 measured pegging 5.999/6 GiB, so it is the one that most needs this.
+Build-stage only; the final image is `FROM runtime-base`, so nothing lands in the shipped image.
+
+The `test` job ends with a **Report peak container memory** step (`always()`, never fails a build)
+that prints the cgroup's `memory.peak` high-water mark to the log and the job step summary. #604
+sizes the runners' caps on that number; before this it was inherited rather than measured. Read it
+off a recent run instead of re-deriving it by hand.
+
+### `services:` containers are capped explicitly (ersatztv#406)
+
+A runner's `container.options` (`--cpus=4 --memory=10g`) applies to the **job container only, not
+to `services:`**. Verified on a live `migrations` job: the job container reported
+`HostConfig.Memory=10737418240`; its `mysql:8.4` service reported `mem=0 nanocpus=0` — unbounded.
+So `migrations` runs added an uncapped MySQL to an already-tight host.
+
+The mysql service now sets `--memory=2g --cpus=2`, sized on measurement rather than inheritance: a
+`mysql:8.4` container with this exact env peaked at **543 MiB** during init and settled at 481 MiB
+idle (probed on bumblebee, 2026-07-17). Applying the migrations is DDL-heavy rather than
+data-heavy and the InnoDB buffer pool defaults to 128M, so 2g leaves ~4× headroom. **Any new
+`services:` container needs its own explicit cap** — it will not inherit one.
+
 ### `test` job
 
 `dotnet restore` → strip the Scanner project ref (`sed -i '/Scanner/d'`, matching the

docs/decisions.md

@@ -87,6 +87,7 @@ in-file entries.
 - [2026-07-17 — Shuffle-source construction extracted to `ShuffleSourceBuilder`; per-family seam, not a god-factory (#380)](#2026-07-17--shuffle-source-construction-extracted-to-shufflesourcebuilder-per-family-seam-not-a-god-factory-380)
 - [2026-07-17 — Seasonal / date-conditional scheduling already exists (alternate schedules / playout templates); #73 closed as implemented](#2026-07-17--seasonal--date-conditional-scheduling-already-exists-alternate-schedules--playout-templates-73-closed-as-implemented)
 - [2026-07-17 — Auto-Tune DetailPanel member list = live search-index roll-up, not EF enumeration (#384)](#2026-07-17--auto-tune-detailpanel-member-list--live-search-index-roll-up-not-ef-enumeration-384)
+- [2026-07-17 — No persistent compiler servers in CI; every `services:` container gets an explicit cap; #390's small-lane move reversed (#406)](#2026-07-17--no-persistent-compiler-servers-in-ci-every-services-container-gets-an-explicit-cap-390s-small-lane-move-reversed-406)
 
 ---
 
@@ -1225,3 +1226,67 @@ the existing `PagedLibraryBrowseItemsResponseModel` / `LibraryBrowseItemResponse
 the cap — the same staleness bound the smart-collection path accepts. Read-only, catalog-read tier (no
 `[RequiresAuthentication]`), so a cold review sufficed. Sibling backend child #385 (write-path overrides +
 weights) and SPA child #386 remain open under the #383 milestone.
+
+---
+
+## 2026-07-17 — No persistent compiler servers in CI; every `services:` container gets an explicit cap; #390's small-lane move reversed (#406)
+
+Three CI changes, all downstream of one incident: on 2026-07-17 bumblebee (the **prod media** Docker
+host, 25 GiB / 12 cores) hit load **340** with **21 GiB swapped** and ~238 MiB free, taking prod
+ersatztv/jellyfin down until reboot. Prod ersatztv itself was healthy at 168 MiB throughout — the
+thrash was **CI-induced**, triggered by a burst of parallel merges to `main`. Infra sizing is
+server-management#604's boundary; these three levers live in this repo and each does more than any
+capacity knob.
+
+**1. No persistent compiler servers.** `VBCSCompiler` is a *persistent* Roslyn server: it outlives
+the `dotnet build` that started it and holds its heap for the next one. Measured at **7.8 GB RSS**
+live on bumblebee — the single largest consumer on the box, and the actual reason each job needed a
+10 GiB cap. In CI it buys **nothing**: each job container is torn down at the end of the run, so
+there is never a "next build" to warm. The workflow's top-level `env:` now sets
+`UseSharedCompilation=false`, `DOTNET_CLI_USE_MSBUILD_SERVER=0`, `MSBUILDDISABLENODEREUSE=1` —
+MSBuild properties set as env vars so they apply to every `dotnet` call without touching each call
+site (MSBuild surfaces env vars as properties; `UseSharedCompilation` only defaults to `true` when
+empty, so the env var wins). Verified locally: a default build leaves 1 `VBCSCompiler` alive, the
+same build under these vars leaves **0**, and `ErsatzTV.sln` still builds clean (0 errors).
+
+**Why the Dockerfile also sets them** — and this is the part the issue's suggestion would have
+missed: the workflow `env:` reaches the *runner-side* dotnet jobs only. The `build` job compiles
+inside `docker build`, where it does not propagate, so the SDK stage of `docker/Dockerfile` sets the
+same three as `ENV`. That is precisely the job server-management#570 measured pegging **5.999/6
+GiB** — the one that most needs it. Build-stage only; the final image is `FROM runtime-base`, so
+nothing lands in the shipped image or affects runtime.
+
+**Trade-off accepted**: without the shared server each project's `csc` is a fresh process, which
+costs some build time. Worth it — the memory spike is what takes prod down, and the cap sizing that
+spike forces is what starves the lanes.
+
+**2. `services:` containers do not inherit the runner's cap.** A runner's `container.options`
+(`--cpus=4 --memory=10g`) applies to the **job container only**. Verified by inspecting a live
+`migrations` job: the job container reported `HostConfig.Memory=10737418240`, its `mysql:8.4`
+service reported `mem=0 nanocpus=0` — **unbounded**. Every migrations run was adding an uncapped
+MySQL to an already-tight host. Now `--memory=2g --cpus=2`, sized on **measurement, not
+inheritance**: a `mysql:8.4` container with this exact env peaked at **543 MiB** during init and
+settled at 481 MiB idle (probed on bumblebee). The migrations workload is DDL-heavy rather than
+data-heavy and the InnoDB buffer pool defaults to 128M, so 2g is ~4× headroom. **Standing rule: any
+new `services:` container needs its own explicit cap** — it will not inherit one.
+
+**3. #390's `small`-lane move for `api-docs`/`format` reversed.** #390 moved them to dodge a ~29 min
+`ubuntu-latest` queue. The queue was real, but the lane was the wrong fix, and **#390's own comment
+flagged why**: *"on an API-touching PR this job does a full `dotnet build`, so it is not always a
+'small' job; capacity 4 absorbs that."* "Capacity 4 absorbs that" held only because **nothing
+enforces the sum** of the lanes' caps — 6 slots × 10 GiB on a 25 GiB host is a 60 GiB promise. These
+are not small jobs: a live `docker stats` caught the `format` job container at **3.95 GiB**, which
+#604's re-sized 2 GiB `small` lane would OOM-kill outright. #604 grows `ubuntu-latest` to 5 slots
+(48 GiB ci-runner at capacity 4 + a bumblebee overflow slot) and fixes the queue at the source. This
+one is **order-coupled with #604**: the small lane's caps can't tighten until it lands.
+
+**Measurement is now continuous, not a one-off.** The `test` job ends with a **Report peak container
+memory** step reading the cgroup's `memory.peak` high-water mark (`always()`, tolerates absence,
+never fails a build). #604 sizes both runners' caps on that number, and until now it was *inherited*
+rather than measured — the 10g cap traces back to #570 observing a different job entirely. The
+lesson generalizes: this repo's CI perf issues keep stating numbers from plausibility rather than
+measurement (see #390's "2–4min" apt-ffmpeg estimate; real 110s, and not load-bearing). Read the
+peak off a recent run instead of re-deriving it by hand.
+
+**Not addressed here**: the 12–35 min queue waits (server-management#604) and the redundant
+triple-build (#398).