ci(406): disable persistent compiler servers, cap the services: mysql, report peak RSS
#411
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
469d725559 |
ci(406): apply the memory-swap rule to our own two sites; stop leaning on the peak reading [decisions-edit]
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m15s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 17s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 15s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 15m20s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 8m13s
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 6s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 5s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 4s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Re-review of the fix commits came back MERGEABLE with three findings worth acting on.
MEDIUM -- the PR documented a standing rule ("--memory without --memory-swap silently
grants 2x in swap") and then didn't apply it to the two sites this repo owns:
docker-build.yml's smoke container and scripts/migration-smoke.sh, both `--memory 2g` with
no --memory-swap. Pre-existing rather than a regression, but a rule you don't follow in
your own repo isn't a rule. The migration-smoke one matters most: it runs on the PROD host
in the release path, so a runaway migration should die against its cap rather than quietly
swap out the box serving media.
LOW -- and this is the important one: the docs leaned "peak 8305 MiB is probably mostly
reclaimable cache". An independent probe (full solution build, same CI image, shared
compilation off) measured peak 9457 MiB / anon 7134 MiB / file 421 MiB. ANON DOMINATED.
Having verified the *mechanism* (peak overstates because it counts page cache), I guessed
the *magnitude* in the direction I preferred -- the exact failure this entry criticises,
committed inside the entry criticising it. Corrected in ci-cd.md, decisions.md and on
server-management#604 (where the previous comment could have led to an unsafe 6g cap).
Consequences now recorded honestly: a 6g cap looks UNSAFE, #570's "6g proved too tight" is
the rule not an outlier, and #406's premise ("if this brings peak RSS well under 6 GiB the
whole budget loosens") is looking DEAD -- the 7134 MiB anon was measured with shared
compilation already off. The switches remain right; the looser budget they were meant to
buy does not follow.
NIT -- dropped the unverified claim that this also disables the Razor build server. The
UseRazorBuildServer -> UseSharedCompilation fallback is .NET 5-era; Razor has been an
in-process source generator since .NET 6, so there is likely no separate server to disable
on .NET 10. Unverified, zero impact, so it has no business in a doc arguing for
measurement over assumption.
[decisions-edit]: the touched docs/decisions.md lines were added by this PR's own earlier
commits, not settled entries on main -- net vs origin/main remains a pure insertion (0
deletions, verified). Also the sanctioned reason: the entry was factually wrong (see LOW).
Verified: both workflows parse; migration-smoke.sh passes bash -n; the parsed mysql option
string is `--memory=2g --memory-swap=2g --cpus=2`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
7859510c78 |
ci(406): report anon/file split — memory.peak counts page cache, not just RSS [decisions-edit]
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 6s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 7s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 7s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 15s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 13s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m35s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 13m30s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 19m1s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
The peak-memory step from the previous commit worked on its first real CI run and immediately produced a number that would have led #604 to the WRONG decision: peak container memory: 8305 MiB (8708448256 bytes, from /sys/fs/cgroup/memory.peak) Read naively that says "the test job needs 8.1 GiB, so the 10g cap must stay and #406's 'the budget loosens' premise is dead". That reading is probably wrong, because `memory.peak` is the high-water mark of `memory.current`, which charges PAGE CACHE to the cgroup as well as anonymous memory. It is not peak RSS, and a build job does heavy NuGet/npm/obj/bin/coverage I/O. Proven on bumblebee -- a container with ZERO anonymous memory that merely reads an 800MB file: memory.peak : 826 MiB anon : 0 MiB file : 800 MiB This inverts the decision: page cache is RECLAIMED under a tighter cap, not OOM-killed, so a large peak that is mostly `file` is NOT evidence that the cap must stay high. `anon` is what forces an OOM. So the step now prints the anon/file split and says, in the log and the docs, to size caps on anon rather than peak. Honest about what this still isn't: the split is read at end-of-job, so it is the composition then, not at the peak instant -- a job peaking mid-`dotnet test` and freeing after would under-report anon. Sampling a true peak-anon is ersatztv#412, filed with the options. Also recorded there and in ci-cd.md: there is NO pre-change baseline from this instrument (7.8 GB VBCSCompiler was measured host-wide across concurrent jobs, not inside one job container), so #406's premise is neither confirmed nor killed yet. What IS established: no persistent compiler server survives a build, and migrations is green with mysql capped at 2g with swap disabled. The irony is the point, and it is now written into decisions.md: this repo's CI perf work keeps stating numbers from plausibility rather than measurement (#390's "2-4min" apt-ffmpeg estimate; real 110s). Measuring the wrong quantity precisely is the same failure wearing a lab coat -- and it bit this very change, whose whole purpose was to replace an inherited number with a measured one. Verified: shellcheck clean; the real step body run inside a --memory=2g container under `bash -e -o pipefail` reads memory.peak, prints the split, writes the step summary, exits 0; and it degrades to a skip + exit 0 where no cgroup file exists. [decisions-edit]: the modified docs/decisions.md lines are ones THIS PR added in its own earlier commits, not a settled entry on main -- net vs origin/main is still a pure insertion (0 deletions, verified). It is also the sanctioned reason: the entry as written was factually wrong about what memory.peak measures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
04e3a5249d |
ci(406): disable swap on the capped mysql; fix peak-step placement and failure mode [decisions-edit]
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 6s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 5s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 6s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 16s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 14s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m6s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 14m42s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 13m43s
Review fixes from the cold adversarial pass on PR #411. Verdict was MERGEABLE, but one
HIGH is a genuine defect worth the round-trip.
[decisions-edit] rationale: the 12 "modified" lines in docs/decisions.md are lines THIS PR
added in its own previous commit (
|
||
|
|
e33205e8eb |
ci(406): disable persistent compiler servers, cap the mysql service, report peak RSS
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 9s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m22s
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 6s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 6s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 5s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 14m9s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 18m23s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 18m45s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Refs #406, server-management#604. Items 1 and 2 of #406 (item 3 landed in PR #407). Root cause of the 2026-07-17 bumblebee incident (load 340, 21 GiB swapped, prod media down until reboot) is CI-induced memory pressure, and the largest single consumer was Roslyn's persistent VBCSCompiler at 7.8 GB RSS. It is a compiler *server*: it outlives the build that started it to keep its heap warm for the next one. In CI there is no next build -- each job container is torn down at the end of the run -- so it is pure cost, and it is the actual reason each job needed a 10 GiB cap. 1. Disable the compiler servers via the workflow's top-level env (UseSharedCompilation, DOTNET_CLI_USE_MSBUILD_SERVER, MSBUILDDISABLENODEREUSE). MSBuild surfaces env vars as properties and only defaults UseSharedCompilation when empty, so the env var wins and every dotnet call is covered without touching each call site. Also set as ENV in the Dockerfile's SDK stage: the workflow env reaches runner-side jobs only, and the `build` job compiles inside `docker build` where it does not propagate -- that is the job server-management#570 measured pegging 5.999/6 GiB, so it is the one that most needs this. Build stage only; the final image is FROM runtime-base, so nothing lands in the shipped image. Verified locally: a default build leaves 1 VBCSCompiler alive, the same build under these vars leaves 0, and ErsatzTV.sln still builds clean (0 errors). Trade-off: csc runs per project instead of via the server, costing some build time. 2. Cap the mysql service (--memory=2g --cpus=2). A runner's container.options applies to the job container ONLY: a live migrations job reported HostConfig.Memory=10737418240 for the job and mem=0 nanocpus=0 for its mysql service, i.e. unbounded. 2g is sized on measurement -- mysql:8.4 with this exact env peaked at 543 MiB during init, 481 MiB idle (probed on bumblebee) -- not on inheritance. Also add a `Report peak container memory` step to the test job (cgroup memory.peak, always(), never fails a build). server-management#604 sizes both runners' caps on that number and it was inherited rather than measured; now every run reports it. Docs: ci-cd.md gains the compiler-server + services-cap sections and the lane table is corrected (PR #407 reverted the #390 lane move without updating it). decisions.md records all three, incl. the standing rule that a services: container never inherits a cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |