ci(412): sample true peak-anon in the test job, not cache-inflated memory.peak #475

Merged
timothy merged 1 commits from ci/412-peak-anon-sampler into main 2026-07-19 21:09:30 +02:00
Owner

Fixes #412.

What

The test job now reports a sampled true peak-anon as its headline memory figure, via scripts/ci-peak-anon.sh (a start step before Build/Test/Coverage launches a detached background sampler of the cgroup anon high-water mark; a report step last stops it and prints peak-anon, keeping memory.peak + end-of-job split as a cache-inflated ceiling + reference). Both continue-on-error + fail-open — never reddens a build.

Why

memory.peak charges reclaimable page cache to the cgroup; a build does heavy I/O so cache can dominate. Page cache is reclaimed under a tighter cap, not OOM-killed, so sizing a per-job cap (server-management#604) off memory.peak inverts the decision. The OOM-forcing quantity is peak anon — which the kernel exposes no counter for, and which the end-of-job snapshot misses (peaks mid-dotnet test then frees).

Validated on bumblebee (CI image)

Survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on kill (TERM trap); fail-open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean.

Compiler-server A/B (Done-when item 3)

Swap-off, sampled peak-anon, n=2 interleaved: OFF (CI config) ~5.84 GiB consistent; ON (defaults) 6.3–7.6 GiB, always higher, + ~3 GiB resident VBCSCompiler. Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the test job adds test+coverage — so #406’s “budget loosens well under 6 GiB” premise is not supported. Size the cap off the live test-job sampler.

Docs

ci-cd.md instrument section rewritten (peak-anon headline + A/B table + premise verdict); decisions.md entry added. No .cs touched.

Review

Cold independent review passed — no confirmed bugs; the one flagged sub-µs TOCTOU nit (report reading $OUT mid-truncate) closed via atomic write (temp + mv).

Fixes #412. ## What The `test` job now reports a **sampled true peak-anon** as its headline memory figure, via `scripts/ci-peak-anon.sh` (a `start` step before Build/Test/Coverage launches a detached background sampler of the cgroup `anon` high-water mark; a `report` step last stops it and prints peak-anon, keeping `memory.peak` + end-of-job split as a cache-inflated ceiling + reference). Both `continue-on-error` + fail-open — never reddens a build. ## Why `memory.peak` charges reclaimable page cache to the cgroup; a build does heavy I/O so cache can dominate. Page cache is reclaimed under a tighter cap, not OOM-killed, so sizing a per-job cap (server-management#604) off `memory.peak` **inverts the decision**. The OOM-forcing quantity is peak anon — which the kernel exposes no counter for, and which the end-of-job snapshot misses (peaks mid-`dotnet test` then frees). ## Validated on bumblebee (CI image) Survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on `kill` (TERM trap); fail-open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean. ## Compiler-server A/B (Done-when item 3) Swap-off, sampled peak-anon, n=2 interleaved: **OFF (CI config) ~5.84 GiB consistent; ON (defaults) 6.3–7.6 GiB, always higher, + ~3 GiB resident VBCSCompiler.** Disabling the servers is worth it, but OFF sits *right at 6 GiB for the build phase alone* and the test job adds test+coverage — so #406’s “budget loosens well under 6 GiB” premise is **not supported**. Size the cap off the live test-job sampler. ## Docs `ci-cd.md` instrument section rewritten (peak-anon headline + A/B table + premise verdict); `decisions.md` entry added. No `.cs` touched. ## Review Cold independent review passed — no confirmed bugs; the one flagged sub-µs TOCTOU nit (report reading $OUT mid-truncate) closed via atomic write (temp + `mv`).
timothy added 1 commit 2026-07-19 20:49:38 +02:00
ci(412): sample true peak-anon in the test job, not cache-inflated memory.peak
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 7s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 13s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m15s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 1m10s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 7m12s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 13m47s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 19m12s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
c5369b1d69
The test-job memory instrument (#411) reported memory.peak — the high-water mark of
memory.current, which charges reclaimable page cache to the cgroup. A build does heavy
NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak, and page cache is reclaimed
under a tighter cap rather than OOM-killed. Sizing a per-job cap (server-management#604) off
memory.peak therefore inverts the decision. The OOM-forcing quantity is peak anon, which the
kernel exposes no counter for and which the end-of-job split misses (a job that peaks
mid-dotnet-test then frees reports a low anon).

New scripts/ci-peak-anon.sh: a `start` step (before Build/Test/Coverage) launches a detached
background sampler tracking the high-water mark of cgroup anon; a `report` step (last) stops it
and prints the sampled peak anon as the headline, keeping memory.peak + end-of-job split as a
cache-inflated ceiling and reference. Both continue-on-error + fail-open so they never redden a
build. Validated on bumblebee: survives step-boundary re-execs, catches a transient 2.5 GiB
anon spike the snapshot reports as 0, stops cleanly on kill, degrades gracefully.

Compiler-server A/B (swap-off, sampled peak-anon, n=2 interleaved): OFF (CI config) ~5.84 GiB
consistent; ON (defaults) 6.3-7.6 GiB, always higher, + a ~3 GiB resident VBCSCompiler.
Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the
test job adds test+coverage, so #406's "budget loosens well under 6 GiB" premise is not
supported. Size the cap off the live test-job sampler.

Docs: ci-cd.md instrument section rewritten (peak-anon headline + A/B table + premise verdict);
decisions.md entry added. No .cs touched.

fixes #412
Author
Owner

Cold independent review (review-only, cold context) passed: no confirmed correctness bugs across start/report/sample_loop, the fail-open guarantees, and the if:/always()/continue-on-error wiring. One earlier-flagged sub-microsecond TOCTOU nit (report reading $OUT mid-truncate) was closed at head via an atomic write (temp + mv); a residual ~40-confidence PID-reuse note on kill is bounded to a stray TERM and cannot redden the build, left as-is.

Review-verdict: MERGEABLE @ c5369b1d69

Cold independent review (review-only, cold context) passed: **no confirmed correctness bugs** across `start`/`report`/`sample_loop`, the fail-open guarantees, and the `if:`/`always()`/`continue-on-error` wiring. One earlier-flagged sub-microsecond TOCTOU nit (report reading `$OUT` mid-truncate) was closed at head via an atomic write (temp + `mv`); a residual ~40-confidence PID-reuse note on `kill` is bounded to a stray TERM and cannot redden the build, left as-is. Review-verdict: MERGEABLE @ c5369b1d695843a6b0ef36fb734d7b7270c78fbd
timothy merged commit 48d7598bae into main 2026-07-19 21:09:30 +02:00
timothy deleted branch ci/412-peak-anon-sampler 2026-07-19 21:09:30 +02:00
Sign in to join this conversation.