The test job now reports a sampled true peak-anon as its headline memory figure, via scripts/ci-peak-anon.sh (a start step before Build/Test/Coverage launches a detached background sampler of the cgroup anon high-water mark; a report step last stops it and prints peak-anon, keeping memory.peak + end-of-job split as a cache-inflated ceiling + reference). Both continue-on-error + fail-open — never reddens a build.
Why
memory.peak charges reclaimable page cache to the cgroup; a build does heavy I/O so cache can dominate. Page cache is reclaimed under a tighter cap, not OOM-killed, so sizing a per-job cap (server-management#604) off memory.peakinverts the decision. The OOM-forcing quantity is peak anon — which the kernel exposes no counter for, and which the end-of-job snapshot misses (peaks mid-dotnet test then frees).
Validated on bumblebee (CI image)
Survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on kill (TERM trap); fail-open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean.
Compiler-server A/B (Done-when item 3)
Swap-off, sampled peak-anon, n=2 interleaved: OFF (CI config) ~5.84 GiB consistent; ON (defaults) 6.3–7.6 GiB, always higher, + ~3 GiB resident VBCSCompiler. Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the test job adds test+coverage — so #406’s “budget loosens well under 6 GiB” premise is not supported. Size the cap off the live test-job sampler.
Cold independent review passed — no confirmed bugs; the one flagged sub-µs TOCTOU nit (report reading $OUT mid-truncate) closed via atomic write (temp + mv).
Fixes #412.
## What
The `test` job now reports a **sampled true peak-anon** as its headline memory figure, via `scripts/ci-peak-anon.sh` (a `start` step before Build/Test/Coverage launches a detached background sampler of the cgroup `anon` high-water mark; a `report` step last stops it and prints peak-anon, keeping `memory.peak` + end-of-job split as a cache-inflated ceiling + reference). Both `continue-on-error` + fail-open — never reddens a build.
## Why
`memory.peak` charges reclaimable page cache to the cgroup; a build does heavy I/O so cache can dominate. Page cache is reclaimed under a tighter cap, not OOM-killed, so sizing a per-job cap (server-management#604) off `memory.peak` **inverts the decision**. The OOM-forcing quantity is peak anon — which the kernel exposes no counter for, and which the end-of-job snapshot misses (peaks mid-`dotnet test` then frees).
## Validated on bumblebee (CI image)
Survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on `kill` (TERM trap); fail-open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean.
## Compiler-server A/B (Done-when item 3)
Swap-off, sampled peak-anon, n=2 interleaved: **OFF (CI config) ~5.84 GiB consistent; ON (defaults) 6.3–7.6 GiB, always higher, + ~3 GiB resident VBCSCompiler.** Disabling the servers is worth it, but OFF sits *right at 6 GiB for the build phase alone* and the test job adds test+coverage — so #406’s “budget loosens well under 6 GiB” premise is **not supported**. Size the cap off the live test-job sampler.
## Docs
`ci-cd.md` instrument section rewritten (peak-anon headline + A/B table + premise verdict); `decisions.md` entry added. No `.cs` touched.
## Review
Cold independent review passed — no confirmed bugs; the one flagged sub-µs TOCTOU nit (report reading $OUT mid-truncate) closed via atomic write (temp + `mv`).
The test-job memory instrument (#411) reported memory.peak — the high-water mark of
memory.current, which charges reclaimable page cache to the cgroup. A build does heavy
NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak, and page cache is reclaimed
under a tighter cap rather than OOM-killed. Sizing a per-job cap (server-management#604) off
memory.peak therefore inverts the decision. The OOM-forcing quantity is peak anon, which the
kernel exposes no counter for and which the end-of-job split misses (a job that peaks
mid-dotnet-test then frees reports a low anon).
New scripts/ci-peak-anon.sh: a `start` step (before Build/Test/Coverage) launches a detached
background sampler tracking the high-water mark of cgroup anon; a `report` step (last) stops it
and prints the sampled peak anon as the headline, keeping memory.peak + end-of-job split as a
cache-inflated ceiling and reference. Both continue-on-error + fail-open so they never redden a
build. Validated on bumblebee: survives step-boundary re-execs, catches a transient 2.5 GiB
anon spike the snapshot reports as 0, stops cleanly on kill, degrades gracefully.
Compiler-server A/B (swap-off, sampled peak-anon, n=2 interleaved): OFF (CI config) ~5.84 GiB
consistent; ON (defaults) 6.3-7.6 GiB, always higher, + a ~3 GiB resident VBCSCompiler.
Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the
test job adds test+coverage, so #406's "budget loosens well under 6 GiB" premise is not
supported. Size the cap off the live test-job sampler.
Docs: ci-cd.md instrument section rewritten (peak-anon headline + A/B table + premise verdict);
decisions.md entry added. No .cs touched.
fixes#412
Cold independent review (review-only, cold context) passed: no confirmed correctness bugs across start/report/sample_loop, the fail-open guarantees, and the if:/always()/continue-on-error wiring. One earlier-flagged sub-microsecond TOCTOU nit (report reading $OUT mid-truncate) was closed at head via an atomic write (temp + mv); a residual ~40-confidence PID-reuse note on kill is bounded to a stray TERM and cannot redden the build, left as-is.
Cold independent review (review-only, cold context) passed: **no confirmed correctness bugs** across `start`/`report`/`sample_loop`, the fail-open guarantees, and the `if:`/`always()`/`continue-on-error` wiring. One earlier-flagged sub-microsecond TOCTOU nit (report reading `$OUT` mid-truncate) was closed at head via an atomic write (temp + `mv`); a residual ~40-confidence PID-reuse note on `kill` is bounded to a stray TERM and cannot redden the build, left as-is.
Review-verdict: MERGEABLE @ c5369b1d695843a6b0ef36fb734d7b7270c78fbd
timothy
merged commit 48d7598bae into main2026-07-19 21:09:30 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #412.
What
The
testjob now reports a sampled true peak-anon as its headline memory figure, viascripts/ci-peak-anon.sh(astartstep before Build/Test/Coverage launches a detached background sampler of the cgroupanonhigh-water mark; areportstep last stops it and prints peak-anon, keepingmemory.peak+ end-of-job split as a cache-inflated ceiling + reference). Bothcontinue-on-error+ fail-open — never reddens a build.Why
memory.peakcharges reclaimable page cache to the cgroup; a build does heavy I/O so cache can dominate. Page cache is reclaimed under a tighter cap, not OOM-killed, so sizing a per-job cap (server-management#604) offmemory.peakinverts the decision. The OOM-forcing quantity is peak anon — which the kernel exposes no counter for, and which the end-of-job snapshot misses (peaks mid-dotnet testthen frees).Validated on bumblebee (CI image)
Survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on
kill(TERM trap); fail-open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean.Compiler-server A/B (Done-when item 3)
Swap-off, sampled peak-anon, n=2 interleaved: OFF (CI config) ~5.84 GiB consistent; ON (defaults) 6.3–7.6 GiB, always higher, + ~3 GiB resident VBCSCompiler. Disabling the servers is worth it, but OFF sits right at 6 GiB for the build phase alone and the test job adds test+coverage — so #406’s “budget loosens well under 6 GiB” premise is not supported. Size the cap off the live test-job sampler.
Docs
ci-cd.mdinstrument section rewritten (peak-anon headline + A/B table + premise verdict);decisions.mdentry added. No.cstouched.Review
Cold independent review passed — no confirmed bugs; the one flagged sub-µs TOCTOU nit (report reading $OUT mid-truncate) closed via atomic write (temp +
mv).Cold independent review (review-only, cold context) passed: no confirmed correctness bugs across
start/report/sample_loop, the fail-open guarantees, and theif:/always()/continue-on-errorwiring. One earlier-flagged sub-microsecond TOCTOU nit (report reading$OUTmid-truncate) was closed at head via an atomic write (temp +mv); a residual ~40-confidence PID-reuse note onkillis bounded to a stray TERM and cannot redden the build, left as-is.Review-verdict: MERGEABLE @
c5369b1d69