---
key: ci.peak-anon-measurement
title: 2026-07-19 — CI `test` job reports a sampled true peak-anon, not cache-inflated `memory.peak` (#412)
status: active
since: '2026-07-19'
supersedes: none
superseded-by: none
stale-after: '2027-03-15'
rule: The `test` job's headline memory figure is a sampled high-water mark of cgroup `anon`, produced by `scripts/ci-peak-anon.sh`; `memory.peak` and the end-of-job `anon`/`file` split are kept only as a cache-inflated reference.
signals: 'CI memory measurement · paths: `scripts/ci-peak-anon.sh`, `.gitea/workflows/*.yml` · issues: #412, #411 (prose-only predecessor, no standalone record — its `memory.peak`-headline approach is superseded by this record)'
mechanics: '`scripts/ci-peak-anon.sh` header; `docs/ci-cd.md` → CI build memory'
sources: '`scripts/ci-peak-anon.sh` (the sampler itself) · cgroup v2 `memory.peak` vs `memory.stat` `anon` accounting, measured on the bumblebee runners #412'
---

**Decision.** The `test` job's memory instrument (added in #411) now reports a **sampled high-water
mark of the cgroup's `anon` memory** as the headline figure, produced by `scripts/ci-peak-anon.sh`
(a `start` step before the dotnet Build/Test/Coverage, a `report` step last). `memory.peak` and the
end-of-job `anon`/`file` split stay in the output as a cache-inflated ceiling and a reference.

**Why not just `memory.peak`.** `memory.peak` is the high-water mark of `memory.current`, which
charges reclaimable **page cache** to the cgroup alongside anon. A build does heavy
NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak — and page cache is *reclaimed* under
a tighter cap, not OOM-killed. Sizing a per-job cap (server-management#604) off `memory.peak`
therefore **inverts the decision**: a big, mostly-`file` peak reads like "the cap must stay high"
when it isn't. The OOM-forcing quantity is peak **anon**. The kernel exposes `memory.peak` but has
**no peak-anon counter**, and the end-of-job `anon` is the composition *then*, not at the peak
instant (a job that peaks mid-`dotnet test` then frees reports a misleadingly low `anon`) — so it
must be **sampled**. Details + the sampler's robustness rationale: `scripts/ci-peak-anon.sh` header
and `docs/ci-cd.md` → "CI build memory".

**Implementation note (do not "simplify" back to `memory.peak`).** The sampler is a detached
`nohup` poller that survives step-boundary re-execs (reparents to the container's PID 1) and is
reaped at container teardown; a TERM trap + `sleep & wait` stops it at once on `report`. Both steps
are `continue-on-error` with a fail-open script, so the instrument can never redden a green build.
Validated on bumblebee: it catches a transient 2.5 GiB anon spike that the end-of-job snapshot
reports as 0.

**Compiler-server A/B verdict (refines the #406 entry's "premise looking dead" read).** Measured
in the CI image, swap-off, sampled peak-anon, n=2 interleaved: **OFF (the CI config) ≈ 5.84 GiB,
consistent; ON (defaults) 6.3–7.6 GiB, always higher, + a ~3 GiB resident `VBCSCompiler`.**
Disabling the servers is worth it (consistent reduction, no resident server), but OFF sits *right at
6 GiB for the build phase alone* and the `test` job adds test + coverage on top — so #406's premise
("disabling brings peak *well under* 6 GiB → the budget loosens") is **not supported**. Size the cap
off the live test-job peak-anon this instrument now reports, not off the build-only A/B. The older
#411 probe (`anon 7134 MiB`) read higher than these swap-off sampled numbers and is superseded
(swap/read-method move the figure >1 GiB).
