CI: peak-memory instrument reports cache-inflated memory.peak; sample true peak anon instead #412

Closed
opened 2026-07-17 15:51:09 +02:00 by timothy · 2 comments
Owner

Follow-up from #406 / PR #411, which added a Report peak container memory step to the test job so server-management#604 can size the runners' per-job caps on evidence instead of inheritance.

The step works, but the metric it reports is the wrong one to size a cap on, and the naive reading inverts the decision.

The problem

memory.peak (cgroup v2) is the high-water mark of memory.current, which charges page cache to the cgroup alongside anonymous memory. It is not "peak RSS".

Demonstrated on bumblebee — a container with zero anonymous memory that merely writes and reads an 800 MB file:

memory.peak    : 826 MiB
memory.current : 824 MiB
anon   0 MiB
file   800 MiB

For a build job doing heavy NuGet/npm/obj/bin/coverage I/O, cache plausibly dominates the number.

Why this inverts the decision: page cache is reclaimed under a tighter cap, not OOM-killed. So a large peak that is mostly file is not evidence the cap must stay high — but it reads exactly like it is. The first real measurement off PR #411 was:

peak container memory: 8305 MiB (8708448256 bytes, from /sys/fs/cgroup/memory.peak)

Read naively, "the test job peaks at 8.1 GiB" says the 10g cap can't come down and #406's whole premise ("if disabling shared compilation brings peak RSS well under 6 GiB, the whole budget loosens") is dead. But we don't actually know how much of that 8.1 GiB is reclaimable cache — quite possibly most of it.

Current mitigation (already in #411)

The step now also prints anon / file from memory.stat with a note to size caps on anon. That's a real improvement but still not right: the split is read at end-of-job, so it's the composition then, not at the peak instant. A job that peaks mid-dotnet test and then frees memory reports a misleadingly low anon.

What's actually wanted

A true peak-anon sample, i.e. the high-water mark of the anon component, which is what forces an OOM. Options:

  1. Background sampler: a &-d loop polling memory.stat's anon every ~2s, tracking the max, written to a file the final step reads. Cheap and exact enough. Main cost: a stray background process per job, and it must not outlive the job or fail it.
  2. memory.high probe: set a memory.high on a throwaway run and read memory.events/reclaim counters — measures the pressure point directly, but perturbs the run.
  3. Per-step deltas: sample around the known-heavy steps (Build, Test, Coverage summary) rather than continuously — coarser but no background process.
  4. Accept peak+split and just document the caveat loudly (status quo after #411).

Option 1 is probably the right cost/benefit; worth a skeptical look at whether a background process in an act_runner container is well-behaved.

Also worth measuring while here

  • A pre-change baseline. #411 has no before-number from this instrument (the 7.8 GB VBCSCompiler figure was measured host-wide across concurrent jobs, not inside one job container). So we cannot currently state what disabling the compiler servers actually saved per job — only that no persistent server survives the build. A one-off run with UseSharedCompilation unset would give the delta and either confirm or kill #406's "the budget loosens" premise.
  • The build job (in-Docker publish) is the job #570's 6 GiB cap actually derives from, and it has no instrument at all. It runs under BuildKit, so memory.peak isn't reachable the same way.

Done-when

  • The test job reports a peak figure that reflects non-reclaimable memory (peak anon), not a cache-inflated total
  • Documented in docs/ci-cd.md with the cache caveat, replacing #411's interim wording
  • A pre/post-change delta for the compiler-server switches measured at least once, and #406's "budget loosens" premise explicitly confirmed or killed on server-management#604
  • Adversarial review passed

Refs #406, PR #411, server-management#604, server-management#570.

Follow-up from #406 / PR #411, which added a **Report peak container memory** step to the `test` job so server-management#604 can size the runners' per-job caps on evidence instead of inheritance. The step works, but the metric it reports is the wrong one to size a cap on, and the naive reading **inverts the decision**. ## The problem `memory.peak` (cgroup v2) is the high-water mark of `memory.current`, which charges **page cache** to the cgroup alongside anonymous memory. It is not "peak RSS". Demonstrated on bumblebee — a container with **zero** anonymous memory that merely writes and reads an 800 MB file: ``` memory.peak : 826 MiB memory.current : 824 MiB anon 0 MiB file 800 MiB ``` For a build job doing heavy NuGet/npm/obj/bin/coverage I/O, cache plausibly dominates the number. **Why this inverts the decision:** page cache is *reclaimed* under a tighter cap, not OOM-killed. So a large peak that is mostly `file` is **not** evidence the cap must stay high — but it reads exactly like it is. The first real measurement off PR #411 was: ``` peak container memory: 8305 MiB (8708448256 bytes, from /sys/fs/cgroup/memory.peak) ``` Read naively, "the `test` job peaks at 8.1 GiB" says the 10g cap can't come down and #406's whole premise (*"if disabling shared compilation brings peak RSS well under 6 GiB, the whole budget loosens"*) is dead. But we don't actually know how much of that 8.1 GiB is reclaimable cache — quite possibly most of it. ## Current mitigation (already in #411) The step now also prints `anon` / `file` from `memory.stat` with a note to size caps on `anon`. That's a real improvement but **still not right**: the split is read at *end-of-job*, so it's the composition *then*, not at the peak instant. A job that peaks mid-`dotnet test` and then frees memory reports a misleadingly low `anon`. ## What's actually wanted A **true peak-anon** sample, i.e. the high-water mark of the anon component, which is what forces an OOM. Options: 1. **Background sampler**: a `&`-d loop polling `memory.stat`'s `anon` every ~2s, tracking the max, written to a file the final step reads. Cheap and exact enough. Main cost: a stray background process per job, and it must not outlive the job or fail it. 2. **`memory.high` probe**: set a `memory.high` on a throwaway run and read `memory.events`/reclaim counters — measures the pressure point directly, but perturbs the run. 3. **Per-step deltas**: sample around the known-heavy steps (`Build`, `Test`, `Coverage summary`) rather than continuously — coarser but no background process. 4. **Accept peak+split** and just document the caveat loudly (status quo after #411). Option 1 is probably the right cost/benefit; worth a skeptical look at whether a background process in an act_runner container is well-behaved. ## Also worth measuring while here - **A pre-change baseline.** #411 has no before-number from this instrument (the 7.8 GB `VBCSCompiler` figure was measured host-wide across concurrent jobs, not inside one job container). So we cannot currently state what disabling the compiler servers actually saved *per job* — only that no persistent server survives the build. A one-off run with `UseSharedCompilation` unset would give the delta and either confirm or kill #406's "the budget loosens" premise. - **The `build` job** (in-Docker publish) is the job #570's 6 GiB cap actually derives from, and it has no instrument at all. It runs under BuildKit, so `memory.peak` isn't reachable the same way. ## Done-when - [x] The `test` job reports a peak figure that reflects non-reclaimable memory (peak anon), not a cache-inflated total - [x] Documented in `docs/ci-cd.md` with the cache caveat, replacing #411's interim wording - [x] A pre/post-change delta for the compiler-server switches measured at least once, and #406's "budget loosens" premise explicitly confirmed or killed on server-management#604 - [x] Adversarial review passed Refs #406, PR #411, server-management#604, server-management#570.
timothy added the ci-cdpriority: medium labels 2026-07-17 15:51:38 +02:00
timothy added the in-progress label 2026-07-19 20:05:20 +02:00
Author
Owner

Claiming (Claude Code orchestrator session). Selected as the lowest-numbered eligible priority: medium backlog issue that is actually workable now.

Note on the queue: #350 mechanically ranked above this, but its remaining optimization work is gated on prod cold-start measurements from the #446 instrumentation — and the running :prod container is still on the 2026-07-16 image (pre-#446), so the v26.10.0 :prod image was pushed but the Komodo stack was never redeployed and zero cold-start log lines exist. That is an operator deploy gate, not code work; flagging separately.

Plan for #412: implement Option 1 (background peak-anon sampler) with a skeptical look at background-process hygiene in the act_runner container, measure the compiler-server pre/post delta at least once (swap disabled, session-unique container prefixes, --rm/trap), update docs/ci-cd.md, and report the #406 premise verdict on server-management#604. Working in a worktree off origin/main.

Claiming (Claude Code orchestrator session). Selected as the lowest-numbered eligible `priority: medium` backlog issue that is actually workable now. Note on the queue: #350 mechanically ranked above this, but its remaining optimization work is gated on prod cold-start measurements from the #446 instrumentation — and the running `:prod` container is still on the 2026-07-16 image (pre-#446), so the v26.10.0 `:prod` image was pushed but the Komodo stack was never redeployed and zero cold-start log lines exist. That is an operator deploy gate, not code work; flagging separately. Plan for #412: implement Option 1 (background peak-anon sampler) with a skeptical look at background-process hygiene in the act_runner container, measure the compiler-server pre/post delta at least once (swap disabled, session-unique container prefixes, `--rm`/trap), update docs/ci-cd.md, and report the #406 premise verdict on server-management#604. Working in a worktree off origin/main.
Author
Owner

Done — peak-anon sampler shipped (PR #475, merged)

What was done. The test job now reports a sampled true peak-anon as its headline memory figure, via new scripts/ci-peak-anon.sh — a start step (before Build/Test/Coverage) launches a detached background sampler of the cgroup anon high-water mark; a report step (last) stops it and prints peak-anon, keeping memory.peak + the end-of-job anon/file split as a cache-inflated ceiling and a reference. Both steps are continue-on-error + fail-open, so the instrument can never redden a green build.

Root cause of the wrong metric. memory.peak is the high-water mark of memory.current, which charges reclaimable page cache to the cgroup alongside anon. A build does heavy NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak — and cache is reclaimed under a tighter cap, not OOM-killed. Sizing server-management#604's per-job caps off memory.peak therefore inverts the decision. The kernel has no peak-anon counter and the end-of-job snapshot misses the peak instant (a job peaks mid-dotnet test then frees), so peak-anon must be sampled.

Validated on bumblebee (CI image, swap-off): the detached sampler survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on kill (TERM trap); fails open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean. Cold independent review passed (no confirmed bugs); one flagged sub-microsecond TOCTOU nit closed via atomic write (temp + mv).

Compiler-server A/B (Done-when #3). Swap-off, sampled peak-anon, n=2 interleaved:

  • OFF (the CI config): 5818 / 5854 MiB (~5.84 GiB, tight)
  • ON (dotnet defaults): 6305 / 7604 MiB (~7.0 GiB, noisy) + a ~3 GiB resident VBCSCompiler

Disabling the compiler servers is worth it (consistent reduction + no resident server), but OFF sits right at 6 GiB for the build phase alone and the test job adds test+coverage on top — so #406's premise ("disabling brings peak well under 6 GiB → the budget loosens") is NOT supported. Size the cap off the live test-job peak-anon this instrument now reports (build+test+coverage), not off the build-only A/B. The older #411 probe (anon 7134 MiB) read higher than these swap-off sampled numbers (swap/read-method move it >1 GiB) and is superseded.

Files. scripts/ci-peak-anon.sh (new); .gitea/workflows/docker-build.yml (two test-job steps); docs/ci-cd.md (instrument section rewritten — peak-anon headline + A/B table + premise verdict); docs/decisions.md (entry added). No .cs touched.

Deferred / not in scope. The build job (in-Docker publish, BuildKit) still has no instrument — memory.peak isn't reachable there the same way; left as a note in the issue, not a Done-when box. The definitive cap-sizing number for #604 is now the live test-job peak-anon on the next main CI run.

## Done — peak-anon sampler shipped (PR #475, merged) **What was done.** The `test` job now reports a **sampled true peak-anon** as its headline memory figure, via new `scripts/ci-peak-anon.sh` — a `start` step (before Build/Test/Coverage) launches a detached background sampler of the cgroup `anon` high-water mark; a `report` step (last) stops it and prints peak-anon, keeping `memory.peak` + the end-of-job `anon`/`file` split as a cache-inflated ceiling and a reference. Both steps are `continue-on-error` + fail-open, so the instrument can never redden a green build. **Root cause of the wrong metric.** `memory.peak` is the high-water mark of `memory.current`, which charges reclaimable **page cache** to the cgroup alongside anon. A build does heavy NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak — and cache is reclaimed under a tighter cap, not OOM-killed. Sizing server-management#604's per-job caps off `memory.peak` therefore inverts the decision. The kernel has no peak-anon counter and the end-of-job snapshot misses the peak instant (a job peaks mid-`dotnet test` then frees), so peak-anon must be sampled. **Validated on bumblebee** (CI image, swap-off): the detached sampler survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on `kill` (TERM trap); fails open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean. Cold independent review passed (no confirmed bugs); one flagged sub-microsecond TOCTOU nit closed via atomic write (temp + `mv`). **Compiler-server A/B (Done-when #3).** Swap-off, sampled peak-anon, n=2 interleaved: - **OFF** (the CI config): **5818 / 5854 MiB** (~5.84 GiB, tight) - **ON** (dotnet defaults): **6305 / 7604 MiB** (~7.0 GiB, noisy) + a **~3 GiB resident `VBCSCompiler`** Disabling the compiler servers is worth it (consistent reduction + no resident server), but OFF sits *right at 6 GiB for the build phase alone* and the `test` job adds test+coverage on top — so **#406's premise ("disabling brings peak well under 6 GiB → the budget loosens") is NOT supported.** Size the cap off the live test-job peak-anon this instrument now reports (build+test+coverage), not off the build-only A/B. The older #411 probe (`anon 7134 MiB`) read higher than these swap-off sampled numbers (swap/read-method move it >1 GiB) and is superseded. **Files.** `scripts/ci-peak-anon.sh` (new); `.gitea/workflows/docker-build.yml` (two `test`-job steps); `docs/ci-cd.md` (instrument section rewritten — peak-anon headline + A/B table + premise verdict); `docs/decisions.md` (entry added). No `.cs` touched. **Deferred / not in scope.** The `build` job (in-Docker publish, BuildKit) still has no instrument — `memory.peak` isn't reachable there the same way; left as a note in the issue, not a Done-when box. The definitive cap-sizing number for #604 is now the live test-job peak-anon on the next `main` CI run.
timothy removed the in-progress label 2026-07-19 21:10:01 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#412