CI: peak-memory instrument reports cache-inflated memory.peak; sample true peak anon instead
#412
Closed
opened 2026-07-17 15:51:09 +02:00 by timothy
·
2 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#412
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Follow-up from #406 / PR #411, which added a Report peak container memory step to the
testjob so server-management#604 can size the runners' per-job caps on evidence instead of inheritance.The step works, but the metric it reports is the wrong one to size a cap on, and the naive reading inverts the decision.
The problem
memory.peak(cgroup v2) is the high-water mark ofmemory.current, which charges page cache to the cgroup alongside anonymous memory. It is not "peak RSS".Demonstrated on bumblebee — a container with zero anonymous memory that merely writes and reads an 800 MB file:
For a build job doing heavy NuGet/npm/obj/bin/coverage I/O, cache plausibly dominates the number.
Why this inverts the decision: page cache is reclaimed under a tighter cap, not OOM-killed. So a large peak that is mostly
fileis not evidence the cap must stay high — but it reads exactly like it is. The first real measurement off PR #411 was:Read naively, "the
testjob peaks at 8.1 GiB" says the 10g cap can't come down and #406's whole premise ("if disabling shared compilation brings peak RSS well under 6 GiB, the whole budget loosens") is dead. But we don't actually know how much of that 8.1 GiB is reclaimable cache — quite possibly most of it.Current mitigation (already in #411)
The step now also prints
anon/filefrommemory.statwith a note to size caps onanon. That's a real improvement but still not right: the split is read at end-of-job, so it's the composition then, not at the peak instant. A job that peaks mid-dotnet testand then frees memory reports a misleadingly lowanon.What's actually wanted
A true peak-anon sample, i.e. the high-water mark of the anon component, which is what forces an OOM. Options:
&-d loop pollingmemory.stat'sanonevery ~2s, tracking the max, written to a file the final step reads. Cheap and exact enough. Main cost: a stray background process per job, and it must not outlive the job or fail it.memory.highprobe: set amemory.highon a throwaway run and readmemory.events/reclaim counters — measures the pressure point directly, but perturbs the run.Build,Test,Coverage summary) rather than continuously — coarser but no background process.Option 1 is probably the right cost/benefit; worth a skeptical look at whether a background process in an act_runner container is well-behaved.
Also worth measuring while here
VBCSCompilerfigure was measured host-wide across concurrent jobs, not inside one job container). So we cannot currently state what disabling the compiler servers actually saved per job — only that no persistent server survives the build. A one-off run withUseSharedCompilationunset would give the delta and either confirm or kill #406's "the budget loosens" premise.buildjob (in-Docker publish) is the job #570's 6 GiB cap actually derives from, and it has no instrument at all. It runs under BuildKit, somemory.peakisn't reachable the same way.Done-when
testjob reports a peak figure that reflects non-reclaimable memory (peak anon), not a cache-inflated totaldocs/ci-cd.mdwith the cache caveat, replacing #411's interim wordingRefs #406, PR #411, server-management#604, server-management#570.
Claiming (Claude Code orchestrator session). Selected as the lowest-numbered eligible
priority: mediumbacklog issue that is actually workable now.Note on the queue: #350 mechanically ranked above this, but its remaining optimization work is gated on prod cold-start measurements from the #446 instrumentation — and the running
:prodcontainer is still on the 2026-07-16 image (pre-#446), so the v26.10.0:prodimage was pushed but the Komodo stack was never redeployed and zero cold-start log lines exist. That is an operator deploy gate, not code work; flagging separately.Plan for #412: implement Option 1 (background peak-anon sampler) with a skeptical look at background-process hygiene in the act_runner container, measure the compiler-server pre/post delta at least once (swap disabled, session-unique container prefixes,
--rm/trap), update docs/ci-cd.md, and report the #406 premise verdict on server-management#604. Working in a worktree off origin/main.Done — peak-anon sampler shipped (PR #475, merged)
What was done. The
testjob now reports a sampled true peak-anon as its headline memory figure, via newscripts/ci-peak-anon.sh— astartstep (before Build/Test/Coverage) launches a detached background sampler of the cgroupanonhigh-water mark; areportstep (last) stops it and prints peak-anon, keepingmemory.peak+ the end-of-jobanon/filesplit as a cache-inflated ceiling and a reference. Both steps arecontinue-on-error+ fail-open, so the instrument can never redden a green build.Root cause of the wrong metric.
memory.peakis the high-water mark ofmemory.current, which charges reclaimable page cache to the cgroup alongside anon. A build does heavy NuGet/npm/obj/bin/coverage I/O, so cache can dominate the peak — and cache is reclaimed under a tighter cap, not OOM-killed. Sizing server-management#604's per-job caps offmemory.peaktherefore inverts the decision. The kernel has no peak-anon counter and the end-of-job snapshot misses the peak instant (a job peaks mid-dotnet testthen frees), so peak-anon must be sampled.Validated on bumblebee (CI image, swap-off): the detached sampler survives step-boundary re-execs (reparents to PID 1); catches a transient 2.5 GiB anon spike the snapshot reports as 0; stops cleanly on
kill(TERM trap); fails open when cgroup unreadable; report-without-start + usage paths. shellcheck-clean. Cold independent review passed (no confirmed bugs); one flagged sub-microsecond TOCTOU nit closed via atomic write (temp +mv).Compiler-server A/B (Done-when #3). Swap-off, sampled peak-anon, n=2 interleaved:
VBCSCompilerDisabling the compiler servers is worth it (consistent reduction + no resident server), but OFF sits right at 6 GiB for the build phase alone and the
testjob adds test+coverage on top — so #406's premise ("disabling brings peak well under 6 GiB → the budget loosens") is NOT supported. Size the cap off the live test-job peak-anon this instrument now reports (build+test+coverage), not off the build-only A/B. The older #411 probe (anon 7134 MiB) read higher than these swap-off sampled numbers (swap/read-method move it >1 GiB) and is superseded.Files.
scripts/ci-peak-anon.sh(new);.gitea/workflows/docker-build.yml(twotest-job steps);docs/ci-cd.md(instrument section rewritten — peak-anon headline + A/B table + premise verdict);docs/decisions.md(entry added). No.cstouched.Deferred / not in scope. The
buildjob (in-Docker publish, BuildKit) still has no instrument —memory.peakisn't reachable there the same way; left as a note in the issue, not a Done-when box. The definitive cap-sizing number for #604 is now the live test-job peak-anon on the nextmainCI run.