workAheadSegmenterLimit is not enforced: the slot check and its increment straddle an await (N concurrent tune-ins all run unthrottled)
#536
Closed
opened 2026-07-21 17:50:29 +02:00 by timothy
·
3 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#536
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found while measuring #532 on prod (v26.12.0, digest-verified). Not a regression from #516/#529 — the defect is older than both, and it is what made #350's original "one slot winner, two throttled" observation look deterministic when it is actually a race.
The defect
ErsatzTV.Application/Streaming/HlsSessionWorker.cs:246_workAheadCountis only incremented insideTranscode(line 467,Interlocked.Increment). So the check and the claim are not atomic, and they are separated by at least oneawait(GetWorkAheadLimitis a DB-backed config read). Every session that reads the counter before any of them increments sees0 < 1and takes the unthrottled work-ahead path. Line 272 (subsequentWorkAhead) has the identical shape.Interlocked.Incrementon the write side gives no mutual exclusion when the read side is a separate, earlierVolatile.Read— this is a plain check-then-act TOCTOU, same class as #231/#250.Evidence (prod,
streamingMinimumLogLevel: Debug,workAheadSegmenterLimit: 1)Three simultaneous tune-ins of idle channels 122/102/105:
3 of 3 sessions ran unthrottled against a configured limit of 1. Staggering the first tune by 3 s makes the limit appear to work (1 work-ahead + 2 throttled with
-readrate 1.05 -readrate_initial_burst 8), which is exactly why this has never been noticed: any measurement that starts sessions sequentially sees correct behaviour.A second variant is visible in the same logs — when the slot holder releases (
will NOT work ahead for channel 122at 17:44:43), the two waiting sessions both re-read0at 17:44:45.206 (same millisecond) and both re-acquire.Why it matters
The limit is a deliberate resource guarantee, not a tuning knob. From #350's fix rationale, on explicitly declining to raise it:
The guarantee does not hold in the one scenario it exists for: a simultaneous burst of viewers. Each unthrottled QSV transcode is also the thing that exhausted the hardware-frame pool in #529, so the blast radius is the failure mode we just shipped a floor for. Prod is not currently at risk (
qsvExtraHardwareFrames: 64absorbs it), but the pool sizing assumes at mostworkAheadSegmenterLimitunthrottled readers.Suggested fix
Claim the slot atomically instead of checking then claiming — e.g. a compare-exchange loop against the limit, or take the limit first and then
Interlocked.Incrementwith a decrement-and-fall-back-to-realtime if the post-increment value exceeds it:with
Transcodeno longer double-incrementing (the ownership then has to be passed in, so the release path inTranscode'sfinallystays symmetric). The interaction with the release path is the fiddly part and deserves the review-round attention, not the increment itself.Testing note
Per the #231/#250 lesson recorded in the queue lore, a one-shot
Barrier(N)+Task.WhenAlltest will not reliably catch this — hammer it over many rounds, and prove non-vacuity by inverting the real condition (never by stubbingif (true), which trips CS0219 → warnings-as-errors → a stale-dll false pass).Done-when
if (true))Post-deploy verification (tracked here; NOT a merge gate)
Split out of Done-when on 2026-07-21: this box cannot be satisfied before the fix is merged and an image carrying it is deployed, so leaving it in the merge checklist would make the gate unsatisfiable by construction. It is still owed, and this issue stays open until it is reported.
workAheadSegmenterLimitarg lines lack-readrateRefs #350, #516, #529, #532.
Claiming this (
in-progress).Approach, and where it deviates from the suggested fix:
ErsatzTV.Application/Streaming/WorkAheadSlots.cs— a pool withTryAcquire(limit)/Release(). Two reasons over patching in place: the staticintwas untestable (a process-wide field shared across tests), and the invariant now has one home to document.qsvExtraHardwareFramessizing (#529) is derived from that bound, I'd rather the invariant never be violated even momentarily.Runclaims the slot;Transcode(bool ownsWorkAheadSlot, …)derivesrealtime = !ownsWorkAheadSloton its first line and releases in its existingfinally. Acquisition deliberately stays in the caller:Runsets_statefrom the outcome, andTranscodereads that state on entry (wasSeekAndWorkAhead) to choose betweenDateTimeOffset.Nowand_transcodedUntilas the item start — folding acquisition inward would have silently flipped that branch. Inverting the parameter also removed theTranscode(!initialWorkAhead, …)double negative, so the contract is in the signature instead of implied.Grepped for other consumers of the budget:
_workAheadCounthad exactly one reader/writer pair, andFFmpegWorkAheadSegmentersis otherwise only read by the settings/troubleshooting queries. No second variant to cover.Negative control (per the #231/#250 lesson): non-vacuity proven. With the pre-fix check-then-act body reinstated,
TryAcquire_ParallelCallers_NeverExceedsLimitfails with 15912 over-claiming rounds out of 20000; 4 of 7 tests go red. Restored via an explicittouch+ rebuild (acprestore alone preserves an old mtime and MSBuild skips the rebuild — the control then "passes" on a stale dll). Build output checked forerror CS, not just a green tail.On the Done-when checklist: box 4 (verify on prod/test with N simultaneous tune-ins) is only satisfiable after the image carrying this change is deployed, so it cannot be ticked pre-merge. I'll treat it as a post-deploy verification against
ersatztv-testonce:latestrebuilds, and report the arg-line count back here.Full .NET suite green locally; independent adversarial review in flight before the push.
Closing — fixed, merged, and verified on a real deployment
Root cause. Structural, not a missing
Interlocked. The write side already usedInterlocked.Increment, which is exactly why the code read as thread-safe to every previous reader. The read side was a separate, earlierVolatile.ReadinRun, with at least oneawait(the DB-backed limit read) between it and the claim. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed0 < limit.What shipped (PR #538, merged as
5e5da9db):ErsatzTV.Application/Streaming/WorkAheadSlots.cs— new pool,TryAcquire(limit)claims by compare-exchange. CAS rather than the issue's suggested increment-then-back-off: both are correct on holder count, but increment-first transiently overshoots, andqsvExtraHardwareFramessizing (#529) is derived from that bound.HlsSessionWorker—Runclaims at both call sites;Transcode(bool ownsWorkAheadSlot, …)derivesrealtime = !ownsWorkAheadSlotand releases in its existingfinally. Acquisition stays in the caller becauseRunsets_statefrom the outcome andTranscodereads that state on entry (wasSeekAndWorkAhead) to pick the item start time — folding it inward would have silently flipped that branch.Releaseclamps at zero and recordsUnbalancedReleases(added from review) — the pool is process-wide, so a-1would have permanently inflated the budget.ErsatzTV.Core.Tests/Streaming/WorkAheadSlotsTests.cs— 9 tests, 8 threads × 20 000 rounds.Negative control. Reinstating the pre-fix check-then-act body: 15 912 over-claiming rounds of 20 000, 4 tests red. An independent reviewer replicated the harness separately and measured 8 966/20 000 — same order, different run. Restored with
touch+ forced rebuild, because acprestore preserves the old mtime and MSBuild would have skipped the rebuild, "passing" the control on a stale dll.Post-deploy verification (the box that was split out of Done-when — it was unsatisfiable pre-merge). Ran the identical 3-way simultaneous burst against
ersatztv-teston jazz,workAheadSegmenterLimit: 1, once on the stale pre-fix image it happened to still be running and once after pulling the merged:latest:8bbd7198)eeb428a4)-readrateSo the original three-simultaneous-tune-in reproduction is now a genuine positive control rather than an anecdote, and the guarantee holds under exactly the burst it exists for.
Two measurement notes for whoever runs this next, both of which would have produced a wrong answer:
version: 082dcbbe-ls75, created 2026-06-27. They are not evidence of what is running. The check that works is the running.Imageagainst the registry'sDocker-Content-Digestfor the tag.[18:48:13 DBG]) whiledocker logs -temits UTC, so--sincewindows silently mis-slice. The post-fix run used line-offset capture (wc -lbefore,tail -n +Nafter) instead.ersatztv-testwas left on the merged:latestwith its settings unchanged; prod was untouched (verified identical container ID andStartedAtbefore and after).Deferred: three Low findings from the re-review — a residual transient over-admission window a never-negative
Releasewould close, the counter's floor-triggered under-count, andUnbalancedReleasesnot being surfaced in any log — are filed as #539. All three require an unbalanced release, which the single guarded release site makes unreachable today.Docs: new
decisions.mdrecordffmpeg.work-ahead-slot-atomic, catalog regenerated; plus a correction on the #350 record, whose "every concurrent tune-in falls back to the throttled path" bullet described the intent rather than the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered.Closing record
Outcome:
workAheadSegmenterLimitis now enforced by an atomic compare-exchange claim on a sharedWorkAheadSlotspool. PR #538, merged as5e5da9db. Follow-up hardening deferred to #539.Root cause: A check-then-act TOCTOU split across an
await, not a missingInterlocked. The write side already usedInterlocked.Increment— which is exactly why the code read as thread-safe to every previous reader — while the read side was a separate, earlierVolatile.ReadinRun, with a DB-backed config read between them. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed0 < limitand started unthrottled.Decisions/conventions changed: added
ffmpeg.work-ahead-slot-atomic. Corrected theffmpeg.hls-cold-start-burst(#350) record: its "every concurrent tune-in falls back to the throttled path" bullet stated the intent, not the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered.Reusable knowledge:
Transcode(bool ownsWorkAheadSlot, …)) beats ownership implied by a re-derived condition: one release site, guarded by the same flag the caller acquired on.cppreserves its old mtime, so MSBuild skips the rebuild and a negative control "passes" against a stale dll.touch+ explicit rebuild, and grep the build forerror CS.2026-06-27on a build minutes old). Compare the running.Imageto the registry'sDocker-Content-Digestinstead.docker logs -temits UTC, so--sincewindows silently mis-slice. Capture by line offset (wc -lbefore,tail -n +Nafter).ersatztv-testis a single-service Komodo compose project on jazz (port 8410); prod is a separate stack, and its api.key is root-owned, so an unsudo'd read fails silently into a 401 that parses as "0 channels".Verification: full .NET suite green pre-push; negative control 15912/20000 over-claiming rounds with the pre-fix body restored (a reviewer independently measured 8966/20000); two cold adversarial passes covering both commits; CI green on all jobs; and a live before/after on
ersatztv-test— the identical 3-way simultaneous burst gave 3 of 3 channels unthrottled on the pre-fix image versus 1 (the configured limit) on the merged one.Deferred: #539 — three Low findings (a residual transient over-admission window a never-negative
Releasewould close, the counter's floor-triggered under-count, andUnbalancedReleasesnot surfaced in any log). All require an unbalanced release, unreachable while there is one guarded release site.Docs updated:
docs/decisions.md(+ regenerateddocs/decisions/README.md). No API/route/SPA surface changed, so no OpenAPI regen or parity-doc update was owed.