Re-measure HLS cold start on prod after v26.12.0+ ships the #529 floor, against #350's 13-sample baseline #532
Closed
opened 2026-07-21 17:02:53 +02:00 by timothy
·
2 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
No labels
priority: medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#532
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Follow-up to #519, which completed the test-side verification of #350/#516's cold-start burst. Split out because this step needs a release to reach prod — a criterion #519 could not satisfy itself.
What's already established
:latest, digest-verified), the burst works end-to-end: throttledfirstGop3.3–4.75 s → 0.94–1.28 s, worst case 4750 → 2081 ms, zero ENOMEM. Full table in #519's closing comment.qsvExtraHardwareFrameswas set to64on 2026-07-21 ahead of the fix shipping, so prod is no longer failing transcodes. But prod still runsv26.11.0, which has neither the burst (#516) nor the floor (#529) in code.What to do
Once a release containing both
2834605d(#529 floor) and34d76095(#516 burst) reaches prod:docker image inspect --format '{{.Id}}'against the registry'sDocker-Content-Digestheader. Note: on jazz (.29) Docker reports.Idas the manifest digest, so comparing it to the manifest body's.config.digestgives a false mismatch on every image. OCI labels are inherited from the linuxserver base image and lie.HLS cold-starttelemetry against the 13-sample baseline table in #350, specifically the 3845/6357 ms non-slot-winner case.streamingMinimumLogLevel: Debug(prod is currentlyInformation, soffmpeg hls argumentsis not emitted). Remember to set it back.qsvExtraHardwareFramesis now64, so the #529 floor will be a no-op there — that is expected, not a failed check.Open questions worth answering while measuring
firstGopin 295–366 ms on test; post-fix the fastest was ~942 ms. Hypothesis (untested): with three sessions now all doing real work on one iGPU, contention raises the floor where two previously died early. Prod has more headroom and different content — worth checking whether the fast path is genuinely slower or that was a test-bed artifact.Done-when
Refs #350, #516, #519, #529.
Claiming (Claude Code orchestrator session, 2026-07-21). Top-ranked eligible pick from
scripts/select-queue.sh, noCLAIM?/UMBRELLA?flags, deps clear.Precondition is now satisfied — Done-when box 1 verified by digest before claiming:
ersatztvon jazz (.29),StartedAt2026-07-21T15:34:14Zsha256:fc8cb8ed…:prodDocker-Content-Digestsha256:fc8cb8ed…:26.12.0Docker-Content-Digestsha256:fc8cb8ed…:b52c9388(the git sha tag)Docker-Content-Digestsha256:fc8cb8ed…v26.12.0b52c9388git merge-base --is-ancestorconfirms both2834605d(#529 floor) and34d76095(#516 burst) are ancestors ofv26.12.0. Per #519's refinement I compared.Idagainst the registry'sDocker-Content-Digestheader, not the manifest body's.config.digest(which gives a false mismatch on every image on this host).Plan for the remaining boxes: raise
streamingMinimumLogLeveltoDebug, run 3-concurrent tunes of idle channels against #350's 3-concurrent table (122/102/105 → firstGop 866 / 3845 / 6357 ms), confirm-readrate_initial_burstis present in the actualffmpeg hls arguments, restore the log level, and record findings on #350. Will also take a position on the ~300 ms best-case question rather than leaving it open.Closing record
Outcome: All five Done-when boxes satisfied. Prod (v26.12.0, digest-verified) re-measured against #350's 3-concurrent baseline: throttled
firstGop3845/6357 ms → 1658–2527 ms across two independent rounds, worst case gone, zero ENOMEM. The burst is confirmed in the actualffmpeg hls arguments, not inferred from latency. Full table posted to #350 as this issue requires.Root cause: n/a (measurement task, no defect fixed here). The incidental defect found during measurement has its own root cause, recorded in #536.
Decisions/conventions changed: none. No code, no docs, no PR — this issue's deliverable is evidence on #350.
Reusable knowledge:
firstGop370 / 447 ms, beating even #350's pre-fix 866 ms. Answered, not deferred.ffmpegInit(1.0–2.1 s), which the burst does not address — it boundsfirstGoponly. That is why prod's post-fix numbers (1.7–2.5 s) are worse than test's (0.94–1.28 s) despite prod having more headroom. Any further cold-start work should target input open/probe, not the read throttle.Verification: image identity closed end to end (container
sha256:fc8cb8ed…== registry:prod==:26.12.0==:b52c9388;merge-base --is-ancestorfor both34d76095and2834605d) before any sample was trusted, per the #350 stale-image lesson. Two independent measurement rounds, 17:46 and 17:48. Per-process arg lines inspected individually rather than counted in aggregate. Host load 0.95 → 2.83 throughout;deadlineExpired falseon every sample; 0 ENOMEM/Could not open encodermarkers.Deferred/follow-ups:
bug,priority: medium) —workAheadSegmenterLimitis not enforced; the slot check atHlsSessionWorker.cs:246is aVolatile.Readseparated from itsInterlocked.Incrementby anawait, so N simultaneous tune-ins all win the slot. Pre-existing, not a #516/#529 regression. Prod is not at risk today (qsvExtraHardwareFrames: 64absorbs it).Docs updated: none required — no convention, route, endpoint, or decision changed.
Prod left clean:
streamingMinimumLogLevelrestored toInformation(verified by read-back), all measurement sessions idled out (0 ffmpeg processes), no config or channel state altered.