Re-measure HLS cold start on prod after v26.12.0+ ships the #529 floor, against #350's 13-sample baseline #532

Closed
opened 2026-07-21 17:02:53 +02:00 by timothy · 2 comments
Owner

Follow-up to #519, which completed the test-side verification of #350/#516's cold-start burst. Split out because this step needs a release to reach prod — a criterion #519 could not satisfy itself.

What's already established

  • On test (:latest, digest-verified), the burst works end-to-end: throttled firstGop 3.3–4.75 s → 0.94–1.28 s, worst case 4750 → 2081 ms, zero ENOMEM. Full table in #519's closing comment.
  • Prod's profile is already mitigated — qsvExtraHardwareFrames was set to 64 on 2026-07-21 ahead of the fix shipping, so prod is no longer failing transcodes. But prod still runs v26.11.0, which has neither the burst (#516) nor the floor (#529) in code.

What to do

Once a release containing both 2834605d (#529 floor) and 34d76095 (#516 burst) reaches prod:

  1. Verify the deployed image by digest first — docker image inspect --format '{{.Id}}' against the registry's Docker-Content-Digest header. Note: on jazz (.29) Docker reports .Id as the manifest digest, so comparing it to the manifest body's .config.digest gives a false mismatch on every image. OCI labels are inherited from the linuxserver base image and lie.
  2. Tune 3 prod channels concurrently and compare HLS cold-start telemetry against the 13-sample baseline table in #350, specifically the 3845/6357 ms non-slot-winner case.
  3. Confirm the burst is in the command rather than inferred from latency — needs streamingMinimumLogLevel: Debug (prod is currently Information, so ffmpeg hls arguments is not emitted). Remember to set it back.
  4. Prod's stored qsvExtraHardwareFrames is now 64, so the #529 floor will be a no-op there — that is expected, not a failed check.

Open questions worth answering while measuring

  • Does the ~300 ms best case come back? Pre-fix, the work-ahead slot winner reached firstGop in 295–366 ms on test; post-fix the fastest was ~942 ms. Hypothesis (untested): with three sessions now all doing real work on one iGPU, contention raises the floor where two previously died early. Prod has more headroom and different content — worth checking whether the fast path is genuinely slower or that was a test-bed artifact.
  • Per-item burst CPU/GPU spike pattern on channels with short items — unassessable on test because the QSV failures dominated every sample (#519's "watch for" item).
  • Does the banked buffer lead help the Dispatcharr join-stutter (server-management#503)?

Done-when

Refs #350, #516, #519, #529.

Follow-up to #519, which completed the **test-side** verification of #350/#516's cold-start burst. Split out because this step needs a release to reach prod — a criterion #519 could not satisfy itself. ## What's already established - **On test** (`:latest`, digest-verified), the burst works end-to-end: throttled `firstGop` **3.3–4.75 s → 0.94–1.28 s**, worst case 4750 → 2081 ms, zero ENOMEM. Full table in [#519's closing comment](http://192.168.1.95:3000/timothy/ersatztv/issues/519#issuecomment-14741). - **Prod's profile is already mitigated** — `qsvExtraHardwareFrames` was set to `64` on 2026-07-21 ahead of the fix shipping, so prod is no longer failing transcodes. But prod still runs `v26.11.0`, which has **neither the burst (#516) nor the floor (#529)** in code. ## What to do Once a release containing both `2834605d` (#529 floor) and `34d76095` (#516 burst) reaches prod: 1. **Verify the deployed image by digest first** — `docker image inspect --format '{{.Id}}'` against the registry's `Docker-Content-Digest` header. Note: on jazz (.29) Docker reports `.Id` as the **manifest** digest, so comparing it to the manifest body's `.config.digest` gives a false mismatch on every image. OCI labels are inherited from the linuxserver base image and lie. 2. Tune 3 prod channels concurrently and compare `HLS cold-start` telemetry against the **13-sample baseline table** in [#350](http://192.168.1.95:3000/timothy/ersatztv/issues/350#issuecomment-14295), specifically the 3845/6357 ms non-slot-winner case. 3. Confirm the burst is in the command rather than inferred from latency — needs `streamingMinimumLogLevel: Debug` (prod is currently `Information`, so `ffmpeg hls arguments` is not emitted). Remember to set it back. 4. Prod's stored `qsvExtraHardwareFrames` is now `64`, so the #529 floor will be a no-op there — that is expected, not a failed check. ## Open questions worth answering while measuring - **Does the ~300 ms best case come back?** Pre-fix, the work-ahead slot winner reached `firstGop` in 295–366 ms on test; post-fix the fastest was ~942 ms. Hypothesis (untested): with three sessions now all doing real work on one iGPU, contention raises the floor where two previously died early. Prod has more headroom and different content — worth checking whether the fast path is genuinely slower or that was a test-bed artifact. - **Per-item burst CPU/GPU spike pattern** on channels with short items — unassessable on test because the QSV failures dominated every sample (#519's "watch for" item). - **Does the banked buffer lead help the Dispatcharr join-stutter** (server-management#503)? ## Done-when - [x] Deployed prod image verified by digest to contain both #516 and #529 - [x] 3-concurrent-tune measured on prod, compared against #350's 13-sample baseline - [x] Burst confirmed present in the actual command at Debug, then log level restored - [x] The ~300 ms best-case question answered or explicitly deferred with a reason - [x] Findings recorded on #350 so its decision record reflects real prod numbers Refs #350, #516, #519, #529.
timothy added the priority: medium label 2026-07-21 17:02:53 +02:00
timothy added the in-progress label 2026-07-21 17:40:31 +02:00
Author
Owner

Claiming (Claude Code orchestrator session, 2026-07-21). Top-ranked eligible pick from scripts/select-queue.sh, no CLAIM?/UMBRELLA? flags, deps clear.

Precondition is now satisfied — Done-when box 1 verified by digest before claiming:

link in the chain value
running container ersatztv on jazz (.29), StartedAt 2026-07-21T15:34:14Z sha256:fc8cb8ed…
registry :prod Docker-Content-Digest sha256:fc8cb8ed…
registry :26.12.0 Docker-Content-Digest sha256:fc8cb8ed…
registry :b52c9388 (the git sha tag) Docker-Content-Digest sha256:fc8cb8ed…
git v26.12.0 b52c9388

git merge-base --is-ancestor confirms both 2834605d (#529 floor) and 34d76095 (#516 burst) are ancestors of v26.12.0. Per #519's refinement I compared .Id against the registry's Docker-Content-Digest header, not the manifest body's .config.digest (which gives a false mismatch on every image on this host).

Plan for the remaining boxes: raise streamingMinimumLogLevel to Debug, run 3-concurrent tunes of idle channels against #350's 3-concurrent table (122/102/105 → firstGop 866 / 3845 / 6357 ms), confirm -readrate_initial_burst is present in the actual ffmpeg hls arguments, restore the log level, and record findings on #350. Will also take a position on the ~300 ms best-case question rather than leaving it open.

Claiming (Claude Code orchestrator session, 2026-07-21). Top-ranked eligible pick from `scripts/select-queue.sh`, no `CLAIM?`/`UMBRELLA?` flags, deps clear. **Precondition is now satisfied — Done-when box 1 verified by digest before claiming:** | link in the chain | value | |---|---| | running container `ersatztv` on jazz (.29), `StartedAt` 2026-07-21T15:34:14Z | `sha256:fc8cb8ed…` | | registry `:prod` `Docker-Content-Digest` | `sha256:fc8cb8ed…` | | registry `:26.12.0` `Docker-Content-Digest` | `sha256:fc8cb8ed…` | | registry `:b52c9388` (the git sha tag) `Docker-Content-Digest` | `sha256:fc8cb8ed…` | | git `v26.12.0` | `b52c9388` | `git merge-base --is-ancestor` confirms **both** `2834605d` (#529 floor) and `34d76095` (#516 burst) are ancestors of `v26.12.0`. Per #519's refinement I compared `.Id` against the registry's `Docker-Content-Digest` header, **not** the manifest body's `.config.digest` (which gives a false mismatch on every image on this host). Plan for the remaining boxes: raise `streamingMinimumLogLevel` to `Debug`, run 3-concurrent tunes of idle channels against #350's [3-concurrent table](http://192.168.1.95:3000/timothy/ersatztv/issues/350#issuecomment-14349) (122/102/105 → firstGop 866 / **3845** / **6357** ms), confirm `-readrate_initial_burst` is present in the actual `ffmpeg hls arguments`, restore the log level, and record findings on #350. Will also take a position on the ~300 ms best-case question rather than leaving it open.
Author
Owner

Closing record

Outcome: All five Done-when boxes satisfied. Prod (v26.12.0, digest-verified) re-measured against #350's 3-concurrent baseline: throttled firstGop 3845/6357 ms → 1658–2527 ms across two independent rounds, worst case gone, zero ENOMEM. The burst is confirmed in the actual ffmpeg hls arguments, not inferred from latency. Full table posted to #350 as this issue requires.

Root cause: n/a (measurement task, no defect fixed here). The incidental defect found during measurement has its own root cause, recorded in #536.

Decisions/conventions changed: none. No code, no docs, no PR — this issue's deliverable is evidence on #350.

Reusable knowledge:

  • The ~300 ms best case was a test-bed artifact, not a burst regression. #519 left this as an open hypothesis; prod's work-ahead winner hit firstGop 370 / 447 ms, beating even #350's pre-fix 866 ms. Answered, not deferred.
  • Prod's residual cold-start cost has moved into ffmpegInit (1.0–2.1 s), which the burst does not address — it bounds firstGop only. That is why prod's post-fix numbers (1.7–2.5 s) are worse than test's (0.94–1.28 s) despite prod having more headroom. Any further cold-start work should target input open/probe, not the read throttle.
  • A staggered measurement hides the work-ahead race. Starting three tunes simultaneously produced 3 unthrottled sessions against a limit of 1; a 3 s stagger produced the textbook 1+2 split. Any future harness that wants the throttled path must stagger deliberately — and any harness that wants to test the limit must not.

Verification: image identity closed end to end (container sha256:fc8cb8ed… == registry :prod == :26.12.0 == :b52c9388; merge-base --is-ancestor for both 34d76095 and 2834605d) before any sample was trusted, per the #350 stale-image lesson. Two independent measurement rounds, 17:46 and 17:48. Per-process arg lines inspected individually rather than counted in aggregate. Host load 0.95 → 2.83 throughout; deadlineExpired false on every sample; 0 ENOMEM/Could not open encoder markers.

Deferred/follow-ups:

  • #536 (bug, priority: medium) — workAheadSegmenterLimit is not enforced; the slot check at HlsSessionWorker.cs:246 is a Volatile.Read separated from its Interlocked.Increment by an await, so N simultaneous tune-ins all win the slot. Pre-existing, not a #516/#529 regression. Prod is not at risk today (qsvExtraHardwareFrames: 64 absorbs it).
  • Per-item burst CPU/GPU spike pattern remains unassessed — it needs sustained cross-item observation, not a cold-start harness. Flagged on #350 rather than silently dropped.
  • Dispatcharr join-stutter (server-management#503) untouched — the banked-buffer question needs a Dispatcharr-side observation, not an ErsatzTV measurement.

Docs updated: none required — no convention, route, endpoint, or decision changed.

Prod left clean: streamingMinimumLogLevel restored to Information (verified by read-back), all measurement sessions idled out (0 ffmpeg processes), no config or channel state altered.

## Closing record **Outcome**: All five Done-when boxes satisfied. Prod (v26.12.0, digest-verified) re-measured against #350's 3-concurrent baseline: throttled `firstGop` **3845/6357 ms → 1658–2527 ms** across two independent rounds, worst case gone, zero ENOMEM. The burst is confirmed in the actual `ffmpeg hls arguments`, not inferred from latency. Full table posted to [#350](http://192.168.1.95:3000/timothy/ersatztv/issues/350#issuecomment-14808) as this issue requires. **Root cause**: n/a (measurement task, no defect fixed here). The incidental defect found *during* measurement has its own root cause, recorded in #536. **Decisions/conventions changed**: none. No code, no docs, no PR — this issue's deliverable is evidence on #350. **Reusable knowledge**: - **The ~300 ms best case was a test-bed artifact, not a burst regression.** #519 left this as an open hypothesis; prod's work-ahead winner hit `firstGop` **370 / 447 ms**, beating even #350's pre-fix 866 ms. Answered, not deferred. - **Prod's residual cold-start cost has moved into `ffmpegInit` (1.0–2.1 s)**, which the burst does not address — it bounds `firstGop` only. That is why prod's post-fix numbers (1.7–2.5 s) are worse than test's (0.94–1.28 s) despite prod having *more* headroom. Any further cold-start work should target input open/probe, not the read throttle. - **A staggered measurement hides the work-ahead race.** Starting three tunes simultaneously produced 3 unthrottled sessions against a limit of 1; a 3 s stagger produced the textbook 1+2 split. Any future harness that wants the throttled path must stagger deliberately — and any harness that wants to *test the limit* must not. **Verification**: image identity closed end to end (container `sha256:fc8cb8ed…` == registry `:prod` == `:26.12.0` == `:b52c9388`; `merge-base --is-ancestor` for both `34d76095` and `2834605d`) before any sample was trusted, per the #350 stale-image lesson. Two independent measurement rounds, 17:46 and 17:48. Per-process arg lines inspected individually rather than counted in aggregate. Host load 0.95 → 2.83 throughout; `deadlineExpired false` on every sample; 0 ENOMEM/`Could not open encoder` markers. **Deferred/follow-ups**: - **#536** (`bug`, `priority: medium`) — `workAheadSegmenterLimit` is not enforced; the slot check at `HlsSessionWorker.cs:246` is a `Volatile.Read` separated from its `Interlocked.Increment` by an `await`, so N simultaneous tune-ins all win the slot. Pre-existing, not a #516/#529 regression. Prod is not at risk today (`qsvExtraHardwareFrames: 64` absorbs it). - **Per-item burst CPU/GPU spike pattern** remains unassessed — it needs sustained cross-item observation, not a cold-start harness. Flagged on #350 rather than silently dropped. - **Dispatcharr join-stutter** (server-management#503) untouched — the banked-buffer question needs a Dispatcharr-side observation, not an ErsatzTV measurement. **Docs updated**: none required — no convention, route, endpoint, or decision changed. **Prod left clean**: `streamingMinimumLogLevel` restored to `Information` (verified by read-back), all measurement sessions idled out (0 ffmpeg processes), no config or channel state altered.
timothy removed the in-progress label 2026-07-21 17:52:47 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#532