Files
ersatztv/docs/decisions/records/ffmpeg/hls-cold-start-burst.md
T
timothyandClaude Opus 5 56afa4652d
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 10s
PR Gates / Docs update reminder (pull_request) Successful in 25s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 24s
Review verdict / Set review-verdict status (pull_request_target) Successful in 10s
PR Gates / decisions lifecycle (pull_request) Successful in 29s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m1s
PR Gates / Script tests (pytest) (pull_request) Failing after 1m53s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 19m4s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 22m36s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 24m58s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
fix(726): let a lagging realtime input catch up so a sparse stream can't pin it
-readrate throttles an INPUT and paces it off whichever of its streams is
furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same
-i as the video -- SubtitleInputFile carries the video's path and ComplexFilter
resolves it to a stream specifier on that input, so CommandGenerator never emits
a second -i for it. Being sparse, that subtitle stream falls further behind every
second and drags video throughput down with it: measured 0.53x realtime against
the 1.0x a live client consumes at, which drains the client buffer until it
stalls. FFmpeg names the culprit itself at -loglevel warning:

  [sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000
                          after a lag of 0.922s   (then 1.24, 1.56, ... 3.80)

Add -readrate_catchup (6.0) to realtime video/audio inputs, capability-gated
through FFmpegKnownOption.HasOption exactly as -readrate_initial_burst is, so a
binary without it silently keeps today's behavior instead of failing to start.
The option first shipped in ffmpeg 8.0, which is NEWER than 7.1 -- hence runtime
detection rather than a version assumption. Still images and concat inputs are
excluded, mirroring #350.

Measured on prod (QSV, -threads 1, dvd_subtitle -> overlay), 45s steady-state
window after a 6s settle, replaying the captured production command line:

  baseline 1.05        0.533x  (x3 runs)
  + catchup 2.0        0.711x
  + catchup 6.0        1.067x  (x2 runs)
  + catchup 20.0       1.067x
  no subtitle overlay  1.067x  (control)

Baseline reproduces the reported 0.53x and the control the reported 1.07x, so
the harness is validated on both sides. Reproduces on software libx264 too
(0.533x -> 1.067x), as expected for an input-pacing option. Raising the base
-readrate is not an alternative and was measured: 2.0 -> 0.62x, 3.0 -> 0.80x,
4.0 -> 0.80x, 6.0 -> 0.89x -- it asymptotes below realtime because the rate
ceiling was never the binding constraint.

On #529 (readrate was incidentally bounding QSV hardware-frame allocation): the
20.0-vs-6.0 row is why 6.0 was chosen, NOT evidence about allocation -- it is a
steady-state throughput number, not a count of frames in flight. Nor is the bound
safe because read rate is allocation-irrelevant: #529 measured that it is not (at
extra_hw_frames=0, 1.05 without a burst exits 0 while 1.05+burst hits ENOMEM).
Read rate changes how fast frames enter the graph, not how deep its queues are,
and #529's failure only appeared with NO pool headroom. The 64-frame floor now
guarantees headroom, so the load-bearing measurement is row 5 of that truth
table -- no -readrate at all with 64 frames -> 14 segments, exit 0 -- and a 6x
ceiling is strictly less aggressive than no throttle. Reinforcing it,
-readrate_initial_burst 8 has read flat out at the start of every playout item
since #350, so an unbounded read here is not new. A 240s QSV soak at
QsvExtraHardwareFrames=64 across 60 segment boundaries corroborates: 1.043x
sustained, zero "Cannot allocate memory" / "Could not open encoder", RSS 166MB
vs 156MB at baseline -- corroborates rather than demonstrates, since it stayed
largely caught up.

Catchup does NOT subsume the #350 burst; measured time-to-first-segment:
-readrate alone 3.71s, +burst 0.72s, +catchup alone 3.65s, both 0.67s. They fix
orthogonal metrics.

The regression test is built on a BITMAP subtitle deliberately -- a text subtitle
is fetched by the libass filter outside the demuxer, so the same assertions would
pass vacuously while the bug is fully present. It asserts on the "[0:0][0:2]overlay"
label, which is the mechanism: subtitle stream 2 resolving onto the video's input.
Every new test was mutation-checked, each producing exactly its own expected red.

Fixes #726

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 22:50:09 +02:00

6.2 KiB
Raw Blame History

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
ffmpeg.hls-cold-start-burst 2026-07-20 — HLS cold start is fixed with `-readrate_initial_burst`, not by raising the work-ahead limit (#350) active 2026-07-20 none none HLS cold-start latency is fixed with a bounded `-readrate_initial_burst` (gated on FFmpeg ≥6.1 capability detection), not by raising `work_ahead_limit`, which would remove the concurrency guarantee it exists for. HLS cold start, readrate, work_ahead_limit, HlsSessionWorker, FFmpegKnownOption capability gate · paths: `HlsSessionWorker`, `SetRealtimeInput`, `FFmpegPlaybackSettingsCalculator`, `FFmpegKnownOption`/`HasOption` · issues: #350 `SetRealtimeInput` readrate-burst option; `FFmpegKnownOption.HasOption` version-capability gate

Correction (2026-07-21, #529): the claim below that the burst is "bounded" is true only in seconds of input — it is not a bound on memory or hardware surfaces. -readrate was also incidentally bounding how fast decoded frames enter the filter graph, and removing that on a QSV pipeline whose profile stores qsvExtraHardwareFrames: 0 exhausts the upload pool: the graph fails with -12 (Cannot allocate memory), h264_qsv never opens, and zero segments are written. The burst is not the root cause (a work-ahead start takes no -readrate at all and was already failing the same way in production), but it removed the throttle on every realtime session and so made the failure near-deterministic. See ffmpeg.qsv-extra-hw-frames-floor; the decision recorded here still stands, with that floor in place.

Correction (2026-07-21, #536): the bullet below stating that "every concurrent tune-in falls back to the throttled path" described the intent, not the behaviour. The slot check and its increment straddled an await, so N simultaneous tune-ins all read 0 < limit and all started unthrottled; the "one winner, two throttled" observation held only because those tunes were effectively staggered. The measurement and the conclusions drawn from it stand — slot availability was the variable behind the bimodality — but the guarantee itself was not enforced until ffmpeg.work-ahead-slot-atomic.

  • -readrate throttles from the first read, so it sets a floor on time-to-first-segment. The realtime playback path pins input reading to 1.05× wall clock so a channel behaves like live TV. Since OutputFormatHls.SegmentSeconds is 4 and the segmenter serves the playlist only once the first segment exists, the playlist cannot appear sooner than ~4/1.05 ≈ 3.8 s. Measured on real media: time-to-first-playlist 5369/5344 ms with -readrate 1.05, 648/649 ms with an initial burst.
  • The cold-start bimodality was never about the media. HlsSessionWorker grants an unthrottled SeekAndWorkAhead start only while _workAheadCount < ffmpeg.segmenter.work_ahead_limit (prod: 1); every concurrent tune-in falls back to the throttled path. Three concurrent tunes on prod: the one that won the slot reached firstGop in 866 ms, the other two in 3845 ms and 6357 ms. This is why the earlier rounds found no correlation with subtitle burn-in, GOP length, or source file — the variable was slot availability, and the same channel could differ 7.2× between tunes.
  • Two earlier hypotheses are falsified, not deferred. Accurate-seek decode-discard (the issue's ranked #1 driver) costs 30100 ms on real prod media, and capping -probesize/-analyzeduration buys 2050 ms. Neither can account for seconds. Recorded here so they are not re-proposed.
  • Burst rather than a bigger work-ahead budget. Raising work_ahead_limit would fix latency by deleting the guarantee that limit exists for — it caps how many unthrottled transcodes N viewers can start at once. The burst is bounded (SegmentSeconds * 2 = 8 s of input, enough for the first segments at the default InitialSegmentCount of 1), after which live pacing resumes. An operator who raises InitialSegmentCount above 2 gets less of the benefit; that is a deliberate trade.
  • The burst is per ffmpeg process — i.e. per playout item — not per session. SetRealtimeInput runs on every pipeline build and HlsSessionWorker spawns a process per item, so each item boundary bursts too; this is not only the session's cold start. Two consequences, both accepted: item transitions get the same head start (a benefit), and on a channel whose items are shorter than the burst every item transcodes unthrottled, so the instantaneous-concurrency guarantee that work_ahead_limit provides is weaker than before — weaker, not absent, because HlsSessionWorker's transcodedBuffer <= 1min gate still stops the loop at a 60 s buffer, leaving average CPU unchanged. Making the burst strictly cold-start-only would mean plumbing a "first process of this session" flag through FFmpegState; that complexity was not judged worth a bounded peak.
  • Still images are excluded. Their video input is paced by the realtime filter and takes no readrate at all, so a burst would only run the audio input ahead of the video for songs and offline filler, with no cold-start gain to show for it. (-readrate_catchup mirrors this exclusion for the same reason — ffmpeg.readrate-catchup-sparse-streams, #726.)
  • Non-HLS realtime outputs (TransportStream, HLS-Direct) burst too, since FFmpegPlaybackSettingsCalculator makes them unconditionally realtime. That is untested by the benchmark, which was segmenter-only; it is kept because the same first-read throttle delays those clients identically, and the outer WrapSegmenter/Concat processes still pace at readrate 1.0 with no burst.
  • Gated on runtime capability, not on a parsed version. -readrate_initial_burst needs FFmpeg ≥ 6.1, and FFmpegKnownOption/HasOption already existed for exactly this (its AllOptions list had simply been empty). Detection parses ffmpeg -h long, so an older binary silently keeps today's behavior instead of failing to start — the same fail-safe posture as the other capability gates, and cheaper to reason about than the version-string parsing in NvidiaHardwareCapabilities.