-readrate throttles an input and paces it off whichever of its streams is furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same -i as the video — SubtitleInputFile carries the video's path and ComplexFilter resolves it to a stream specifier on that input, so CommandGenerator never emits a second -i for it. Being sparse, that subtitle stream falls further behind every second and drags video throughput down with it: 0.53x realtime against the 1.0x a live client consumes at, which drains the client buffer until the channel stalls.
FFmpeg names the culprit itself at -loglevel warning — the subtitle stream's pts stays pinned while its lag grows monotonically:
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000 after a lag of 0.922s
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000 after a lag of 1.242s
... 1.562s, 1.882s, 2.202s, 2.522s, 2.842s, 3.162s, 3.482s, 3.802s
The fix
-readrate_catchup 6.0 on realtime video/audio inputs, capability-gated through FFmpegKnownOption.HasOption exactly as -readrate_initial_burst is. Still-image and concat inputs are excluded, mirroring #350.
The option first shipped in FFmpeg 8.0 (added 2025-02-15, 6232f416b) — newer than 7.1, which is precisely why it is detected at runtime rather than assumed from a version. Our pinned base image is ersatztv-ffmpeg:8.1.2 and both prod and ersatztv-test run it.
Measurements (prod, QSV h264_qsv, -threads 1, real production command line)
Protocol per the issue: steady-state segment production, 45s window after a 6s settle, no -t bound.
Variant
Throughput
baseline
0.533x (×3 runs)
-readrate_catchup 2.0
0.711x
-readrate_catchup 6.0
1.067x (×2 runs)
-readrate_catchup 20.0
1.067x
no subtitle overlay (control)
1.067x
Baseline reproduces the reported 0.53x exactly and the control the reported 1.07x exactly, so the harness is validated on both sides; variance across runs was zero. Also reproduces on software libx264 (0.533x → 1.067x), so this is not QSV-specific.
Raising the base -readrate is not an alternative and was measured: 2.0→0.62x, 3.0→0.80x, 4.0→0.80x, 6.0→0.89x — it asymptotes below realtime.
catchup 20.0 matching 6.0 is why 6.0 was chosen: above the catch-up point the value is not a throughput dial.
Stated carefully, because an earlier draft of this reasoning was wrong and a reviewer caught it:
This is not safe because read rate is allocation-irrelevant — #529 measured that it is not (at extra_hw_frames=0, 1.05 without a burst exits 0 while 1.05+burst hits ENOMEM).
Read rate changes how fast frames enter the graph, not how deep its queues are, and #529's failure only appeared with no pool headroom. The 64-frame floor now guarantees headroom.
So the load-bearing evidence is row 5 of #529's own truth table — no -readrate at all with 64 frames → 14 segments, exit 0. A 6x ceiling is strictly less aggressive than no throttle.
Reinforcing it: -readrate_initial_burst 8 has read flat out at the start of every playout item since #350, so an unbounded read on this path is not new.
A 240s QSV soak (QsvExtraHardwareFrames=64, 60 segment boundaries) corroborates: 1.043x sustained, zero Cannot allocate memory / Could not open encoder, RSS 166MB vs 156MB. It stayed largely caught-up, so it corroborates rather than demonstrates.
The regression test is built on a bitmap subtitle deliberately — a text subtitle is fetched by the libass filter outside the demuxer, so the same assertions would pass vacuously while the bug is fully present. It asserts on [0:0][0:2]overlay, which pins the mechanism: subtitle stream 2 resolving onto the video's input.
Verification beyond unit tests
Real capability parser verified against actual ffmpeg -h long output — extracts readrate_catchup exactly (the unit tests use a fake capabilities object, so this closes that gap).
scripts/e2e-local.sh instance started cleanly, ffmpeg located, no new errors, stopped by exact PID.
Local E2E cannot reproduce the behavior (no local media with embedded bitmap subs, no local QSV); the prod measurement above is the stronger evidence and is what the fix rests on.
Review
Three independent cold review rounds, each in an isolated worktree. Rounds 1 and 2 returned MERGEABLE; round 3 returned BLOCKED on one finding — the retracted #529 claim had been fixed in the decision record but survived verbatim in the commit message — now corrected in both, mechanically verified.
Docs
New decision record ffmpeg.readrate-catchup-sparse-streams; forward reference added to ffmpeg.hls-cold-start-burst; catalog regenerated.
Corrections to the issue
Affected population is 3,182 media versions (12.9%), not 6,369/25.8% — the earlier figure counted metadata records, not versions.
964 of those are bitmap-only, so subtitle-selection preference could never have closed this; the pipeline fix covers all 3,182.
The subtitleMode: null side finding is a non-bug — the list DTO deliberately omits the field, GET /api/v1/channels/{id} returns it correctly, and jq renders an absent key as null.
Filed separately
#733 — dvbsub/dvb_subtitle missing from both IsImage lists (109 streams misrouted to the text path)
#734, #735, #736 — UI/API follow-ups on exposing and explaining FFmpeg tunables
fixes #726
## What was wrong
`-readrate` throttles an **input** and paces it off whichever of its streams is *furthest behind*. An embedded bitmap subtitle (PGS/DVD) is read through the **same `-i` as the video** — `SubtitleInputFile` carries the video's path and `ComplexFilter` resolves it to a stream specifier on that input, so `CommandGenerator` never emits a second `-i` for it. Being sparse, that subtitle stream falls further behind every second and drags video throughput down with it: **0.53x realtime** against the 1.0x a live client consumes at, which drains the client buffer until the channel stalls.
FFmpeg names the culprit itself at `-loglevel warning` — the subtitle stream's `pts` stays pinned while its lag grows monotonically:
```
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000 after a lag of 0.922s
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000 after a lag of 1.242s
... 1.562s, 1.882s, 2.202s, 2.522s, 2.842s, 3.162s, 3.482s, 3.802s
```
## The fix
`-readrate_catchup 6.0` on realtime video/audio inputs, capability-gated through `FFmpegKnownOption.HasOption` exactly as `-readrate_initial_burst` is. Still-image and concat inputs are excluded, mirroring #350.
The option first shipped in **FFmpeg 8.0** (added 2025-02-15, `6232f416b`) — *newer* than 7.1, which is precisely why it is detected at runtime rather than assumed from a version. Our pinned base image is `ersatztv-ffmpeg:8.1.2` and both prod and `ersatztv-test` run it.
## Measurements (prod, QSV `h264_qsv`, `-threads 1`, real production command line)
Protocol per the issue: steady-state segment production, 45s window after a 6s settle, no `-t` bound.
| Variant | Throughput |
|---|---|
| **baseline** | **0.533x** (×3 runs) |
| `-readrate_catchup 2.0` | 0.711x |
| **`-readrate_catchup 6.0`** | **1.067x** (×2 runs) |
| `-readrate_catchup 20.0` | 1.067x |
| no subtitle overlay (control) | 1.067x |
Baseline reproduces the reported 0.53x **exactly** and the control the reported 1.07x **exactly**, so the harness is validated on both sides; variance across runs was zero. Also reproduces on **software libx264** (0.533x → 1.067x), so this is not QSV-specific.
Raising the base `-readrate` is not an alternative and was measured: 2.0→0.62x, 3.0→0.80x, 4.0→0.80x, 6.0→0.89x — it asymptotes *below* realtime.
`catchup 20.0` matching `6.0` is why **6.0** was chosen: above the catch-up point the value is not a throughput dial.
## On #529 (the allocation hazard)
Stated carefully, because an earlier draft of this reasoning was wrong and a reviewer caught it:
- This is **not** safe because read rate is allocation-irrelevant — #529 measured that it is not (at `extra_hw_frames=0`, `1.05` without a burst exits 0 while `1.05`+burst hits ENOMEM).
- Read rate changes how fast frames *enter* the graph, not how deep its queues are, and #529's failure only appeared with **no pool headroom**. The 64-frame floor now guarantees headroom.
- So the load-bearing evidence is **row 5 of #529's own truth table — no `-readrate` at all with 64 frames → 14 segments, exit 0**. A 6x ceiling is strictly less aggressive than no throttle.
- Reinforcing it: `-readrate_initial_burst 8` has read *flat out* at the start of every playout item since #350, so an unbounded read on this path is not new.
- A 240s QSV soak (`QsvExtraHardwareFrames=64`, 60 segment boundaries) **corroborates**: 1.043x sustained, zero `Cannot allocate memory` / `Could not open encoder`, RSS 166MB vs 156MB. It stayed largely caught-up, so it corroborates rather than demonstrates.
## Catchup does not replace the #350 burst
Measured time-to-first-segment: `-readrate` alone 3.71s, `+burst` **0.72s**, `+catchup` alone **3.65s**, both 0.67s. Orthogonal metrics — removing the burst would regress tune-in ~5x.
## Tests
`ErsatzTV.FFmpeg.Tests` 80/80. Four new tests; **every one mutation-checked**, each producing exactly its own expected red:
- `Realtime_Input_Should_Catch_Up_When_Option_Is_Supported`
- `Realtime_Input_Should_Not_Catch_Up_When_Option_Is_Unsupported`
- `Realtime_Input_Should_Not_Catch_Up_A_Still_Image`
- `Concat_Should_Never_Catch_Up`
- `Bitmap_Subtitle_Burn_In_Should_Catch_Up_On_The_Shared_Video_Input`
The regression test is built on a **bitmap** subtitle deliberately — a text subtitle is fetched by the libass filter outside the demuxer, so the same assertions would pass vacuously while the bug is fully present. It asserts on `[0:0][0:2]overlay`, which pins the mechanism: subtitle stream 2 resolving onto the video's input.
## Verification beyond unit tests
- Real capability parser verified against actual `ffmpeg -h long` output — extracts `readrate_catchup` exactly (the unit tests use a fake capabilities object, so this closes that gap).
- `scripts/e2e-local.sh` instance started cleanly, ffmpeg located, no new errors, stopped by exact PID.
- Local E2E cannot reproduce the *behavior* (no local media with embedded bitmap subs, no local QSV); the prod measurement above is the stronger evidence and is what the fix rests on.
## Review
Three independent cold review rounds, each in an isolated worktree. Rounds 1 and 2 returned MERGEABLE; round 3 returned BLOCKED on one finding — the retracted #529 claim had been fixed in the decision record but survived verbatim in the commit message — now corrected in both, mechanically verified.
## Docs
New decision record `ffmpeg.readrate-catchup-sparse-streams`; forward reference added to `ffmpeg.hls-cold-start-burst`; catalog regenerated.
## Corrections to the issue
- Affected population is **3,182 media versions (12.9%)**, not 6,369/25.8% — the earlier figure counted metadata records, not versions.
- 964 of those are bitmap-only, so **subtitle-selection preference could never have closed this**; the pipeline fix covers all 3,182.
- The `subtitleMode: null` side finding is a **non-bug** — the list DTO deliberately omits the field, `GET /api/v1/channels/{id}` returns it correctly, and `jq` renders an absent key as `null`.
## Filed separately
- **#733** — `dvbsub`/`dvb_subtitle` missing from both `IsImage` lists (109 streams misrouted to the text path)
- **#734**, **#735**, **#736** — UI/API follow-ups on exposing and explaining FFmpeg tunables
-readrate throttles an INPUT and paces it off whichever of its streams is
furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same
-i as the video -- SubtitleInputFile carries the video's path and ComplexFilter
resolves it to a stream specifier on that input, so CommandGenerator never emits
a second -i for it. Being sparse, that subtitle stream falls further behind every
second and drags video throughput down with it: measured 0.53x realtime against
the 1.0x a live client consumes at, which drains the client buffer until it
stalls. FFmpeg names the culprit itself at -loglevel warning:
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000
after a lag of 0.922s (then 1.24, 1.56, ... 3.80)
Add -readrate_catchup (6.0) to realtime video/audio inputs, capability-gated
through FFmpegKnownOption.HasOption exactly as -readrate_initial_burst is, so a
binary without it silently keeps today's behavior instead of failing to start.
The option first shipped in ffmpeg 8.0, which is NEWER than 7.1 -- hence runtime
detection rather than a version assumption. Still images and concat inputs are
excluded, mirroring #350.
Measured on prod (QSV, -threads 1, dvd_subtitle -> overlay), 45s steady-state
window after a 6s settle, replaying the captured production command line:
baseline 1.05 0.533x (x3 runs)
+ catchup 2.0 0.711x
+ catchup 6.0 1.067x (x2 runs)
+ catchup 20.0 1.067x
no subtitle overlay 1.067x (control)
Baseline reproduces the reported 0.53x and the control the reported 1.07x, so
the harness is validated on both sides. Reproduces on software libx264 too
(0.533x -> 1.067x), as expected for an input-pacing option. Raising the base
-readrate is not an alternative and was measured: 2.0 -> 0.62x, 3.0 -> 0.80x,
4.0 -> 0.80x, 6.0 -> 0.89x -- it asymptotes below realtime because the rate
ceiling was never the binding constraint.
On #529 (readrate was incidentally bounding QSV hardware-frame allocation): the
20.0-vs-6.0 row is why 6.0 was chosen, NOT evidence about allocation -- it is a
steady-state throughput number, not a count of frames in flight. Nor is the bound
safe because read rate is allocation-irrelevant: #529 measured that it is not (at
extra_hw_frames=0, 1.05 without a burst exits 0 while 1.05+burst hits ENOMEM).
Read rate changes how fast frames enter the graph, not how deep its queues are,
and #529's failure only appeared with NO pool headroom. The 64-frame floor now
guarantees headroom, so the load-bearing measurement is row 5 of that truth
table -- no -readrate at all with 64 frames -> 14 segments, exit 0 -- and a 6x
ceiling is strictly less aggressive than no throttle. Reinforcing it,
-readrate_initial_burst 8 has read flat out at the start of every playout item
since #350, so an unbounded read here is not new. A 240s QSV soak at
QsvExtraHardwareFrames=64 across 60 segment boundaries corroborates: 1.043x
sustained, zero "Cannot allocate memory" / "Could not open encoder", RSS 166MB
vs 156MB at baseline -- corroborates rather than demonstrates, since it stayed
largely caught up.
Catchup does NOT subsume the #350 burst; measured time-to-first-segment:
-readrate alone 3.71s, +burst 0.72s, +catchup alone 3.65s, both 0.67s. They fix
orthogonal metrics.
The regression test is built on a BITMAP subtitle deliberately -- a text subtitle
is fetched by the libass filter outside the demuxer, so the same assertions would
pass vacuously while the bug is fully present. It asserts on the "[0:0][0:2]overlay"
label, which is the mechanism: subtitle stream 2 resolving onto the video's input.
Every new test was mutation-checked, each producing exactly its own expected red.
Fixes#726
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`PR Gates / Script tests (pytest)` went red on
test_frontmatter_reader_matches_pyyaml_on_every_real_record. The rule: value was
an unquoted YAML plain scalar containing " #350's exclusion", and an unquoted
" #" starts a YAML comment -- PyYAML truncated the whole rule at "(mirroring",
so the catalog row and the mirrored MemPalace drawer carried half a sentence.
The hand-rolled frontmatter reader used by scripts/decisions_validate.py does NOT
tokenize comments, so it read the full line and reported OK; only the script-tests
job, which cross-checks the two parsers against every real record, can see this
class of defect. That is exactly what it exists for.
Fixed by single-quoting the scalar (doubling the internal apostrophe in
"image''s") and dropping the possessive from "#350's exclusion" so the token is
plain "#350". Verified both ways: PyYAML now returns the full sentence ending
"...race ahead.", and reverting the quoting reproduces the red, so the fix is
what makes the test pass rather than the test being insensitive.
Follow-up commit rather than an amend -- 56afa4652 is already pushed.
Refs #726
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
That commit message overstates the blast radius. It says:
PyYAML truncated the whole rule at "(mirroring", so the catalog row and the mirrored MemPalace drawer carried half a sentence.
The catalog row was not truncated. Verified against the broken commit itself:
$ git show 56afa4652:docs/decisions/README.md | grep -o 'readrate-catchup-sparse-streams`.*'
readrate-catchup-sparse-streams` | a realtime video/audio input also gets `-readrate_catchup` (6.0)
when the binary supports it — but NOT a still-image input (mirroring #350's exclusion) and NOT a
concat input, which keep at most bare `-readrate` (a still image's video input takes none at all).
Reason: `-readrate` pa...
build_decisions_catalog.py uses the same dependency-free hand parser as the validator, so the generated row was complete. The defect was latent — it would only have surfaced for a PyYAML-based consumer (the MemPalace export path), not in the checked-in catalog. The fix and its verification stand; only that one clause of the rationale was wrong.
Retracting it here rather than amending, since 56afa4652/57ad5efb3 are already pushed (process.pr-routine-sequence).
Why the local gate missed it
Worth recording, because the obvious conclusion is wrong. scripts/decisions_validate.pyon main does not cross-check against PyYAML:
$ git show origin/main:scripts/decisions_validate.py | grep -c pyyaml_frontmatter_faults
0
The pyyaml_frontmatter_faults() cross-check is in #725, which is still open. So a truncating record passes decisions_validate cleanly today, and only python3 -m pytest scripts/tests catches it — via test_frontmatter_reader_matches_pyyaml_on_every_real_record, which diffs the two parsers against every real record. I ran the validator and not the pytest suite, which is the actual process miss.
This is a decent argument for #725 landing: it moves this class of defect from "caught in CI after a push" to "caught by the local gate."
Verification of the fix
Not assumed — mutation-checked. Reverting the quoting reproduces the red:
FAILED scripts/tests/test_decisions_lib.py::test_frontmatter_reader_matches_pyyaml_on_every_real_record
mine ="...(mirroring #350's exclusion) and NOT a concat input... race ahead."
pyyaml='...(mirroring'
1 failed, 410 passed
With the fix: 411 passed, and PyYAML returns the full sentence ending ...race ahead.
## Correction to commit `57ad5efb3`'s message
That commit message overstates the blast radius. It says:
> PyYAML truncated the whole rule at "(mirroring", **so the catalog row and the mirrored MemPalace drawer carried half a sentence.**
**The catalog row was not truncated.** Verified against the broken commit itself:
```
$ git show 56afa4652:docs/decisions/README.md | grep -o 'readrate-catchup-sparse-streams`.*'
readrate-catchup-sparse-streams` | a realtime video/audio input also gets `-readrate_catchup` (6.0)
when the binary supports it — but NOT a still-image input (mirroring #350's exclusion) and NOT a
concat input, which keep at most bare `-readrate` (a still image's video input takes none at all).
Reason: `-readrate` pa...
```
`build_decisions_catalog.py` uses the same dependency-free hand parser as the validator, so the generated row was **complete**. The defect was **latent** — it would only have surfaced for a PyYAML-based consumer (the MemPalace export path), not in the checked-in catalog. The fix and its verification stand; only that one clause of the rationale was wrong.
Retracting it here rather than amending, since `56afa4652`/`57ad5efb3` are already pushed (`process.pr-routine-sequence`).
## Why the local gate missed it
Worth recording, because the obvious conclusion is wrong. `scripts/decisions_validate.py` **on `main`** does not cross-check against PyYAML:
```
$ git show origin/main:scripts/decisions_validate.py | grep -c pyyaml_frontmatter_faults
0
```
The `pyyaml_frontmatter_faults()` cross-check is in **#725, which is still open**. So a truncating record passes `decisions_validate` cleanly today, and only `python3 -m pytest scripts/tests` catches it — via `test_frontmatter_reader_matches_pyyaml_on_every_real_record`, which diffs the two parsers against every real record. I ran the validator and not the pytest suite, which is the actual process miss.
This is a decent argument for #725 landing: it moves this class of defect from "caught in CI after a push" to "caught by the local gate."
## Verification of the fix
Not assumed — mutation-checked. Reverting the quoting reproduces the red:
```
FAILED scripts/tests/test_decisions_lib.py::test_frontmatter_reader_matches_pyyaml_on_every_real_record
mine ="...(mirroring #350's exclusion) and NOT a concat input... race ahead."
pyyaml='...(mirroring'
1 failed, 410 passed
```
With the fix: **411 passed**, and PyYAML returns the full sentence ending `...race ahead.`
Three cold review rounds in isolated worktrees (R1 MERGEABLE, R2 MERGEABLE no code defects, R3 BLOCKED on a retracted #529 claim surviving in the commit message - now fixed in both artifacts and verified mechanically). All findings applied. Prod-measured 0.533x -> 1.067x on QSV and software libx264; 240s QSV soak clean. 80/80 tests, every new test mutation-checked. CI fully green on this head incl. a real 23m52s Build & test.
Review-verdict: MERGEABLE @ 57ad5ef
Three cold review rounds in isolated worktrees (R1 MERGEABLE, R2 MERGEABLE no code defects, R3 BLOCKED on a retracted #529 claim surviving in the commit message - now fixed in both artifacts and verified mechanically). All findings applied. Prod-measured 0.533x -> 1.067x on QSV and software libx264; 240s QSV soak clean. 80/80 tests, every new test mutation-checked. CI fully green on this head incl. a real 23m52s Build & test.
timothy
merged commit 772277e255 into main2026-08-04 23:24:12 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
fixes #726
What was wrong
-readratethrottles an input and paces it off whichever of its streams is furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same-ias the video —SubtitleInputFilecarries the video's path andComplexFilterresolves it to a stream specifier on that input, soCommandGeneratornever emits a second-ifor it. Being sparse, that subtitle stream falls further behind every second and drags video throughput down with it: 0.53x realtime against the 1.0x a live client consumes at, which drains the client buffer until the channel stalls.FFmpeg names the culprit itself at
-loglevel warning— the subtitle stream'sptsstays pinned while its lag grows monotonically:The fix
-readrate_catchup 6.0on realtime video/audio inputs, capability-gated throughFFmpegKnownOption.HasOptionexactly as-readrate_initial_burstis. Still-image and concat inputs are excluded, mirroring #350.The option first shipped in FFmpeg 8.0 (added 2025-02-15,
6232f416b) — newer than 7.1, which is precisely why it is detected at runtime rather than assumed from a version. Our pinned base image isersatztv-ffmpeg:8.1.2and both prod andersatztv-testrun it.Measurements (prod, QSV
h264_qsv,-threads 1, real production command line)Protocol per the issue: steady-state segment production, 45s window after a 6s settle, no
-tbound.-readrate_catchup 2.0-readrate_catchup 6.0-readrate_catchup 20.0Baseline reproduces the reported 0.53x exactly and the control the reported 1.07x exactly, so the harness is validated on both sides; variance across runs was zero. Also reproduces on software libx264 (0.533x → 1.067x), so this is not QSV-specific.
Raising the base
-readrateis not an alternative and was measured: 2.0→0.62x, 3.0→0.80x, 4.0→0.80x, 6.0→0.89x — it asymptotes below realtime.catchup 20.0matching6.0is why 6.0 was chosen: above the catch-up point the value is not a throughput dial.On #529 (the allocation hazard)
Stated carefully, because an earlier draft of this reasoning was wrong and a reviewer caught it:
extra_hw_frames=0,1.05without a burst exits 0 while1.05+burst hits ENOMEM).-readrateat all with 64 frames → 14 segments, exit 0. A 6x ceiling is strictly less aggressive than no throttle.-readrate_initial_burst 8has read flat out at the start of every playout item since #350, so an unbounded read on this path is not new.QsvExtraHardwareFrames=64, 60 segment boundaries) corroborates: 1.043x sustained, zeroCannot allocate memory/Could not open encoder, RSS 166MB vs 156MB. It stayed largely caught-up, so it corroborates rather than demonstrates.Catchup does not replace the #350 burst
Measured time-to-first-segment:
-readratealone 3.71s,+burst0.72s,+catchupalone 3.65s, both 0.67s. Orthogonal metrics — removing the burst would regress tune-in ~5x.Tests
ErsatzTV.FFmpeg.Tests80/80. Four new tests; every one mutation-checked, each producing exactly its own expected red:Realtime_Input_Should_Catch_Up_When_Option_Is_SupportedRealtime_Input_Should_Not_Catch_Up_When_Option_Is_UnsupportedRealtime_Input_Should_Not_Catch_Up_A_Still_ImageConcat_Should_Never_Catch_UpBitmap_Subtitle_Burn_In_Should_Catch_Up_On_The_Shared_Video_InputThe regression test is built on a bitmap subtitle deliberately — a text subtitle is fetched by the libass filter outside the demuxer, so the same assertions would pass vacuously while the bug is fully present. It asserts on
[0:0][0:2]overlay, which pins the mechanism: subtitle stream 2 resolving onto the video's input.Verification beyond unit tests
ffmpeg -h longoutput — extractsreadrate_catchupexactly (the unit tests use a fake capabilities object, so this closes that gap).scripts/e2e-local.shinstance started cleanly, ffmpeg located, no new errors, stopped by exact PID.Review
Three independent cold review rounds, each in an isolated worktree. Rounds 1 and 2 returned MERGEABLE; round 3 returned BLOCKED on one finding — the retracted #529 claim had been fixed in the decision record but survived verbatim in the commit message — now corrected in both, mechanically verified.
Docs
New decision record
ffmpeg.readrate-catchup-sparse-streams; forward reference added toffmpeg.hls-cold-start-burst; catalog regenerated.Corrections to the issue
subtitleMode: nullside finding is a non-bug — the list DTO deliberately omits the field,GET /api/v1/channels/{id}returns it correctly, andjqrenders an absent key asnull.Filed separately
dvbsub/dvb_subtitlemissing from bothIsImagelists (109 streams misrouted to the text path)-readrate throttles an INPUT and paces it off whichever of its streams is furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same -i as the video -- SubtitleInputFile carries the video's path and ComplexFilter resolves it to a stream specifier on that input, so CommandGenerator never emits a second -i for it. Being sparse, that subtitle stream falls further behind every second and drags video throughput down with it: measured 0.53x realtime against the 1.0x a live client consumes at, which drains the client buffer until it stalls. FFmpeg names the culprit itself at -loglevel warning: [sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000 after a lag of 0.922s (then 1.24, 1.56, ... 3.80) Add -readrate_catchup (6.0) to realtime video/audio inputs, capability-gated through FFmpegKnownOption.HasOption exactly as -readrate_initial_burst is, so a binary without it silently keeps today's behavior instead of failing to start. The option first shipped in ffmpeg 8.0, which is NEWER than 7.1 -- hence runtime detection rather than a version assumption. Still images and concat inputs are excluded, mirroring #350. Measured on prod (QSV, -threads 1, dvd_subtitle -> overlay), 45s steady-state window after a 6s settle, replaying the captured production command line: baseline 1.05 0.533x (x3 runs) + catchup 2.0 0.711x + catchup 6.0 1.067x (x2 runs) + catchup 20.0 1.067x no subtitle overlay 1.067x (control) Baseline reproduces the reported 0.53x and the control the reported 1.07x, so the harness is validated on both sides. Reproduces on software libx264 too (0.533x -> 1.067x), as expected for an input-pacing option. Raising the base -readrate is not an alternative and was measured: 2.0 -> 0.62x, 3.0 -> 0.80x, 4.0 -> 0.80x, 6.0 -> 0.89x -- it asymptotes below realtime because the rate ceiling was never the binding constraint. On #529 (readrate was incidentally bounding QSV hardware-frame allocation): the 20.0-vs-6.0 row is why 6.0 was chosen, NOT evidence about allocation -- it is a steady-state throughput number, not a count of frames in flight. Nor is the bound safe because read rate is allocation-irrelevant: #529 measured that it is not (at extra_hw_frames=0, 1.05 without a burst exits 0 while 1.05+burst hits ENOMEM). Read rate changes how fast frames enter the graph, not how deep its queues are, and #529's failure only appeared with NO pool headroom. The 64-frame floor now guarantees headroom, so the load-bearing measurement is row 5 of that truth table -- no -readrate at all with 64 frames -> 14 segments, exit 0 -- and a 6x ceiling is strictly less aggressive than no throttle. Reinforcing it, -readrate_initial_burst 8 has read flat out at the start of every playout item since #350, so an unbounded read here is not new. A 240s QSV soak at QsvExtraHardwareFrames=64 across 60 segment boundaries corroborates: 1.043x sustained, zero "Cannot allocate memory" / "Could not open encoder", RSS 166MB vs 156MB at baseline -- corroborates rather than demonstrates, since it stayed largely caught up. Catchup does NOT subsume the #350 burst; measured time-to-first-segment: -readrate alone 3.71s, +burst 0.72s, +catchup alone 3.65s, both 0.67s. They fix orthogonal metrics. The regression test is built on a BITMAP subtitle deliberately -- a text subtitle is fetched by the libass filter outside the demuxer, so the same assertions would pass vacuously while the bug is fully present. It asserts on the "[0:0][0:2]overlay" label, which is the mechanism: subtitle stream 2 resolving onto the video's input. Every new test was mutation-checked, each producing exactly its own expected red. Fixes #726 Decisions-Edit: yes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Correction to commit
57ad5efb3's messageThat commit message overstates the blast radius. It says:
The catalog row was not truncated. Verified against the broken commit itself:
build_decisions_catalog.pyuses the same dependency-free hand parser as the validator, so the generated row was complete. The defect was latent — it would only have surfaced for a PyYAML-based consumer (the MemPalace export path), not in the checked-in catalog. The fix and its verification stand; only that one clause of the rationale was wrong.Retracting it here rather than amending, since
56afa4652/57ad5efb3are already pushed (process.pr-routine-sequence).Why the local gate missed it
Worth recording, because the obvious conclusion is wrong.
scripts/decisions_validate.pyonmaindoes not cross-check against PyYAML:The
pyyaml_frontmatter_faults()cross-check is in #725, which is still open. So a truncating record passesdecisions_validatecleanly today, and onlypython3 -m pytest scripts/testscatches it — viatest_frontmatter_reader_matches_pyyaml_on_every_real_record, which diffs the two parsers against every real record. I ran the validator and not the pytest suite, which is the actual process miss.This is a decent argument for #725 landing: it moves this class of defect from "caught in CI after a push" to "caught by the local gate."
Verification of the fix
Not assumed — mutation-checked. Reverting the quoting reproduces the red:
With the fix: 411 passed, and PyYAML returns the full sentence ending
...race ahead.Review-verdict: MERGEABLE @
57ad5efThree cold review rounds in isolated worktrees (R1 MERGEABLE, R2 MERGEABLE no code defects, R3 BLOCKED on a retracted #529 claim surviving in the commit message - now fixed in both artifacts and verified mechanically). All findings applied. Prod-measured 0.533x -> 1.067x on QSV and software libx264; 240s QSV soak clean. 80/80 tests, every new test mutation-checked. CI fully green on this head incl. a real 23m52s Build & test.