-readrate throttles an INPUT and paces it off whichever of its streams is
furthest behind. An embedded bitmap subtitle (PGS/DVD) is read through the same
-i as the video -- SubtitleInputFile carries the video's path and ComplexFilter
resolves it to a stream specifier on that input, so CommandGenerator never emits
a second -i for it. Being sparse, that subtitle stream falls further behind every
second and drags video throughput down with it: measured 0.53x realtime against
the 1.0x a live client consumes at, which drains the client buffer until it
stalls. FFmpeg names the culprit itself at -loglevel warning:
[sist#0:3/dvd_subtitle] Resumed reading at pts 10.400 with rate 6.000
after a lag of 0.922s (then 1.24, 1.56, ... 3.80)
Add -readrate_catchup (6.0) to realtime video/audio inputs, capability-gated
through FFmpegKnownOption.HasOption exactly as -readrate_initial_burst is, so a
binary without it silently keeps today's behavior instead of failing to start.
The option first shipped in ffmpeg 8.0, which is NEWER than 7.1 -- hence runtime
detection rather than a version assumption. Still images and concat inputs are
excluded, mirroring #350.
Measured on prod (QSV, -threads 1, dvd_subtitle -> overlay), 45s steady-state
window after a 6s settle, replaying the captured production command line:
baseline 1.05 0.533x (x3 runs)
+ catchup 2.0 0.711x
+ catchup 6.0 1.067x (x2 runs)
+ catchup 20.0 1.067x
no subtitle overlay 1.067x (control)
Baseline reproduces the reported 0.53x and the control the reported 1.07x, so
the harness is validated on both sides. Reproduces on software libx264 too
(0.533x -> 1.067x), as expected for an input-pacing option. Raising the base
-readrate is not an alternative and was measured: 2.0 -> 0.62x, 3.0 -> 0.80x,
4.0 -> 0.80x, 6.0 -> 0.89x -- it asymptotes below realtime because the rate
ceiling was never the binding constraint.
On #529 (readrate was incidentally bounding QSV hardware-frame allocation): the
20.0-vs-6.0 row is why 6.0 was chosen, NOT evidence about allocation -- it is a
steady-state throughput number, not a count of frames in flight. Nor is the bound
safe because read rate is allocation-irrelevant: #529 measured that it is not (at
extra_hw_frames=0, 1.05 without a burst exits 0 while 1.05+burst hits ENOMEM).
Read rate changes how fast frames enter the graph, not how deep its queues are,
and #529's failure only appeared with NO pool headroom. The 64-frame floor now
guarantees headroom, so the load-bearing measurement is row 5 of that truth
table -- no -readrate at all with 64 frames -> 14 segments, exit 0 -- and a 6x
ceiling is strictly less aggressive than no throttle. Reinforcing it,
-readrate_initial_burst 8 has read flat out at the start of every playout item
since #350, so an unbounded read here is not new. A 240s QSV soak at
QsvExtraHardwareFrames=64 across 60 segment boundaries corroborates: 1.043x
sustained, zero "Cannot allocate memory" / "Could not open encoder", RSS 166MB
vs 156MB at baseline -- corroborates rather than demonstrates, since it stayed
largely caught up.
Catchup does NOT subsume the #350 burst; measured time-to-first-segment:
-readrate alone 3.71s, +burst 0.72s, +catchup alone 3.65s, both 0.67s. They fix
orthogonal metrics.
The regression test is built on a BITMAP subtitle deliberately -- a text subtitle
is fetched by the libass filter outside the demuxer, so the same assertions would
pass vacuously while the bug is fully present. It asserts on the "[0:0][0:2]overlay"
label, which is the mechanism: subtitle stream 2 resolving onto the video's input.
Every new test was mutation-checked, each producing exactly its own expected red.
Fixes#726
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Independent cold review (Codex, no implementation role) found no Blocker and
three real defects, all fixed here:
HIGH — HDR was re-tagged bt709 only when the profile had NormalizeColors on.
The colorspace filter sat behind desiredState.ColorsAreBt709, so an operator
with normalization disabled got tonemapped SDR pixels still tagged bt2020 and
the player converted them a second time. The guard is now
"tonemapped || (ColorsAreBt709 && ...)". Deliberately NOT fixed by hoisting
usesVppQsv out of the guard, which would force bt709 on scale-only non-HDR
chains that legitimately opted out.
MEDIUM — image subtitles stopped being resized. The subtitle canvas is scaled
only when the video chain contains a recognized scale filter, and that predicate
listed the QSV filters only; swapping ScaleQsvFilter for ScaleVaapiFilter left a
4K HDR + PGS source with a 720p video and a source-size subtitle overlay.
VaapiPipelineBuilder already listed ScaleVaapiFilter; QsvPipelineBuilder does now.
MEDIUM — anamorphic HDR now falls back to the software tonemap. ScaleQsvFilter
is handed the SAR VideoStream calculates (it has a 0:0 fallback); ScaleVaapiFilter
multiplies by ffmpeg's runtime `sar`, which differs when the decoded frame leaves
SAR unspecified. Rather than ship a graph nobody has run, exclude anamorphic --
which leaves those sources exactly where they were before this change.
LOW — tests now pin the exact validated graph as an ordered substring (the prior
assertions would have passed with setFormat off, hwdownload dropped, or the wrong
tonemap output format), assert against the vpp_qsv OPTION rather than a bare
"tonemap=1" substring, and cover NormalizeColors=false, anamorphic and image
subtitles. Each of the three fixes was negative-controlled: reverting it fails
exactly one test, and no others.
The remaining LOW (deriveDevice's defaulted bool is a future-call-site trap) is
recorded as an accepted residual rather than fixed, since a named factory would
push this diff into the VA-API pipeline for no behavior change.
The record is 86 prose lines, over the 60-line ceiling. Declining to cut: every
bullet is a distinct measured finding, which docs.corpus-size-signal names as a
legitimate decline.
Decisions-Edit: yes
UseOpenClTonemap excluded the two known ways frames reach a hardware surface
before the tonemap (the QSV decoder, deinterlace_qsv). That is only correct for
as long as the enumeration stays complete -- and the failure mode if it ever
stops being complete is a second hwupload stacked on frames that are already on
a surface.
Check currentState.FrameDataLocation directly instead. The route begins with
hwupload, so "frames are in software" is the actual precondition; the specific
exclusions stay as documentation of the known cases, but the state check is what
makes it safe. A future filter landing ahead of the tonemap now degrades to the
software tonemap rather than emitting a broken graph.
#505 asked to route the #498 native-decode path through TonemapQsvFilter to
move HDR tonemapping off the CPU. Measured on the Intel host (jazz: FFmpeg
8.1.2 / iHD 25.1.4 / UHD 630) against real HDR HEVC Main10, that filter is a
SILENT no-op: a graph ending in vpp_qsv=tonemap=1 returns a frame byte-identical
(same md5) to the same graph with no tonemap at all, with no warning and no
error. QSV VPP tonemapping needs Gen11+; pre-Gen11 iHD ignores it.
So the issue's premise was inverted, and the branch it wanted to extend was
already broken: the existing DecoderHardwareAccelerationMode == Qsv path shipped
untonemapped HDR whenever QsvPreferNativeDecoder was off -- which is exactly the
escape hatch #498/#523 recommend. Prod was unaffected (native-decode is the
default and took the working software branch).
Tonemap on the GPU via OpenCL instead, the route VaapiPipelineBuilder already
uses and the one Jellyfin uses. The scale has to run first, in scale_vaapi:
tonemapping full-size is slower than the software path it replaces (15.5s wall
for 12.5s of content, below realtime), while scale-first cuts total CPU ~60%
(35.6s -> 14.1s) and lands at the no-tonemap wall-clock floor. A QSV surface
maps to neither OpenCL nor VA-API, so the gate requires software frames: the
QSV decoder and deinterlace_qsv both fall back to the software tonemap, slower
but correct. TonemapQsvFilter is deleted -- a filter that silently does nothing
is worse than no filter.
Also fixes output tagging: the first end-to-end run was correctly tonemapped yet
still announced bt2020 primaries, because SetPixelFormat's usesVppQsv predicate
("did a hardware filter strip color info") listed only the QSV filters. Both new
filters are now in it.
Validated end to end on jazz with the exact generated command: exit 0,
YAVG 26.39 (software reference 26.6, untonemapped 44.3), and ffprobe reports
bt709 space/transfer/primaries.
fixes#505
OverlayWatermarkFilter.Position had no switch arm for MiddleCenter, so it
silently fell into the BottomRight default and rendered bottom-right.
OverlayWatermarkCudaFilter and OverlayWatermarkQsvFilter both inherit this
Position property without overriding it, so they were affected too (the
whole ErsatzTV.FFmpeg project has only this one WatermarkLocation switch).
- Add an explicit MiddleCenter arm: x=(W-w)/2:y=(H-h)/2
- Make BottomRight an explicit arm instead of the fallthrough default
- The default case now logs a warning (not throw - this runs on the
playback hot path constructing ffmpeg args, and sibling filters in this
project already use safe string fallbacks rather than throwing for an
unmapped enum) and falls back to the BottomRight position
- Strip the pre-existing UTF-8 BOM from OverlayWatermarkFilter.cs (#311
formatting gate: touching a legacy BOM'd file makes removing it ours)
- Add ErsatzTV.FFmpeg.Tests/Filter/OverlayWatermarkFilterTests.cs asserting
the exact position expression for every WatermarkLocation value across
the software, CUDA, and QSV overlay filters, plus the unmapped-value
fallback
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Second review returned MERGEABLE with one Medium and three Lows. Addressed all four:
- Medium: the save-time normalization had zero test coverage, so a later refactor
dropping Math.Max would leave the suite green (the FFmpegState floor keeps the
pipeline correct, hiding the regression until someone reads a stored 0 back).
Added Create/Update_Should_Floor_QsvExtraHardwareFrames over 0, -8, 63, 64 and 128,
plus Create_Should_Leave_Null_QsvExtraHardwareFrames_Null for the null-passthrough
branch, following the existing QsvPreferNativeDecoder tests' seed/handle/re-read
shape. Negative-controlled: reverting both handlers fails exactly 5.
- Low: the SPA `min` was cosmetic. Input does forward it to the DOM, but there is no
<form> — save is an onClick gated only on validate(), which had no branch for this
field, so a typed 10 submitted fine and was silently changed to 64 with a 200 and no
message. validate() now rejects it client-side.
- Low: the warning fires at the top of SetAccelState, before we know whether the
pipeline uploads at all, so a fully-hardware path could be told "using 64 instead"
when nothing consumed either value. Reworded to "will use ... wherever frames are
uploaded".
- Low: recorded in the decision entry that the save-time normalization is
unconditional on hardwareAcceleration (a non-QSV profile's stored value moves too),
and that a client PUTting 0 reads back 64 — a transform the OpenAPI description does
not advertise.
Verified in production, not just asserted. Set prod's profile to 64 (operator-approved)
and drove the exposed pipeline myself via the troubleshooting playback API on an mpeg4
.avi, which forces software decode + hwupload:
hwupload=extra_hw_frames=64,vpp_qsv=w=1875:h=1080 exit 0, speed 12.0x, 0 ENOMEM
Then the negative control on prod's own hardware, same command, only the pool differing:
extra_hw_frames=64 -> exit 0, 8 segments, 0 ENOMEM
extra_hw_frames=0 -> exit 244, 0 segments, 3 ENOMEM
which reproduces the six overnight production failures and confirms the fix.
Full suite green: 4095 .NET, 891 web.
Refs #350, #516, #519.
Cold adversarial review returned BLOCKED on the documentation half. Addressed:
- The new decision record carried no lifecycle metadata block, taking the repo from
82/82 to 83/82 and making it invisible to the by-key catalog lookup that #521
established the same day. Added key/status/since/supersedes/superseded-by
(ffmpeg.qsv-extra-hw-frames-floor) and regenerated docs/decisions/README.md;
decisions_validate.py now reports OK with no legacy-unmigrated notice.
- The entry claimed to correct the #350 record but left that record untouched, so the
stale "the burst is bounded" claim stayed authoritative for anyone resolving
ffmpeg.hls-cold-start-burst. Added a forward-pointing correction note there (hence
the [decisions-edit] token on this commit).
- The floor was applied silently. QsvPipelineBuilder.SetAccelState now logs a warning
naming both the configured and applied value, because raising a deliberately small
pool costs additional surfaces (64 NV12 1080p surfaces is roughly 190 MiB, 760 MiB
at 4K) on memory-constrained iGPUs.
- Narrowed an overstated claim in the entry: 1..63 are untested, not known-bad. We
raise them because the risk is a channel serving nothing, not because asking for
less is illegitimate. Recorded as a deliberate over-reach with a stated cost.
- Corrected a factual error: SubtitleScaleQsvFilter also formats extra_hw_frames but
is dead code with no construction site, so it is NOT covered by the guard.
- Config-vs-behavior mismatch: Create/UpdateFFmpegProfileHandler now normalize on
save so stored rows converge on what the pipeline runs, and the SPA field carries
min=64 rather than defaulting the display to 0. Render-time flooring is kept as the
net that fixes existing deployments with no migration; the remaining gap for
un-resaved rows is recorded as an accepted residual.
- Tests strengthened: pinned to the literal measured 64 rather than to the constant
(so lowering the floor cannot quietly satisfy them), added a negative-value case,
added a deinterlace-upload case, and replaced the narrow ShouldNotContain with a
regex asserting EVERY extra_hw_frames occurrence in the command is >= the minimum.
Negative control re-run against the strengthened tests: reverting the floor fails 5,
with the build verified succeeded first. Full suite green (4086 .NET, 891 web).
Review finding that needed no change: the "single point" claim was independently
verified — no bypass exists, every FFmpegState construction routes through
MaybeQsvExtraHardwareFrames.
Refs #350, #516, #519.
A stored qsvExtraHardwareFrames of 0 reached FFmpeg as hwupload=extra_hw_frames=0,
leaving the QSV upload pool no headroom for frames in flight through the filter graph.
Any input that is not throttled then exhausts it: the graph fails with -12 (Cannot
allocate memory), h264_qsv reports "Could not open encoder before EOF", and zero
segments are written.
Measured against the deployed FFmpeg 8.1.2, one real logged command, only the marked
tokens differing:
readrate 1.05, no burst + frames 0 -> exit 0, 14 segments
readrate 1.05 + burst 2/4/8 + frames 0 -> exit 244, ENOMEM, 0 segments
no readrate at all + frames 0 -> exit 244, ENOMEM, 0 segments
readrate 1.05 + burst 8 + frames 64 -> exit 0, 14 segments
no readrate at all + frames 64 -> exit 0, 14 segments
So the defect predates #350's cold-start burst: a work-ahead start takes no -readrate
and was already failing on a profile with 0. The burst removed the throttle on every
realtime session, turning an intermittent failure into a near-deterministic one, which
is how it surfaced. Input throttling was doing load-bearing allocation-bounding work
that nobody had written down -- which is why the FFmpeg-level benchmark in #350 and the
argument-generation tests in #516 were both green and neither could see it.
Fixed at FFmpegState.QsvExtraHardwareFrames, the single point every QSV upload site
reads, so one guard covers HardwareUploadFilter, HardwareUploadQsvFilter,
WatermarkHardwareUploadFilter, ScaleQsvFilter, DeinterlaceQsvFilter and
SubtitleScaleQsvFilter rather than six call sites that can drift apart. Values above
the floor are still honored.
Tests negative-controlled: reverting the floor fails exactly the three sub-minimum
cases, with the build verified succeeded first.
Refs #350, #516, #519.
`-readrate 1.05` paces input reading at wall clock so a channel behaves like
live TV, but it applies from the very first read. With 4s HLS segments and the
segmenter waiting for the first one, the playlist could not appear sooner than
~4/1.05 = 3.8s, so every tune-in that did not win a work-ahead slot paid a
multi-second wait.
Add `-readrate_initial_burst` (FFmpeg >= 6.1) next to `-readrate` on the normal
playback path, gated on runtime capability detection via the existing
`FFmpegKnownOption`/`HasOption` machinery, whose option list had simply been
empty. Measured on real prod media: time-to-first-playlist 5369/5344ms ->
648/649ms.
Root cause detail: the cold-start bimodality earlier rounds could not explain
was never about the media. `HlsSessionWorker` grants an unthrottled start only
while `_workAheadCount < work_ahead_limit` (prod: 1), so concurrent tune-ins
fall back to the throttled path. Confirmed on prod with three concurrent tunes:
firstGop 866ms for the slot winner vs 3845ms and 6357ms. This also falsifies the
issue's ranked #1 driver — accurate-seek decode-discard measures 30-100ms on
real media, and probe caps 20-50ms; neither can account for seconds.
Still images are excluded: their video input is paced by the realtime filter and
takes no readrate, so a burst would only run a song's separate audio input ahead
of the video. Concat/WrapSegmenter keep the unburst single-arg constructor.
fixes#350
From the Fable whole-branch review:
- M1: guard native VA-API decode with !OperatingSystem.IsWindows() (no vaapi
hwaccel on Windows; QSV caps over-reported there) — DRY'd into a preferNativeDecode local.
- L2: IsIntelVaapiOrQsv also matches decoder mode Vaapi, preserving Intel
audio-dup parity on the (producerless) Nut-output branch.
- L1: replace the vacuous ShouldNotContain(" deinterlace_qsv") with an
occurrence-count assertion that actually catches a second bare occurrence.
- N1/H1: decisions.md — correct the column to nullable-with-default (not NOT NULL),
and record the accepted HDR software-tonemap trade-off + Linux-only guard, with
the tonemap_qsv optimization tracked in #505.
H1 (HDR tonemap reroute) accepted-and-deferred per that decision; #505 filed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Whitespace-only (git diff -w is empty); the #311 format job checks whole
touched files, and these two legacy files carried pre-existing violations
never caught before (no PR had touched them since the gate landed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Centralize SDK and threading analyzers, baseline the .NET 10 All rule inventory at suggestion severity, and promote S3981 repo-wide. Fix the always-true worker count predicate and cover the idle/active branches.
Fixes#15
Co-Authored-By: Codex <codex@openai.com>
Move all 154 PackageReference versions (96 distinct packages) out of the 14
project files into a single central Directory.Packages.props with
ManagePackageVersionsCentrally=true. No version changes — every package was
already pinned identically across projects (no conflicts detected), so this is a
pure relocation: updates become one-line and cross-project version drift is
structurally impossible.
Also copy Directory.Packages.props into the Docker image build before restore:
with CPM the csproj carry no versions, so the image's `dotnet restore` fails
without the central manifest (verified: NU1015 across every project).
Restore + Release build verified locally, plus a simulation of the image's
restore layer under linux-x64 (0 errors; only the pre-existing
NCalcSync/SQLitePCLRaw advisories remain, demoted to warnings, tracked in #8).
Part of #14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* add new fields to database
* update editor
* audio and video normalization settings appear to work
* implement optional color normalization
* fix transcoding tests
* update changelog
* fix nvenc playback when color metadata changes mid-stream
* update dependencies (needed to fix unit test runner)
* limit noautoscale to when it's not already present
* improve build time by only running analyzers explicitly
* don't exclude scanner from analyzers
* Revert "don't exclude scanner from analyzers"
This reverts commit d927f9850a.
* fix sed syntax for linux
