Three Low findings from the #536 clamp re-review, unreachable today (one
guarded release site) but filed against the day a second release site is added.
- §1: Release() now reads the count and CAS-decrements only when current > 0,
so it never publishes a negative count even transiently. The prior
decrement-first-then-clamp shape dipped to -1, which a concurrent TryAcquire
could read as phantom room and over-admit at the limit (re-opening the #529
QSV pool exhaustion). It records the unbalanced release synchronously on the
offending thread rather than blaming a later innocent release.
- §3: Release() returns bool; HlsSessionWorker logs a warning on the false
(unbalanced) return — the one in-band signal a future second release site
would need. WorkAheadSlots stays logger-free by design.
- §2: UnbalancedReleases doc-comment corrected — it can under-count (an
over-release while count > 0 cancels a coexisting leak and goes unrecorded);
no false positives, but zero does not prove correctness.
Test: Release_Unbalanced_NeverPublishesNegativeCount (2M unbalanced releases vs
4 count-samplers) with a documented, verified negative control (reverting to
the decrement-first body makes readers observe the transient -1).
Adds a decisions.md entry (ffmpeg.work-ahead-slot-release-never-negative).
fixes#539
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-push adversarial review, Low finding. The pool is process-wide and
lives for the life of the app, so a `Release()` not matched by a
successful `TryAcquire` would drive the count to -1 permanently: with a
limit of 1 that silently admits two unthrottled transcodes forever, which
is exactly the #529 QSV pool exhaustion with nothing in the logs to find
it by. Clamp at zero and record the breakage in `UnbalancedReleases`
rather than throwing — the sole caller releases from a `finally`, where a
throw would swallow the real exception.
The hammer tests now also assert `UnbalancedReleases == 0`, so the clamp
cannot mask drift it was added to survive.
Also moves the #536 index line to the end of the in-file decisions index
(it was inserted in the 2026-07-11 block while its body appends at the
end) — review nit, anchors were already correct.
refs #536
The slot check and its increment straddled an await: `Run` compared
`Volatile.Read(ref _workAheadCount)` against a DB-backed limit, and the
increment happened later inside `Transcode`. Every simultaneous tune-in
therefore observed `0 < limit` and started unthrottled — three concurrent
tunes on prod with a limit of 1 all ran with no `-readrate`. `Interlocked`
on the write side alone buys nothing when the read side is a separate,
earlier load (same class as #231/#250).
Extract the counter into a `WorkAheadSlots` pool whose `TryAcquire(limit)`
claims via compare-exchange, so the count never even transiently exceeds
the limit that the QSV hardware-frame pool sizing (#529) is derived from.
`Run` claims the slot and passes ownership in; `Transcode(bool
ownsWorkAheadSlot, ...)` derives `realtime` from it and releases it in its
existing `finally`, keeping acquire/release one-for-one. Acquisition stays
in the caller because `Run` sets `_state` from the outcome and `Transcode`
reads that state on entry to pick the item start time.
Tests hammer 8 threads x 20k rounds (a single Barrier round does not
collide on this hardware); the documented negative control reinstates the
check-then-act body and produces 15912 over-claiming rounds of 20000.
Also annotates the #350 decision record, whose "every concurrent tune-in
falls back to the throttled path" bullet described the intent rather than
the behaviour.
fixes#536
From the cold adversarial review of the initial diff. No blockers were
found; these address what the numbers MEAN, which is the whole point of
an instrumentation change.
- The buckets span the worker's Run entry, not the startup stopwatch, so
prep overlaps the tail of `setup`. Rather than let the log imply an
invariant it does not satisfy, say "spans runEntry" in the line, spell
it out in the doc comment, and rename the test that had codified the
false `sum == startup` claim.
- Guard `processLaunched > playlistExists` -> Unavailable: a stale
live.m3u8 from a previous session (Run warns about a non-empty
transcode folder but does not delete it) would otherwise yield a
plausible-looking sample whose prep exceeds the measured phase.
- Split the two-way fallback into TwoWay vs TwoWayLateProgress. They are
different stories about the pipeline and discriminating stories is
what this issue is for.
- Document the 100ms playlist-poll quantization (it lands entirely in
firstGop, the smallest bucket) and the first-process-failed case where
ffmpegInit spans a retry.
- Short-circuit the per-line timestamp call; static readonly Unavailable.
- Tests for the new guard, progress-before-launch, and boundary equality
(so tightening >= to > later cannot pass silently).
#350's measurement showed `startup` is 81% of tune-in latency and carries
100% of its variance, while remaining one opaque bucket spanning FFmpeg
spawn -> input open/probe -> encoder init -> first GOP. Two hypotheses
survive that measurement (NFS input open vs VAAPI init under contention)
and they need opposite fixes, so split before optimizing.
Adds `prep` (ErsatzTV-side work before FFmpeg exists) + `ffmpegInit`
(launch -> first `-progress` output) + `firstGop` (-> live.m3u8 exists)
to the existing Information-level cold-start line.
The pipeline runs `-loglevel error -nostats -hide_banner`, so a healthy
FFmpeg writes nothing to stderr; the `-progress` stream is the only
zero-cost milestone available and `ffmpegInit` therefore still lumps
input-open with encoder-init. That limit is documented rather than
papered over, and the split degrades to the two-way form #472 accepts
when no progress arrives before the playlist.
Log-only: no transcode behavior change, no new endpoint or config knob.
fixes#472
Third review pass: MERGEABLE WITH NITS. Taking the one finding it asked
for before merge, plus a doc nit.
The cancellation filter added last commit had no token check, and it spans
the whole Transcode body -- including every mediator send (ffprobe via
CliWrap, media-server API calls, subtitle extraction, song-video
generation). TaskCanceledException is also what HttpClient throws on its
OWN timeout, so a real timeout in any of those was being downgraded from
an ERROR with a stack trace to a routine "Terminating HLS session"
Information line. Behaviour was unchanged (both arms return false) but the
fault signal was lost, and this repo has been bitten before by
"empty log != the event didn't happen". Now filters on
cancellationToken.IsCancellationRequested, so only genuine caller
cancellation is treated as a graceful teardown.
Doc nit: the <exception> block said cancellation "is thrown"; it is only
thrown when the token trips while the probe is in flight -- cancelling
after it completes returns normally. Now says "may propagate".
Declined the reviewer's optional suggestion to drain until a 0-return
instead of reading exactly one byte: reading exactly one byte is what
makes the guard safe BY CONSTRUCTION, since a server or proxy that
answers 206 with a wider range than requested still cannot be drained
unboundedly. 206-only was confirmed correct rather than extended to short
200s, since deciding "short" from Content-Length would reopen the
unbounded path for a chunked or Content-Length-less response.
Also records the operator's standing rule in the handoff lore: a lone
`decisions.md append-only` red is a known infra flake -- do not
investigate, rebase, amend or push to clear it; the operator reruns that
job from the UI. I violated this earlier in this PR with a
tidy-but-wrong "my entry is no longer at EOF" theory, and the rebase did
not fix it -- the job went red again on a verified pure-append diff, which
is the proof the red was never about the diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second review pass returned BLOCKED on two findings introduced by the
first fix commit. Both were right.
BLOCKER 1 — the drain added for "return the connection to the pool" was
unbounded. `response.Content.ReadAsByteArrayAsync()` buffers the WHOLE
body, and it ran for every non-404 response. A server that ignores
`Range: bytes=0-0` answers 200 with the entire file, so this would
download at line rate into a byte[] on the streaming hot path for up to
the 2s timeout -- strictly worse than the aborted socket it replaced, and
it defeated the ResponseHeadersRead the probe deliberately uses. Now the
single byte is read only on 206 (where the server honoured the range and
the body really is one byte); any other status aborts the socket, which
is much the cheaper evil. Two tests pin both directions; verified
non-vacuous (restoring the unbounded drain fails the 200-with-body test).
BLOCKER 2 — IRemoteStreamProber's doc-comment still described pre-fix
behaviour. I had told the reviewer it was updated; it was not -- only the
implementation's <remarks> had been. It claimed `false` on any 404 (now
only a redirected one) and that every other outcome returns `true` (caller
cancellation throws). Both clauses corrected, and the throwing contract is
now documented with <exception>.
Also fixed the reviewer's own follow-on finding: the cancellation rethrow
it asked for reached HlsSessionWorker's catch-all, which logs a
channel-level ERROR with a stack trace. The graceful
TaskCanceledException/OperationCanceledException handler at :662 wraps only
the inner ffmpeg block, not the mediator sends, so every client disconnect
on a remote-streaming channel would have produced a spurious ERROR -- in
exactly the logs a #350 cold-start investigation reads. Added a
cancellation filter on the outer try that logs Information instead.
Nit: stale SeedAll doc-comment now mentions the emby case.
Deferred, per reviewer's explicit agreement: Plex-branch handler coverage
(follow-up), and HEAD-with-GET-fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tuning a channel intermittently hard-failed with ffmpeg exit 8 and
`Server returned 404 Not Found` on /media/jellyfin/{itemId}.
Root cause: ValidatePlayoutItemPath checked `File.Exists` on the local
branch, but the three media-server remote-stream branches returned
`http://localhost:{port}/media/{plex,jellyfin,emby}/{id}` unconditionally.
When the media was gone from the media server too, validation "succeeded"
and ffmpeg was launched against a URL that 404s.
That bypassed the good error path the handler already had
(PlayoutItemDoesNotExistOnDisk renders an error card sized to run until
the NEXT playout item, so the dead item is skipped) and instead landed in
HlsSessionWorker's generic ffmpeg-failure path, which sizes its error card
to the failed 44s work-ahead chunk and then re-selects the SAME broken
item -- a repeating error card for the item's whole slot (~22 min).
Restore the method's own invariant: every PlayoutItemWithPath it returns
has been checked for existence. A definitive 404 now returns the new
PlayoutItemNotAvailableFromMediaServer error, handled in the same switch
arm as PlayoutItemDoesNotExistOnDisk.
The probe is deliberately fail-open: only a 404 reports the media gone.
A timeout, 5xx, auth error or transport failure reports available, so a
probe that cannot answer can never break a tune that would have worked.
That contract is pinned by tests so a later refactor cannot invert it.
Rejected alternatives (see docs/decisions.md): resizing the
HlsSessionWorker retry loop (cannot distinguish a dead item from a
transient transcoder failure -- prod has live VAAPI hwupload -22 failures
that must keep retrying), and writing MediaItemState from the streaming
path (breaks scanner ownership, and would not have fixed this: the item
is RemoteOnly, which PlayoutBuilder's skip does not exclude).
Scanner-side follow-ups filed separately: #476 (FileNotFound does not
cascade show -> episodes, the reason dead items keep being scheduled),
fixes#473
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds one Information-level structured log per HLS tune-in cold-start so the
real driver breakdown can be measured on prod before optimizing the transcode
pipeline (measure-before-optimize). Log-only; no transcode behavior change.
- WaitForPlaylistSegments returns a PlaylistSegmentsResult: Phase A (process
startup -> playlist exists) vs Phase B (segment fill), segments reached,
deadline-expired.
- StartFFmpegSessionHandler emits one summary: total = setup + startup + fill,
plus cleanly-detectable feature flags (subtitle burn-in, hwaccel family).
- ColdStartFeatures: pure, unit-tested args->features helper (14 cases).
Watermark / HDR->SDR / image-subtitle burn-in are deliberately not flagged
(all reduce to overlay= in the args, indistinguishable); the full ffmpeg
arguments remain available at Debug.
Refs #350 (instrumentation slice; optimization deferred pending real data).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
timothy
