Three Low findings from the #536 clamp re-review, unreachable today (one
guarded release site) but filed against the day a second release site is added.
- §1: Release() now reads the count and CAS-decrements only when current > 0,
so it never publishes a negative count even transiently. The prior
decrement-first-then-clamp shape dipped to -1, which a concurrent TryAcquire
could read as phantom room and over-admit at the limit (re-opening the #529
QSV pool exhaustion). It records the unbalanced release synchronously on the
offending thread rather than blaming a later innocent release.
- §3: Release() returns bool; HlsSessionWorker logs a warning on the false
(unbalanced) return — the one in-band signal a future second release site
would need. WorkAheadSlots stays logger-free by design.
- §2: UnbalancedReleases doc-comment corrected — it can under-count (an
over-release while count > 0 cancels a coexisting leak and goes unrecorded);
no false positives, but zero does not prove correctness.
Test: Release_Unbalanced_NeverPublishesNegativeCount (2M unbalanced releases vs
4 count-samplers) with a documented, verified negative control (reverting to
the decrement-first body makes readers observe the transient -1).
Adds a decisions.md entry (ffmpeg.work-ahead-slot-release-never-negative).
fixes#539
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The slot check and its increment straddled an await: `Run` compared
`Volatile.Read(ref _workAheadCount)` against a DB-backed limit, and the
increment happened later inside `Transcode`. Every simultaneous tune-in
therefore observed `0 < limit` and started unthrottled — three concurrent
tunes on prod with a limit of 1 all ran with no `-readrate`. `Interlocked`
on the write side alone buys nothing when the read side is a separate,
earlier load (same class as #231/#250).
Extract the counter into a `WorkAheadSlots` pool whose `TryAcquire(limit)`
claims via compare-exchange, so the count never even transiently exceeds
the limit that the QSV hardware-frame pool sizing (#529) is derived from.
`Run` claims the slot and passes ownership in; `Transcode(bool
ownsWorkAheadSlot, ...)` derives `realtime` from it and releases it in its
existing `finally`, keeping acquire/release one-for-one. Acquisition stays
in the caller because `Run` sets `_state` from the outcome and `Transcode`
reads that state on entry to pick the item start time.
Tests hammer 8 threads x 20k rounds (a single Barrier round does not
collide on this hardware); the documented negative control reinstates the
check-then-act body and produces 15912 over-claiming rounds of 20000.
Also annotates the #350 decision record, whose "every concurrent tune-in
falls back to the throttled path" bullet described the intent rather than
the behaviour.
fixes#536
From the cold adversarial review of the initial diff. No blockers were
found; these address what the numbers MEAN, which is the whole point of
an instrumentation change.
- The buckets span the worker's Run entry, not the startup stopwatch, so
prep overlaps the tail of `setup`. Rather than let the log imply an
invariant it does not satisfy, say "spans runEntry" in the line, spell
it out in the doc comment, and rename the test that had codified the
false `sum == startup` claim.
- Guard `processLaunched > playlistExists` -> Unavailable: a stale
live.m3u8 from a previous session (Run warns about a non-empty
transcode folder but does not delete it) would otherwise yield a
plausible-looking sample whose prep exceeds the measured phase.
- Split the two-way fallback into TwoWay vs TwoWayLateProgress. They are
different stories about the pipeline and discriminating stories is
what this issue is for.
- Document the 100ms playlist-poll quantization (it lands entirely in
firstGop, the smallest bucket) and the first-process-failed case where
ffmpegInit spans a retry.
- Short-circuit the per-line timestamp call; static readonly Unavailable.
- Tests for the new guard, progress-before-launch, and boundary equality
(so tightening >= to > later cannot pass silently).
#350's measurement showed `startup` is 81% of tune-in latency and carries
100% of its variance, while remaining one opaque bucket spanning FFmpeg
spawn -> input open/probe -> encoder init -> first GOP. Two hypotheses
survive that measurement (NFS input open vs VAAPI init under contention)
and they need opposite fixes, so split before optimizing.
Adds `prep` (ErsatzTV-side work before FFmpeg exists) + `ffmpegInit`
(launch -> first `-progress` output) + `firstGop` (-> live.m3u8 exists)
to the existing Information-level cold-start line.
The pipeline runs `-loglevel error -nostats -hide_banner`, so a healthy
FFmpeg writes nothing to stderr; the `-progress` stream is the only
zero-cost milestone available and `ffmpegInit` therefore still lumps
input-open with encoder-init. That limit is documented rather than
papered over, and the split degrades to the two-way form #472 accepts
when no progress arrives before the playlist.
Log-only: no transcode behavior change, no new endpoint or config knob.
fixes#472
Third review pass: MERGEABLE WITH NITS. Taking the one finding it asked
for before merge, plus a doc nit.
The cancellation filter added last commit had no token check, and it spans
the whole Transcode body -- including every mediator send (ffprobe via
CliWrap, media-server API calls, subtitle extraction, song-video
generation). TaskCanceledException is also what HttpClient throws on its
OWN timeout, so a real timeout in any of those was being downgraded from
an ERROR with a stack trace to a routine "Terminating HLS session"
Information line. Behaviour was unchanged (both arms return false) but the
fault signal was lost, and this repo has been bitten before by
"empty log != the event didn't happen". Now filters on
cancellationToken.IsCancellationRequested, so only genuine caller
cancellation is treated as a graceful teardown.
Doc nit: the <exception> block said cancellation "is thrown"; it is only
thrown when the token trips while the probe is in flight -- cancelling
after it completes returns normally. Now says "may propagate".
Declined the reviewer's optional suggestion to drain until a 0-return
instead of reading exactly one byte: reading exactly one byte is what
makes the guard safe BY CONSTRUCTION, since a server or proxy that
answers 206 with a wider range than requested still cannot be drained
unboundedly. 206-only was confirmed correct rather than extended to short
200s, since deciding "short" from Content-Length would reopen the
unbounded path for a chunked or Content-Length-less response.
Also records the operator's standing rule in the handoff lore: a lone
`decisions.md append-only` red is a known infra flake -- do not
investigate, rebase, amend or push to clear it; the operator reruns that
job from the UI. I violated this earlier in this PR with a
tidy-but-wrong "my entry is no longer at EOF" theory, and the rebase did
not fix it -- the job went red again on a verified pure-append diff, which
is the proof the red was never about the diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second review pass returned BLOCKED on two findings introduced by the
first fix commit. Both were right.
BLOCKER 1 — the drain added for "return the connection to the pool" was
unbounded. `response.Content.ReadAsByteArrayAsync()` buffers the WHOLE
body, and it ran for every non-404 response. A server that ignores
`Range: bytes=0-0` answers 200 with the entire file, so this would
download at line rate into a byte[] on the streaming hot path for up to
the 2s timeout -- strictly worse than the aborted socket it replaced, and
it defeated the ResponseHeadersRead the probe deliberately uses. Now the
single byte is read only on 206 (where the server honoured the range and
the body really is one byte); any other status aborts the socket, which
is much the cheaper evil. Two tests pin both directions; verified
non-vacuous (restoring the unbounded drain fails the 200-with-body test).
BLOCKER 2 — IRemoteStreamProber's doc-comment still described pre-fix
behaviour. I had told the reviewer it was updated; it was not -- only the
implementation's <remarks> had been. It claimed `false` on any 404 (now
only a redirected one) and that every other outcome returns `true` (caller
cancellation throws). Both clauses corrected, and the throwing contract is
now documented with <exception>.
Also fixed the reviewer's own follow-on finding: the cancellation rethrow
it asked for reached HlsSessionWorker's catch-all, which logs a
channel-level ERROR with a stack trace. The graceful
TaskCanceledException/OperationCanceledException handler at :662 wraps only
the inner ffmpeg block, not the mediator sends, so every client disconnect
on a remote-streaming channel would have produced a spurious ERROR -- in
exactly the logs a #350 cold-start investigation reads. Added a
cancellation filter on the outer try that logs Information instead.
Nit: stale SeedAll doc-comment now mentions the emby case.
Deferred, per reviewer's explicit agreement: Plex-branch handler coverage
(follow-up), and HEAD-with-GET-fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds one Information-level structured log per HLS tune-in cold-start so the
real driver breakdown can be measured on prod before optimizing the transcode
pipeline (measure-before-optimize). Log-only; no transcode behavior change.
- WaitForPlaylistSegments returns a PlaylistSegmentsResult: Phase A (process
startup -> playlist exists) vs Phase B (segment fill), segments reached,
deadline-expired.
- StartFFmpegSessionHandler emits one summary: total = setup + startup + fill,
plus cleanly-detectable feature flags (subtitle burn-in, hwaccel family).
- ColdStartFeatures: pure, unit-tested args->features helper (14 cases).
Watermark / HDR->SDR / image-subtitle burn-in are deliberately not flagged
(all reduce to overlay= in the args, indistinguishable); the full ffmpeg
arguments remain available at Debug.
Refs #350 (instrumentation slice; optimization deferred pending real data).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* add multiple watermarks per playout item
* fixes
* update yaml playout watermark to support multiple watermarks
* use graphics engine for intermittent watermarks
* fix validation in new form layout
* pin mediatr to last oss version
* update dependencies
* cleanup code in core
* cleanup code in ffmpeg
* cleanup code in infra
* cleanup code in scanner
* cleanup code in application
* cleanup main code
* cleanup test code
* solution-wide code cleanup
timothyandClaude Opus 4.8