feat(350): instrument HLS cold-start latency (phase split + feature flags) #446

Merged
timothy merged 1 commits from feat/350-coldstart-instrumentation into main 2026-07-18 16:16:39 +02:00
Owner

Refs #350 — instrumentation slice only; the transcode-pipeline optimization stays deferred pending real data.

Why

#350's cost-driver ranking (subtitle burn-in #1, HDR→SDR, watermark, VAAPI init) comes from a single prod observation. Per measure-before-optimize, this PR ships instrumentation so the real driver breakdown can be measured on prod first, then the optimization lands data-driven. Approach was brainstormed + approved with the operator (instrumentation-only scope; structured log; Phase A/B split + feature flags).

What

One Information-level structured log line per cold-start (new-session tune-in only; the warm/already-active path is untouched):

HLS cold-start channel {Channel} mode {Mode}: total {TotalMs}ms
(setup {SetupMs}ms + startup {ProcessStartupMs}ms + fill {SegmentFillMs}ms),
segments {SegmentsReached}/{InitialSegmentCount}, deadlineExpired {DeadlineExpired},
subtitleBurnIn {SubtitleBurnIn}, hwaccel {HwAccel}
  • Phase split: setup (handler config/mediator/worker-spawn overhead) + startup (Phase A: FFmpeg spawn → live.m3u8 exists — probe + libass/encoder init + first GOP, where the hypothesized drivers live) + fill (Phase B: playlist → N segments, or the 8s deadline). total = setup + startup + fill accounts for every ms.
  • Self-describing feature flags: SubtitleBurnIn (libass subtitles= filter) and HwAccel family (encoder suffix / -hwaccel) so latency can be attributed to a driver without cross-referencing runs.

Design boundaries (stated deviations)

  • Watermark / HDR→SDR / image-based subtitle burn-in are deliberately NOT flagged: in the args they all reduce to overlay=, indistinguishable from one another — a boolean would mislead the measurement. The full ffmpeg command remains available at the existing Debug ffmpeg hls arguments log for those details.
  • Feature detection is a case-sensitive substring scan of the whole args string (includes the input path), so a media filename literally containing subtitles= / _amf could false-positive; telemetry-only, low-probability, documented in the type.

Files

  • ErsatzTV.Core/FFmpeg/ColdStartFeatures.cs (new) — pure args→features helper
  • ErsatzTV.Core/FFmpeg/PlaylistSegmentsResult.cs (new) — phase-timing result struct
  • ErsatzTV.Application/Streaming/HlsSessionWorker.cs — WaitForPlaylistSegments returns the struct; captures first FFmpeg args (volatile, happens-before via segment existence)
  • ErsatzTV.Application/Streaming/Commands/StartFFmpegSessionHandler.cs — request-scoped stopwatch + the summary log
  • ErsatzTV.Core/Interfaces/FFmpeg/IHlsSessionWorker.cs — return-type change (only impl is HlsSessionWorker; no mock breakage)
  • ErsatzTV.Core.Tests/FFmpeg/ColdStartFeaturesTests.cs (new) — 14 cases

Verification

  • Build clean (0 warnings), 14/14 helper tests, IHlsSessionWorker mock test green, #311 BOM gate clean, dotnet format --verify-no-changes exit 0.
  • Independent cold-context adversarial review: MERGEABLE — no Blocker/High. Verified the cross-thread happens-before (volatile write before ffmpeg spawn → segment file → volatile read), phase-timing edge cases (deadline / count=0 / cancellation), interface-change safety, and log-once semantics. The two Low findings (non-summing total, substring caveat) were folded into this diff.
  • Log-only, read-path, no transcode behavior change → no live-E2E required (final confirmation is the log line appearing on the operator's next real tune-in).

Docs

No API/SPA/route/convention change → no convention-doc update required. Deferred optimization directions (warm-start pre-roll, conditional subtitle skip, readrate burst) remain tracked on #350.

Refs #350 — **instrumentation slice only**; the transcode-pipeline optimization stays deferred pending real data. ## Why #350's cost-driver ranking (subtitle burn-in #1, HDR→SDR, watermark, VAAPI init) comes from a **single prod observation**. Per measure-before-optimize, this PR ships instrumentation so the real driver breakdown can be measured on prod first, then the optimization lands data-driven. Approach was brainstormed + approved with the operator (instrumentation-only scope; structured log; Phase A/B split + feature flags). ## What One **Information-level structured log line per cold-start** (new-session tune-in only; the warm/already-active path is untouched): ``` HLS cold-start channel {Channel} mode {Mode}: total {TotalMs}ms (setup {SetupMs}ms + startup {ProcessStartupMs}ms + fill {SegmentFillMs}ms), segments {SegmentsReached}/{InitialSegmentCount}, deadlineExpired {DeadlineExpired}, subtitleBurnIn {SubtitleBurnIn}, hwaccel {HwAccel} ``` - **Phase split**: `setup` (handler config/mediator/worker-spawn overhead) + `startup` (Phase A: FFmpeg spawn → `live.m3u8` exists — probe + libass/encoder init + first GOP, where the hypothesized drivers live) + `fill` (Phase B: playlist → N segments, or the 8s deadline). `total = setup + startup + fill` accounts for every ms. - **Self-describing feature flags**: `SubtitleBurnIn` (libass `subtitles=` filter) and `HwAccel` family (encoder suffix / `-hwaccel`) so latency can be attributed to a driver without cross-referencing runs. ## Design boundaries (stated deviations) - **Watermark / HDR→SDR / image-based subtitle burn-in are deliberately NOT flagged**: in the args they all reduce to `overlay=`, indistinguishable from one another — a boolean would mislead the measurement. The full ffmpeg command remains available at the existing Debug `ffmpeg hls arguments` log for those details. - Feature detection is a case-sensitive substring scan of the whole args string (includes the input path), so a media filename literally containing `subtitles=` / `_amf` could false-positive; telemetry-only, low-probability, documented in the type. ## Files - `ErsatzTV.Core/FFmpeg/ColdStartFeatures.cs` (new) — pure args→features helper - `ErsatzTV.Core/FFmpeg/PlaylistSegmentsResult.cs` (new) — phase-timing result struct - `ErsatzTV.Application/Streaming/HlsSessionWorker.cs` — `WaitForPlaylistSegments` returns the struct; captures first FFmpeg args (`volatile`, happens-before via segment existence) - `ErsatzTV.Application/Streaming/Commands/StartFFmpegSessionHandler.cs` — request-scoped stopwatch + the summary log - `ErsatzTV.Core/Interfaces/FFmpeg/IHlsSessionWorker.cs` — return-type change (only impl is `HlsSessionWorker`; no mock breakage) - `ErsatzTV.Core.Tests/FFmpeg/ColdStartFeaturesTests.cs` (new) — 14 cases ## Verification - Build clean (0 warnings), 14/14 helper tests, `IHlsSessionWorker` mock test green, #311 BOM gate clean, `dotnet format --verify-no-changes` exit 0. - **Independent cold-context adversarial review: MERGEABLE** — no Blocker/High. Verified the cross-thread happens-before (volatile write before ffmpeg spawn → segment file → volatile read), phase-timing edge cases (deadline / count=0 / cancellation), interface-change safety, and log-once semantics. The two Low findings (non-summing total, substring caveat) were folded into this diff. - Log-only, read-path, no transcode behavior change → no live-E2E required (final confirmation is the log line appearing on the operator's next real tune-in). ## Docs No API/SPA/route/convention change → no convention-doc update required. Deferred optimization directions (warm-start pre-roll, conditional subtitle skip, readrate burst) remain tracked on #350.
timothy added 1 commit 2026-07-18 15:45:58 +02:00
feat(350): instrument HLS cold-start latency (phase split + feature flags)
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 9s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 5s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 5s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 7m6s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 14m27s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 19m10s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 13m24s
6c8c7feeba
Adds one Information-level structured log per HLS tune-in cold-start so the
real driver breakdown can be measured on prod before optimizing the transcode
pipeline (measure-before-optimize). Log-only; no transcode behavior change.

- WaitForPlaylistSegments returns a PlaylistSegmentsResult: Phase A (process
  startup -> playlist exists) vs Phase B (segment fill), segments reached,
  deadline-expired.
- StartFFmpegSessionHandler emits one summary: total = setup + startup + fill,
  plus cleanly-detectable feature flags (subtitle burn-in, hwaccel family).
- ColdStartFeatures: pure, unit-tested args->features helper (14 cases).

Watermark / HDR->SDR / image-subtitle burn-in are deliberately not flagged
(all reduce to overlay= in the args, indistinguishable); the full ffmpeg
arguments remain available at Debug.

Refs #350 (instrumentation slice; optimization deferred pending real data).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
Owner

Independent cold-context adversarial review (review-only agent, ran locally on the working-tree diff before push): no Blocker/High. Verified sound: the cross-thread happens-before for _coldStartFFmpegArguments (volatile write before ffmpeg spawn → segment file appears → volatile read), phase-timing across edge cases (deadline hit / initialSegmentCount==0 / cancellation mid-wait / return-inside-try vs finally), the IHlsSessionWorker return-type change (only impl + mock unaffected), and log-fires-once-per-cold-start semantics. No analyzer/WaE risk.

Two Low findings were folded into the current head commit (they were the reviewer's own suggestions): (1) added an explicit setup field so total = setup + startup + fill sums, and (2) documented the substring-scan input-path caveat + the image-subtitle/watermark overlay ambiguity. Both are log/doc-only; the log-arg fix is build-verified (0 warnings ⇒ Serilog analyzer confirms 11 tokens match 11 args).

Review-verdict: MERGEABLE @ 6c8c7feeba

Independent cold-context adversarial review (review-only agent, ran locally on the working-tree diff before push): **no Blocker/High**. Verified sound: the cross-thread happens-before for `_coldStartFFmpegArguments` (volatile write before ffmpeg spawn → segment file appears → volatile read), phase-timing across edge cases (deadline hit / `initialSegmentCount==0` / cancellation mid-wait / return-inside-try vs finally), the `IHlsSessionWorker` return-type change (only impl + mock unaffected), and log-fires-once-per-cold-start semantics. No analyzer/WaE risk. Two **Low** findings were folded into the current head commit (they were the reviewer's own suggestions): (1) added an explicit `setup` field so `total = setup + startup + fill` sums, and (2) documented the substring-scan input-path caveat + the image-subtitle/watermark overlay ambiguity. Both are log/doc-only; the log-arg fix is build-verified (0 warnings ⇒ Serilog analyzer confirms 11 tokens match 11 args). Review-verdict: MERGEABLE @ 6c8c7feeba843ebf2a7887a59ddb524da6fa60f7
timothy merged commit 1cd06395aa into main 2026-07-18 16:16:39 +02:00
timothy deleted branch feat/350-coldstart-instrumentation 2026-07-18 16:16:39 +02:00
Sign in to join this conversation.