TroubleshootController playback segment-wait loop doesn't check notifier.IsFailed (spins until client cancel if ffmpeg dies) #320

Closed
opened 2026-07-12 21:06:14 +02:00 by timothy · 2 comments
Owner

Split out of PR #316 review (fork nit 4, ratified by Fable as a real-but-pre-existing defect, out of scope for #316).

What

In ErsatzTV/Controllers/Api/TroubleshootController.cs, the second segment-readiness wait loop (around lines 237-253, the while (!hasSegments) poll) does not check notifier.IsFailed. If ffmpeg dies after writing the HLS playlist but before the first segments appear, the loop spins until the client cancels the request.

Why it matters more after #316

Pre-existing and unchanged by #316 — but #301 turned the manifest fetch into POST /api/troubleshoot/playback/start, which now holds the request open server-side during this wait (previously hls.js held it client-side). So a hung wait ties up the request thread + keeps the troubleshooting lock until the client gives up.

Fix direction

Add a notifier.IsFailed (or exit-code) check to the loop's exit condition so a dead ffmpeg breaks the wait and returns a 422/500 (or the existing error path) promptly, releasing the lock.

Priority: low (needs ffmpeg to die in a narrow window; self-heals on client cancel). Refs #316 #301.

Done-when

  • notifier.IsFailed / cancellation check added to the segment-wait loop so a dead ffmpeg breaks the wait
  • Non-vacuous regression test added (verified it fails on the pre-fix spinning loop)
  • Full local test suite green
  • CI green
  • Adversarial/independent review passed
  • No docs update needed: internal handler behavior fix, no new/changed endpoint contract or convention (the route + response codes are unchanged)
Split out of PR #316 review (fork nit 4, ratified by Fable as a real-but-pre-existing defect, out of scope for #316). ## What In `ErsatzTV/Controllers/Api/TroubleshootController.cs`, the second segment-readiness wait loop (around lines 237-253, the `while (!hasSegments)` poll) does **not** check `notifier.IsFailed`. If ffmpeg dies **after** writing the HLS playlist but **before** the first segments appear, the loop spins until the client cancels the request. ## Why it matters more after #316 Pre-existing and **unchanged** by #316 — but #301 turned the manifest fetch into `POST /api/troubleshoot/playback/start`, which now holds the request open server-side during this wait (previously hls.js held it client-side). So a hung wait ties up the request thread + keeps the troubleshooting lock until the client gives up. ## Fix direction Add a `notifier.IsFailed` (or exit-code) check to the loop's exit condition so a dead ffmpeg breaks the wait and returns a 422/500 (or the existing error path) promptly, releasing the lock. Priority: low (needs ffmpeg to die in a narrow window; self-heals on client cancel). Refs #316 #301. ## Done-when - [x] `notifier.IsFailed` / cancellation check added to the segment-wait loop so a dead ffmpeg breaks the wait - [x] Non-vacuous regression test added (verified it fails on the pre-fix spinning loop) - [x] Full local test suite green - [x] CI green - [x] Adversarial/independent review passed - No docs update needed: internal handler behavior fix, no new/changed endpoint contract or convention (the route + response codes are unchanged)
timothy added the bugpriority: low labels 2026-07-12 21:06:14 +02:00
timothy added priority: medium and removed priority: low labels 2026-07-17 00:25:09 +02:00
timothy added the in-progress label 2026-07-17 19:45:31 +02:00
Author
Owner

🔧 Claiming — clean self-contained Tier-5 bug pickup. Milestone tier was re-checked (selector had misread #385 as still blocked by #70, which merged today); operator opted for a clean bug instead. Fix direction is exactly as the issue states: add the missing notifier.IsFailed(sessionId) / cancellation check to the second (segment-wait) poll loop so a dead ffmpeg breaks the wait instead of spinning until client-cancel (holding the request thread + troubleshooting lock). Will add a regression test + a ## Done-when section, and run a cold independent review (API write-path handler touching the lock-hold window).

🔧 **Claiming** — clean self-contained Tier-5 bug pickup. Milestone tier was re-checked (selector had misread #385 as still blocked by #70, which merged today); operator opted for a clean bug instead. Fix direction is exactly as the issue states: add the missing `notifier.IsFailed(sessionId)` / cancellation check to the second (segment-wait) poll loop so a dead ffmpeg breaks the wait instead of spinning until client-cancel (holding the request thread + troubleshooting lock). Will add a regression test + a `## Done-when` section, and run a cold independent review (API write-path handler touching the lock-hold window).
timothy removed the in-progress label 2026-07-17 20:43:33 +02:00
Author
Owner

Done — merged to main (PR #423, de63603a)

What was done: Added the missing failure/exit check to the second (segment-readiness) poll loop in POST /api/v1/troubleshoot/playback/start. It now breaks on notifier.IsFailed(sessionId), mirroring the first (playlist-existence) wait loop above it.

Root cause: The loop's only exit condition was !hasSegments. If ffmpeg died after writing the HLS playlist but before the first segments appeared, nothing set hasSegments, so the loop spun until the client cancelled — and since #301 moved the manifest fetch server-side, that hung request held the request thread + the troubleshooting lock the whole time. The first loop already had the IsFailed/cancellation check; the second was simply missing it (an omission, not a design choice).

Files changed:

  • ErsatzTV/Controllers/Api/TroubleshootController.cs — break the segment-wait loop on IsFailed; the existing if (!IsFailed) gate then falls through to the terminal NotFoundProblem, releasing the thread + lock.
  • ErsatzTV.Tests/Controllers/TroubleshootControllerTests.cs — regression test TroubleshootPlayback_Should_Return_404_And_Not_Spin_When_Ffmpeg_Fails_Before_Segments.

Deliberate detail (from review): the break checks IsFailed only, not cancellationToken.IsCancellationRequested — the Task.Delay(ct) already throws on cancel (→ 404), and adding the cancel term would let a cancelled request fall into the Ok gate and return a spurious 200.

Testing: Non-vacuous regression test with a bounded 10s cancellation deadline — negative-control-verified (removing the IsFailed break makes it fail in ~10s, proving it catches the spin). Full local suite + CI green. Two cold independent review passes; all findings addressed (see PR #423 Review-verdict).

Docs: none required — internal behavior of an existing endpoint; route and response codes unchanged.

Deferred / follow-ups: none.

## Done — merged to main (PR #423, `de63603a`) **What was done:** Added the missing failure/exit check to the second (segment-readiness) poll loop in `POST /api/v1/troubleshoot/playback/start`. It now breaks on `notifier.IsFailed(sessionId)`, mirroring the first (playlist-existence) wait loop above it. **Root cause:** The loop's only exit condition was `!hasSegments`. If ffmpeg died *after* writing the HLS playlist but *before* the first segments appeared, nothing set `hasSegments`, so the loop spun until the client cancelled — and since #301 moved the manifest fetch server-side, that hung request held the request thread + the troubleshooting lock the whole time. The first loop already had the `IsFailed`/cancellation check; the second was simply missing it (an omission, not a design choice). **Files changed:** - `ErsatzTV/Controllers/Api/TroubleshootController.cs` — break the segment-wait loop on `IsFailed`; the existing `if (!IsFailed)` gate then falls through to the terminal `NotFoundProblem`, releasing the thread + lock. - `ErsatzTV.Tests/Controllers/TroubleshootControllerTests.cs` — regression test `TroubleshootPlayback_Should_Return_404_And_Not_Spin_When_Ffmpeg_Fails_Before_Segments`. **Deliberate detail (from review):** the break checks `IsFailed` only, *not* `cancellationToken.IsCancellationRequested` — the `Task.Delay(ct)` already throws on cancel (→ 404), and adding the cancel term would let a cancelled request fall into the Ok gate and return a spurious 200. **Testing:** Non-vacuous regression test with a bounded 10s cancellation deadline — negative-control-verified (removing the `IsFailed` break makes it fail in ~10s, proving it catches the spin). Full local suite + CI green. Two cold independent review passes; all findings addressed (see PR #423 `Review-verdict`). **Docs:** none required — internal behavior of an existing endpoint; route and response codes unchanged. **Deferred / follow-ups:** none.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#320