fix(locking): release troubleshooting + playout locks on all terminal paths (#233, #234) #250

Merged
timothy merged 2 commits from fix/233-234-lock-leaks into main 2026-07-11 14:12:53 +02:00
Owner

Batches the two reviewer#20 lock-lifecycle High findings (arc item 2 remainder). Both consume the EntityLocker ownership contract landed in #231/#241 (Lock* returns true iff this caller won the slot; release in a finally gated on that bool, or hand off to one designated releaser).

#233 (F3) — troubleshooting playback lock leak + 409

  • PrepareTroubleshootingPlaybackHandler: both lock sites now acquire via if (!LockTroubleshootingPlayback()) — this kills the check-then-set TOCTOU (two racers could both pass the old IsLocked check and both proceed as "owner"). The empty-media-path Left return now releases the lock it acquired; previously it leaked, wedging GET /api/troubleshoot/playback/status at "running" forever for a file gone from disk.
  • TroubleshootController.TroubleshootPlayback: lock conflict → 409 ProblemDetails (was a bare 404, indistinguishable from a bad id); a startEnqueued guard releases the lock if we never hand off to StartTroubleshootingPlayback (cancellation/exception in the Prepare-success → enqueue window).

#234 (F4 + F5.2) — playout build lock leaks

  • ExtractEmbeddedSubtitlesHandler: try/finally releases exactly the playouts it locked, on every terminal path (cancellation early-return, swallowed OperationCanceledException, any exception). Fixes the permanent leak after a cancelled mid-extraction (runs after every build + hourly) and stops cross-releasing playouts held by someone else.
  • BuildPlayoutHandler: skips (logs, returns Right) when LockPlayout returns false instead of building unlocked and then cross-releasing the other owner's lock in finally. (F4 + F5.2 land together, per the issue: fixing F5.2 alone would make F4's leaks permanent.)

Tests

Handler-level release-discipline tests via the InMemoryTvContext harness:

  • PrepareTroubleshootingPlaybackHandlerTests — empty-media-path releases the lock (verified non-vacuous against a negative control).
  • ExtractEmbeddedSubtitlesHandlerTests — releases all locks on cancellation; releases only locks it acquired (no cross-release).
  • BuildPlayoutHandlerTests — skips + no cross-release when already locked; releases when owned even on validation failure.
  • TroubleshootControllerTests — 409 when already locked (+ DidNotReceive dispatch).
  • OpenApiErrorResponseContractTests — 409 contract cases for /api/troubleshoot/playback.m3u8 (get + head).

OpenAPI regenerated (v1.json; endpoint-index unchanged, no new op). SPA API client in sync (endpoint returns a redirect, no typed model consumed). No SPA consumer of the status-code change (troubleshooting UI is still Blazor).

Full ErsatzTV.Tests suite green locally (1059 passed).

Fixes #233, #234

🤖 Generated with Claude Code

Batches the two reviewer#20 lock-lifecycle High findings (arc item 2 remainder). Both consume the EntityLocker ownership contract landed in #231/#241 (`Lock*` returns `true` iff this caller won the slot; release in a `finally` gated on that bool, or hand off to one designated releaser). ## #233 (F3) — troubleshooting playback lock leak + 409 - **`PrepareTroubleshootingPlaybackHandler`**: both lock sites now acquire via `if (!LockTroubleshootingPlayback())` — this kills the check-then-set TOCTOU (two racers could both pass the old `IsLocked` check and both proceed as "owner"). The empty-media-path `Left` return now releases the lock it acquired; previously it leaked, wedging `GET /api/troubleshoot/playback/status` at `"running"` forever for a file gone from disk. - **`TroubleshootController.TroubleshootPlayback`**: lock conflict → **409 ProblemDetails** (was a bare 404, indistinguishable from a bad id); a `startEnqueued` guard releases the lock if we never hand off to `StartTroubleshootingPlayback` (cancellation/exception in the Prepare-success → enqueue window). ## #234 (F4 + F5.2) — playout build lock leaks - **`ExtractEmbeddedSubtitlesHandler`**: `try/finally` releases **exactly the playouts it locked**, on every terminal path (cancellation early-return, swallowed `OperationCanceledException`, any exception). Fixes the permanent leak after a cancelled mid-extraction (runs after every build + hourly) and stops cross-releasing playouts held by someone else. - **`BuildPlayoutHandler`**: skips (logs, returns `Right`) when `LockPlayout` returns `false` instead of building unlocked and then cross-releasing the other owner's lock in `finally`. (F4 + F5.2 land together, per the issue: fixing F5.2 alone would make F4's leaks permanent.) ## Tests Handler-level release-discipline tests via the `InMemoryTvContext` harness: - `PrepareTroubleshootingPlaybackHandlerTests` — empty-media-path releases the lock (**verified non-vacuous against a negative control**). - `ExtractEmbeddedSubtitlesHandlerTests` — releases all locks on cancellation; releases only locks it acquired (no cross-release). - `BuildPlayoutHandlerTests` — skips + no cross-release when already locked; releases when owned even on validation failure. - `TroubleshootControllerTests` — 409 when already locked (+ `DidNotReceive` dispatch). - `OpenApiErrorResponseContractTests` — 409 contract cases for `/api/troubleshoot/playback.m3u8` (get + head). OpenAPI regenerated (`v1.json`; endpoint-index unchanged, no new op). SPA API client in sync (endpoint returns a redirect, no typed model consumed). No SPA consumer of the status-code change (troubleshooting UI is still Blazor). Full `ErsatzTV.Tests` suite green locally (1059 passed). Fixes #233, #234 🤖 Generated with [Claude Code](https://claude.com/claude-code)
timothy added 1 commit 2026-07-11 13:45:16 +02:00
fix(locking): release troubleshooting + playout locks on all terminal paths (#233, #234)
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 5s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 5m55s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 8m24s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
084d4c4ca1
Consume the EntityLocker ownership contract (#231/#241: Lock* returns true iff
this caller won the slot) at three lock-leak sites surfaced by adversarial-reviewer#20.

#233 (F3) — troubleshooting playback:
- PrepareTroubleshootingPlaybackHandler: both lock sites now acquire via
  `if (!LockTroubleshootingPlayback())` (kills the check-then-set TOCTOU) and the
  empty-media-path Left return releases the lock it acquired — previously it leaked,
  wedging the status endpoint at "running" forever for a file gone from disk.
- TroubleshootController.TroubleshootPlayback: lock conflict is now 409 ProblemDetails
  (was a bare 404, indistinguishable from a bad id); the Prepare-success -> enqueue
  window releases the lock if we never hand off to StartTroubleshootingPlayback.

#234 (F4 + F5.2) — playout builds:
- ExtractEmbeddedSubtitlesHandler: try/finally releases exactly the playouts it
  locked, on every terminal path (cancellation early-return, swallowed cancellation,
  any exception) — no more permanent leaks after cancelled mid-extraction, and no
  cross-release of playouts held by someone else.
- BuildPlayoutHandler: skips (logs, returns Right) when LockPlayout returns false
  instead of building unlocked and cross-releasing the other owner's lock in finally.

Tests: handler-level release-discipline tests (Prepare empty-path, Extract
cancellation + no-cross-release, BuildPlayout skip + finally-release) via the
InMemoryTvContext harness, a TroubleshootController 409 test, and OpenApi contract
cases for the m3u8 endpoint's 409. OpenAPI regenerated. All non-vacuous (F3 verified
against a negative control).

Fixes #233, #234

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
timothy added 1 commit 2026-07-11 14:02:05 +02:00
fix(review): centralize troubleshooting lock ownership in Handle (Codex High)
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 9s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m3s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 10m4s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
8cdc963636
Independent Codex review of PR #250 caught a cross-release the fork missed: the
outer catch in PrepareTroubleshootingPlaybackHandler released the troubleshooting
lock unconditionally, so an exception BEFORE this caller acquired it (e.g. request
cancellation during validation, or a DB error) would release a lock held by another
session.

Fix: track ownership with a Handle-scoped `lockAcquired` flag and gate the catch on
it. Acquisition for the media-item path moves out of GetProcess up into Handle (after
validation succeeds), so one place owns the full lifecycle: acquire -> on Left release
-> on success hand off to StartTroubleshootingPlayback -> on any exception release only
if we own it. GetProcess is now lock-free.

Test: Handle_Should_Not_Release_Lock_It_Never_Acquired (exception before acquisition ->
no Unlock, no cross-release), verified non-vacuous against an inverted-condition
negative control. Strengthened the empty-path test to also assert acquisition happened.

Deferred (noted for close comment): worker-dispatch-failure lock leak and
BuildPlayout silent-skip observability are pre-existing / sanctioned -> #235.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
Owner

Adversarial review round 1 — reconciled

Two independent passes: a fork review (full design context) + an independent Codex pass.

Fork: no Blockers/Highs. Verified no double-unlock in the troubleshooting handler, the controller enqueue-guard correct, Extract releases exactly what it locked, and all new tests non-vacuous. Verdict: ship.

Codex found one High the fork missed — a real cross-release: the outer catch in PrepareTroubleshootingPlaybackHandler released the troubleshooting lock unconditionally, so an exception before this caller acquired the lock (request cancellation during validation, or a DB error) would release a lock held by another session.

Fixed in 8cdc9636: centralized lock ownership in Handle — a lockAcquired flag gates the catch, acquisition for the media path moved up out of GetProcess (after validation), so one place owns acquire → release-on-Left → hand-off-on-success → release-on-exception-only-if-owned. GetProcess is now lock-free. New regression test Handle_Should_Not_Release_Lock_It_Never_Acquired (throwing db-context factory → exception before acquisition → asserts no Unlock), verified non-vacuous against an inverted-condition negative control. Empty-path test strengthened to also assert acquisition happened.

Deferred (not blocking), with rationale:

  • Worker-dispatch-failure leak (Codex Med): if the worker fails to reach StartTroubleshootingPlaybackHandler after enqueue, the lock leaks. Pre-existing; the hand-off-to-a-background-worker guarantee is #232/#235 territory (the "in-memory locks die with the process" decision). → #235.
  • BuildPlayoutHandler silent skip returns Right (both reviewers, Med): a build skipped due to a held lock is reported as success with no distinguishable signal. Strictly safer than the pre-fix concurrent-build+cross-release, and explicitly sanctioned by #234 ("skip explicitly"); the observability/202 surface is #235.
  • Extract proceeds on a playout it couldn't lock (Codex Med): deliberate and benign — extraction only reads playout items; the lock is advisory (tells ResetAll to skip). Not a regression.
  • TOCTOU race-loser returns 404 not 409 (fork Low): cosmetic, matches the repo's existing controller pattern.

Full ErsatzTV.Tests green locally (1060). Follow-up re-review of the refactor in progress.

## Adversarial review round 1 — reconciled Two independent passes: a fork review (full design context) + an independent **Codex** pass. **Fork**: no Blockers/Highs. Verified no double-unlock in the troubleshooting handler, the controller enqueue-guard correct, Extract releases exactly what it locked, and all new tests non-vacuous. Verdict: ship. **Codex** found one **High the fork missed** — a real cross-release: the outer `catch` in `PrepareTroubleshootingPlaybackHandler` released the troubleshooting lock *unconditionally*, so an exception **before** this caller acquired the lock (request cancellation during validation, or a DB error) would release a lock held by **another** session. **Fixed in `8cdc9636`**: centralized lock ownership in `Handle` — a `lockAcquired` flag gates the `catch`, acquisition for the media path moved up out of `GetProcess` (after validation), so one place owns acquire → release-on-Left → hand-off-on-success → release-on-exception-only-if-owned. `GetProcess` is now lock-free. New regression test `Handle_Should_Not_Release_Lock_It_Never_Acquired` (throwing db-context factory → exception before acquisition → asserts no Unlock), **verified non-vacuous against an inverted-condition negative control**. Empty-path test strengthened to also assert acquisition happened. **Deferred (not blocking), with rationale:** - **Worker-dispatch-failure leak** (Codex Med): if the worker fails to reach `StartTroubleshootingPlaybackHandler` after enqueue, the lock leaks. Pre-existing; the hand-off-to-a-background-worker guarantee is #232/#235 territory (the "in-memory locks die with the process" decision). → **#235**. - **`BuildPlayoutHandler` silent skip returns `Right`** (both reviewers, Med): a build skipped due to a held lock is reported as success with no distinguishable signal. Strictly safer than the pre-fix concurrent-build+cross-release, and explicitly sanctioned by #234 ("skip explicitly"); the observability/202 surface is **#235**. - **Extract proceeds on a playout it couldn't lock** (Codex Med): deliberate and benign — extraction only reads playout items; the lock is advisory (tells `ResetAll` to skip). Not a regression. - TOCTOU race-loser returns 404 not 409 (fork Low): cosmetic, matches the repo's existing controller pattern. Full `ErsatzTV.Tests` green locally (1060). Follow-up re-review of the refactor in progress.
timothy merged commit 77272ed5a7 into main 2026-07-11 14:12:53 +02:00
timothy deleted branch fix/233-234-lock-leaks 2026-07-11 14:12:53 +02:00
Sign in to join this conversation.