workAheadSegmenterLimit is not enforced: the slot check and its increment straddle an await (N concurrent tune-ins all run unthrottled) #536

Closed
opened 2026-07-21 17:50:29 +02:00 by timothy · 3 comments
Owner

Found while measuring #532 on prod (v26.12.0, digest-verified). Not a regression from #516/#529 — the defect is older than both, and it is what made #350's original "one slot winner, two throttled" observation look deterministic when it is actually a race.

The defect

ErsatzTV.Application/Streaming/HlsSessionWorker.cs:246

bool initialWorkAhead = Volatile.Read(ref _workAheadCount) < await GetWorkAheadLimit(cancellationToken);
_state = initialWorkAhead ? HlsSessionState.SeekAndWorkAhead : HlsSessionState.SeekAndRealtime;

if (!await Transcode(!initialWorkAhead, cancellationToken))

_workAheadCount is only incremented inside Transcode (line 467, Interlocked.Increment). So the check and the claim are not atomic, and they are separated by at least one await (GetWorkAheadLimit is a DB-backed config read). Every session that reads the counter before any of them increments sees 0 < 1 and takes the unthrottled work-ahead path. Line 272 (subsequentWorkAhead) has the identical shape.

Interlocked.Increment on the write side gives no mutual exclusion when the read side is a separate, earlier Volatile.Read — this is a plain check-then-act TOCTOU, same class as #231/#250.

Evidence (prod, streamingMinimumLogLevel: Debug, workAheadSegmenterLimit: 1)

Three simultaneous tune-ins of idle channels 122/102/105:

17:41:48 [DBG] ffmpeg hls arguments … (ch105)   ← no -readrate at all
17:41:48 [DBG] ffmpeg hls arguments … (ch102)   ← no -readrate at all
17:41:48 [DBG] ffmpeg hls arguments … (ch122)   ← no -readrate at all

3 of 3 sessions ran unthrottled against a configured limit of 1. Staggering the first tune by 3 s makes the limit appear to work (1 work-ahead + 2 throttled with -readrate 1.05 -readrate_initial_burst 8), which is exactly why this has never been noticed: any measurement that starts sessions sequentially sees correct behaviour.

A second variant is visible in the same logs — when the slot holder releases (will NOT work ahead for channel 122 at 17:44:43), the two waiting sessions both re-read 0 at 17:44:45.206 (same millisecond) and both re-acquire.

Why it matters

The limit is a deliberate resource guarantee, not a tuning knob. From #350's fix rationale, on explicitly declining to raise it:

that budget exists to stop N viewers from launching N unthrottled transcodes, and a config bump trades a real resource guarantee for something the burst gets for free

The guarantee does not hold in the one scenario it exists for: a simultaneous burst of viewers. Each unthrottled QSV transcode is also the thing that exhausted the hardware-frame pool in #529, so the blast radius is the failure mode we just shipped a floor for. Prod is not currently at risk (qsvExtraHardwareFrames: 64 absorbs it), but the pool sizing assumes at most workAheadSegmenterLimit unthrottled readers.

Suggested fix

Claim the slot atomically instead of checking then claiming — e.g. a compare-exchange loop against the limit, or take the limit first and then Interlocked.Increment with a decrement-and-fall-back-to-realtime if the post-increment value exceeds it:

int limit = await GetWorkAheadLimit(cancellationToken);
bool initialWorkAhead = Interlocked.Increment(ref _workAheadCount) <= limit;
if (!initialWorkAhead) Interlocked.Decrement(ref _workAheadCount);

with Transcode no longer double-incrementing (the ownership then has to be passed in, so the release path in Transcode's finally stays symmetric). The interaction with the release path is the fiddly part and deserves the review-round attention, not the increment itself.

Testing note

Per the #231/#250 lesson recorded in the queue lore, a one-shot Barrier(N) + Task.WhenAll test will not reliably catch this — hammer it over many rounds, and prove non-vacuity by inverting the real condition (never by stubbing if (true), which trips CS0219 → warnings-as-errors → a stale-dll false pass).

Done-when

  • Slot acquisition is atomic at both call sites (initial, line 246; subsequent, line 272)
  • Release path stays symmetric — no leaked or double-decremented slots
  • Multi-round concurrency test with a documented negative control (inverted condition, not if (true))
  • Independent adversarial review passed

Post-deploy verification (tracked here; NOT a merge gate)

Split out of Done-when on 2026-07-21: this box cannot be satisfied before the fix is merged and an image carrying it is deployed, so leaving it in the merge checklist would make the gate unsatisfiable by construction. It is still owed, and this issue stays open until it is reported.

  • Verified on prod or test with N simultaneous tune-ins: at most workAheadSegmenterLimit arg lines lack -readrate

Refs #350, #516, #529, #532.

Found while measuring #532 on prod (v26.12.0, digest-verified). Not a regression from #516/#529 — the defect is older than both, and it is what made #350's original "one slot winner, two throttled" observation look deterministic when it is actually a race. ## The defect `ErsatzTV.Application/Streaming/HlsSessionWorker.cs:246` ```csharp bool initialWorkAhead = Volatile.Read(ref _workAheadCount) < await GetWorkAheadLimit(cancellationToken); _state = initialWorkAhead ? HlsSessionState.SeekAndWorkAhead : HlsSessionState.SeekAndRealtime; if (!await Transcode(!initialWorkAhead, cancellationToken)) ``` `_workAheadCount` is only incremented inside `Transcode` (line 467, `Interlocked.Increment`). So the check and the claim are **not atomic**, and they are separated by at least one `await` (`GetWorkAheadLimit` is a DB-backed config read). Every session that reads the counter before any of them increments sees `0 < 1` and takes the unthrottled work-ahead path. Line 272 (`subsequentWorkAhead`) has the identical shape. `Interlocked.Increment` on the write side gives no mutual exclusion when the read side is a separate, earlier `Volatile.Read` — this is a plain check-then-act TOCTOU, same class as #231/#250. ## Evidence (prod, `streamingMinimumLogLevel: Debug`, `workAheadSegmenterLimit: 1`) Three simultaneous tune-ins of idle channels 122/102/105: ``` 17:41:48 [DBG] ffmpeg hls arguments … (ch105) ← no -readrate at all 17:41:48 [DBG] ffmpeg hls arguments … (ch102) ← no -readrate at all 17:41:48 [DBG] ffmpeg hls arguments … (ch122) ← no -readrate at all ``` **3 of 3 sessions ran unthrottled** against a configured limit of **1**. Staggering the first tune by 3 s makes the limit appear to work (1 work-ahead + 2 throttled with `-readrate 1.05 -readrate_initial_burst 8`), which is exactly why this has never been noticed: any measurement that starts sessions sequentially sees correct behaviour. A second variant is visible in the same logs — when the slot holder releases (`will NOT work ahead for channel 122` at 17:44:43), the two waiting sessions both re-read `0` at 17:44:45.206 (same millisecond) and **both** re-acquire. ## Why it matters The limit is a deliberate resource guarantee, not a tuning knob. From #350's fix rationale, on explicitly declining to raise it: > that budget exists to stop N viewers from launching N unthrottled transcodes, and a config bump trades a real resource guarantee for something the burst gets for free The guarantee does not hold in the one scenario it exists for: a simultaneous burst of viewers. Each unthrottled QSV transcode is also the thing that exhausted the hardware-frame pool in #529, so the blast radius is the failure mode we just shipped a floor for. Prod is not currently at risk (`qsvExtraHardwareFrames: 64` absorbs it), but the pool sizing assumes at most `workAheadSegmenterLimit` unthrottled readers. ## Suggested fix Claim the slot atomically instead of checking then claiming — e.g. a compare-exchange loop against the limit, or take the limit first and then `Interlocked.Increment` with a decrement-and-fall-back-to-realtime if the post-increment value exceeds it: ```csharp int limit = await GetWorkAheadLimit(cancellationToken); bool initialWorkAhead = Interlocked.Increment(ref _workAheadCount) <= limit; if (!initialWorkAhead) Interlocked.Decrement(ref _workAheadCount); ``` with `Transcode` no longer double-incrementing (the ownership then has to be passed in, so the release path in `Transcode`'s `finally` stays symmetric). The interaction with the release path is the fiddly part and deserves the review-round attention, not the increment itself. ## Testing note Per the #231/#250 lesson recorded in the queue lore, a one-shot `Barrier(N)` + `Task.WhenAll` test will **not** reliably catch this — hammer it over many rounds, and prove non-vacuity by inverting the real condition (never by stubbing `if (true)`, which trips CS0219 → warnings-as-errors → a stale-dll false pass). ## Done-when - [x] Slot acquisition is atomic at both call sites (initial, line 246; subsequent, line 272) - [x] Release path stays symmetric — no leaked or double-decremented slots - [x] Multi-round concurrency test with a documented negative control (inverted condition, not `if (true)`) - [x] Independent adversarial review passed ## Post-deploy verification (tracked here; NOT a merge gate) Split out of Done-when on 2026-07-21: this box cannot be satisfied before the fix is merged and an image carrying it is deployed, so leaving it in the merge checklist would make the gate unsatisfiable by construction. It is still owed, and this issue stays open until it is reported. - [x] Verified on prod or test with N simultaneous tune-ins: at most `workAheadSegmenterLimit` arg lines lack `-readrate` Refs #350, #516, #529, #532.
timothy added the bugpriority: medium labels 2026-07-21 17:50:42 +02:00
timothy added the in-progress label 2026-07-21 18:08:57 +02:00
Author
Owner

Claiming this (in-progress).

Approach, and where it deviates from the suggested fix:

  • Extracted the counter into ErsatzTV.Application/Streaming/WorkAheadSlots.cs — a pool with TryAcquire(limit) / Release(). Two reasons over patching in place: the static int was untestable (a process-wide field shared across tests), and the invariant now has one home to document.
  • Compare-exchange rather than the suggested increment-then-decrement. Both are correct on holder count, but increment-first transiently overshoots, so a concurrent reader of the count can observe a value above the limit. Since qsvExtraHardwareFrames sizing (#529) is derived from that bound, I'd rather the invariant never be violated even momentarily.
  • Ownership is passed in, as the issue anticipated. Run claims the slot; Transcode(bool ownsWorkAheadSlot, …) derives realtime = !ownsWorkAheadSlot on its first line and releases in its existing finally. Acquisition deliberately stays in the caller: Run sets _state from the outcome, and Transcode reads that state on entry (wasSeekAndWorkAhead) to choose between DateTimeOffset.Now and _transcodedUntil as the item start — folding acquisition inward would have silently flipped that branch. Inverting the parameter also removed the Transcode(!initialWorkAhead, …) double negative, so the contract is in the signature instead of implied.

Grepped for other consumers of the budget: _workAheadCount had exactly one reader/writer pair, and FFmpegWorkAheadSegmenters is otherwise only read by the settings/troubleshooting queries. No second variant to cover.

Negative control (per the #231/#250 lesson): non-vacuity proven. With the pre-fix check-then-act body reinstated, TryAcquire_ParallelCallers_NeverExceedsLimit fails with 15912 over-claiming rounds out of 20000; 4 of 7 tests go red. Restored via an explicit touch + rebuild (a cp restore alone preserves an old mtime and MSBuild skips the rebuild — the control then "passes" on a stale dll). Build output checked for error CS, not just a green tail.

On the Done-when checklist: box 4 (verify on prod/test with N simultaneous tune-ins) is only satisfiable after the image carrying this change is deployed, so it cannot be ticked pre-merge. I'll treat it as a post-deploy verification against ersatztv-test once :latest rebuilds, and report the arg-line count back here.

Full .NET suite green locally; independent adversarial review in flight before the push.

Claiming this (`in-progress`). Approach, and where it deviates from the suggested fix: - Extracted the counter into `ErsatzTV.Application/Streaming/WorkAheadSlots.cs` — a pool with `TryAcquire(limit)` / `Release()`. Two reasons over patching in place: the static `int` was untestable (a process-wide field shared across tests), and the invariant now has one home to document. - **Compare-exchange rather than the suggested increment-then-decrement.** Both are correct on holder count, but increment-first transiently overshoots, so a concurrent reader of the count can observe a value above the limit. Since `qsvExtraHardwareFrames` sizing (#529) is derived from that bound, I'd rather the invariant never be violated even momentarily. - **Ownership is passed in, as the issue anticipated.** `Run` claims the slot; `Transcode(bool ownsWorkAheadSlot, …)` derives `realtime = !ownsWorkAheadSlot` on its first line and releases in its existing `finally`. Acquisition deliberately stays in the caller: `Run` sets `_state` from the outcome, and `Transcode` reads that state on entry (`wasSeekAndWorkAhead`) to choose between `DateTimeOffset.Now` and `_transcodedUntil` as the item start — folding acquisition inward would have silently flipped that branch. Inverting the parameter also removed the `Transcode(!initialWorkAhead, …)` double negative, so the contract is in the signature instead of implied. Grepped for other consumers of the budget: `_workAheadCount` had exactly one reader/writer pair, and `FFmpegWorkAheadSegmenters` is otherwise only read by the settings/troubleshooting queries. No second variant to cover. **Negative control (per the #231/#250 lesson): non-vacuity proven.** With the pre-fix check-then-act body reinstated, `TryAcquire_ParallelCallers_NeverExceedsLimit` fails with **15912 over-claiming rounds out of 20000**; 4 of 7 tests go red. Restored via an explicit `touch` + rebuild (a `cp` restore alone preserves an old mtime and MSBuild skips the rebuild — the control then "passes" on a stale dll). Build output checked for `error CS`, not just a green tail. On the Done-when checklist: box 4 (verify on prod/test with N simultaneous tune-ins) is only satisfiable **after** the image carrying this change is deployed, so it cannot be ticked pre-merge. I'll treat it as a post-deploy verification against `ersatztv-test` once `:latest` rebuilds, and report the arg-line count back here. Full .NET suite green locally; independent adversarial review in flight before the push.
timothy reopened this issue 2026-07-21 18:45:42 +02:00
Author
Owner

Closing — fixed, merged, and verified on a real deployment

Root cause. Structural, not a missing Interlocked. The write side already used Interlocked.Increment, which is exactly why the code read as thread-safe to every previous reader. The read side was a separate, earlier Volatile.Read in Run, with at least one await (the DB-backed limit read) between it and the claim. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed 0 < limit.

What shipped (PR #538, merged as 5e5da9db):

  • ErsatzTV.Application/Streaming/WorkAheadSlots.cs — new pool, TryAcquire(limit) claims by compare-exchange. CAS rather than the issue's suggested increment-then-back-off: both are correct on holder count, but increment-first transiently overshoots, and qsvExtraHardwareFrames sizing (#529) is derived from that bound.
  • HlsSessionWorker — Run claims at both call sites; Transcode(bool ownsWorkAheadSlot, …) derives realtime = !ownsWorkAheadSlot and releases in its existing finally. Acquisition stays in the caller because Run sets _state from the outcome and Transcode reads that state on entry (wasSeekAndWorkAhead) to pick the item start time — folding it inward would have silently flipped that branch.
  • An unbalanced Release clamps at zero and records UnbalancedReleases (added from review) — the pool is process-wide, so a -1 would have permanently inflated the budget.
  • ErsatzTV.Core.Tests/Streaming/WorkAheadSlotsTests.cs — 9 tests, 8 threads × 20 000 rounds.

Negative control. Reinstating the pre-fix check-then-act body: 15 912 over-claiming rounds of 20 000, 4 tests red. An independent reviewer replicated the harness separately and measured 8 966/20 000 — same order, different run. Restored with touch + forced rebuild, because a cp restore preserves the old mtime and MSBuild would have skipped the rebuild, "passing" the control on a stale dll.

Post-deploy verification (the box that was split out of Done-when — it was unsatisfiable pre-merge). Ran the identical 3-way simultaneous burst against ersatztv-test on jazz, workAheadSegmenterLimit: 1, once on the stale pre-fix image it happened to still be running and once after pulling the merged :latest:

pre-fix (8bbd7198) post-fix (eeb428a4)
work-ahead grants at tune-in 3 of 3 channels 1 (ch901), 902/903 throttled
arg lines without -readrate 12 of 25 1 of 4

So the original three-simultaneous-tune-in reproduction is now a genuine positive control rather than an anecdote, and the guarantee holds under exactly the burst it exists for.

Two measurement notes for whoever runs this next, both of which would have produced a wrong answer:

  • The deployed image's OCI labels are inherited from the linuxserver base image and claim version: 082dcbbe-ls75, created 2026-06-27. They are not evidence of what is running. The check that works is the running .Image against the registry's Docker-Content-Digest for the tag.
  • Container log lines carry a local-time bracket ([18:48:13 DBG]) while docker logs -t emits UTC, so --since windows silently mis-slice. The post-fix run used line-offset capture (wc -l before, tail -n +N after) instead.

ersatztv-test was left on the merged :latest with its settings unchanged; prod was untouched (verified identical container ID and StartedAt before and after).

Deferred: three Low findings from the re-review — a residual transient over-admission window a never-negative Release would close, the counter's floor-triggered under-count, and UnbalancedReleases not being surfaced in any log — are filed as #539. All three require an unbalanced release, which the single guarded release site makes unreachable today.

Docs: new decisions.md record ffmpeg.work-ahead-slot-atomic, catalog regenerated; plus a correction on the #350 record, whose "every concurrent tune-in falls back to the throttled path" bullet described the intent rather than the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered.

## Closing — fixed, merged, and verified on a real deployment **Root cause.** Structural, not a missing `Interlocked`. The write side already used `Interlocked.Increment`, which is exactly why the code read as thread-safe to every previous reader. The read side was a *separate, earlier* `Volatile.Read` in `Run`, with at least one `await` (the DB-backed limit read) between it and the claim. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed `0 < limit`. **What shipped** (PR #538, merged as `5e5da9db`): - `ErsatzTV.Application/Streaming/WorkAheadSlots.cs` — new pool, `TryAcquire(limit)` claims by compare-exchange. CAS rather than the issue's suggested increment-then-back-off: both are correct on holder count, but increment-first transiently overshoots, and `qsvExtraHardwareFrames` sizing (#529) is derived from that bound. - `HlsSessionWorker` — `Run` claims at both call sites; `Transcode(bool ownsWorkAheadSlot, …)` derives `realtime = !ownsWorkAheadSlot` and releases in its existing `finally`. Acquisition stays in the caller because `Run` sets `_state` from the outcome and `Transcode` reads that state on entry (`wasSeekAndWorkAhead`) to pick the item start time — folding it inward would have silently flipped that branch. - An unbalanced `Release` clamps at zero and records `UnbalancedReleases` (added from review) — the pool is process-wide, so a `-1` would have permanently inflated the budget. - `ErsatzTV.Core.Tests/Streaming/WorkAheadSlotsTests.cs` — 9 tests, 8 threads × 20 000 rounds. **Negative control.** Reinstating the pre-fix check-then-act body: **15 912 over-claiming rounds of 20 000**, 4 tests red. An independent reviewer replicated the harness separately and measured 8 966/20 000 — same order, different run. Restored with `touch` + forced rebuild, because a `cp` restore preserves the old mtime and MSBuild would have skipped the rebuild, "passing" the control on a stale dll. **Post-deploy verification** (the box that was split out of Done-when — it was unsatisfiable pre-merge). Ran the identical 3-way simultaneous burst against `ersatztv-test` on jazz, `workAheadSegmenterLimit: 1`, once on the stale pre-fix image it happened to still be running and once after pulling the merged `:latest`: | | pre-fix (`8bbd7198`) | post-fix (`eeb428a4`) | |---|---|---| | work-ahead grants at tune-in | **3 of 3** channels | **1** (ch901), 902/903 throttled | | arg lines without `-readrate` | 12 of 25 | **1 of 4** | So the original three-simultaneous-tune-in reproduction is now a genuine positive control rather than an anecdote, and the guarantee holds under exactly the burst it exists for. Two measurement notes for whoever runs this next, both of which would have produced a wrong answer: - The deployed image's OCI labels are **inherited from the linuxserver base image** and claim `version: 082dcbbe-ls75, created 2026-06-27`. They are not evidence of what is running. The check that works is the running `.Image` against the registry's `Docker-Content-Digest` for the tag. - Container log lines carry a **local-time** bracket (`[18:48:13 DBG]`) while `docker logs -t` emits **UTC**, so `--since` windows silently mis-slice. The post-fix run used line-offset capture (`wc -l` before, `tail -n +N` after) instead. `ersatztv-test` was left on the merged `:latest` with its settings unchanged; prod was untouched (verified identical container ID and `StartedAt` before and after). **Deferred:** three Low findings from the re-review — a residual transient over-admission window a never-negative `Release` would close, the counter's floor-triggered under-count, and `UnbalancedReleases` not being surfaced in any log — are filed as **#539**. All three require an unbalanced release, which the single guarded release site makes unreachable today. **Docs:** new `decisions.md` record `ffmpeg.work-ahead-slot-atomic`, catalog regenerated; plus a correction on the #350 record, whose "every concurrent tune-in falls back to the throttled path" bullet described the intent rather than the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered.
timothy removed the in-progress label 2026-07-21 19:09:52 +02:00
Author
Owner

Closing record

Outcome: workAheadSegmenterLimit is now enforced by an atomic compare-exchange claim on a shared WorkAheadSlots pool. PR #538, merged as 5e5da9db. Follow-up hardening deferred to #539.
Root cause: A check-then-act TOCTOU split across an await, not a missing Interlocked. The write side already used Interlocked.Increment — which is exactly why the code read as thread-safe to every previous reader — while the read side was a separate, earlier Volatile.Read in Run, with a DB-backed config read between them. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed 0 < limit and started unthrottled.
Decisions/conventions changed: added ffmpeg.work-ahead-slot-atomic. Corrected the ffmpeg.hls-cold-start-burst (#350) record: its "every concurrent tune-in falls back to the throttled path" bullet stated the intent, not the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered.
Reusable knowledge:

  • A resource guarantee expressed as check here, claim there is not a guarantee, however atomic the claim is. Grep for the read, not the write.
  • Ownership passed as a parameter (Transcode(bool ownsWorkAheadSlot, …)) beats ownership implied by a re-derived condition: one release site, guarded by the same flag the caller acquired on.
  • Restoring a mutated file with cp preserves its old mtime, so MSBuild skips the rebuild and a negative control "passes" against a stale dll. touch + explicit rebuild, and grep the build for error CS.
  • On jazz, a deployed image's OCI labels are inherited from the linuxserver base image (they claimed 2026-06-27 on a build minutes old). Compare the running .Image to the registry's Docker-Content-Digest instead.
  • ErsatzTV container logs carry a local-time bracket while docker logs -t emits UTC, so --since windows silently mis-slice. Capture by line offset (wc -l before, tail -n +N after).
  • ersatztv-test is a single-service Komodo compose project on jazz (port 8410); prod is a separate stack, and its api.key is root-owned, so an unsudo'd read fails silently into a 401 that parses as "0 channels".
    Verification: full .NET suite green pre-push; negative control 15912/20000 over-claiming rounds with the pre-fix body restored (a reviewer independently measured 8966/20000); two cold adversarial passes covering both commits; CI green on all jobs; and a live before/after on ersatztv-test — the identical 3-way simultaneous burst gave 3 of 3 channels unthrottled on the pre-fix image versus 1 (the configured limit) on the merged one.
    Deferred: #539 — three Low findings (a residual transient over-admission window a never-negative Release would close, the counter's floor-triggered under-count, and UnbalancedReleases not surfaced in any log). All require an unbalanced release, unreachable while there is one guarded release site.
    Docs updated: docs/decisions.md (+ regenerated docs/decisions/README.md). No API/route/SPA surface changed, so no OpenAPI regen or parity-doc update was owed.
## Closing record **Outcome:** `workAheadSegmenterLimit` is now enforced by an atomic compare-exchange claim on a shared `WorkAheadSlots` pool. PR #538, merged as `5e5da9db`. Follow-up hardening deferred to #539. **Root cause:** A check-then-act TOCTOU split across an `await`, not a missing `Interlocked`. The write side already used `Interlocked.Increment` — which is exactly why the code read as thread-safe to every previous reader — while the read side was a separate, earlier `Volatile.Read` in `Run`, with a DB-backed config read between them. An atomic operation on one half of a check-then-act buys nothing. Every simultaneous tune-in observed `0 < limit` and started unthrottled. **Decisions/conventions changed:** added `ffmpeg.work-ahead-slot-atomic`. Corrected the `ffmpeg.hls-cold-start-burst` (#350) record: its "every concurrent tune-in falls back to the throttled path" bullet stated the intent, not the behaviour — its measurement and conclusions stand, but the "one winner, two throttled" observation held only because those tunes were effectively staggered. **Reusable knowledge:** - A resource guarantee expressed as *check here, claim there* is not a guarantee, however atomic the claim is. Grep for the read, not the write. - Ownership passed as a parameter (`Transcode(bool ownsWorkAheadSlot, …)`) beats ownership implied by a re-derived condition: one release site, guarded by the same flag the caller acquired on. - Restoring a mutated file with `cp` preserves its old mtime, so MSBuild skips the rebuild and a negative control "passes" against a stale dll. `touch` + explicit rebuild, and grep the build for `error CS`. - On jazz, a deployed image's OCI labels are **inherited from the linuxserver base image** (they claimed `2026-06-27` on a build minutes old). Compare the running `.Image` to the registry's `Docker-Content-Digest` instead. - ErsatzTV container logs carry a **local-time** bracket while `docker logs -t` emits **UTC**, so `--since` windows silently mis-slice. Capture by line offset (`wc -l` before, `tail -n +N` after). - `ersatztv-test` is a single-service Komodo compose project on jazz (port 8410); prod is a separate stack, and its api.key is root-owned, so an unsudo'd read fails *silently* into a 401 that parses as "0 channels". **Verification:** full .NET suite green pre-push; negative control 15912/20000 over-claiming rounds with the pre-fix body restored (a reviewer independently measured 8966/20000); two cold adversarial passes covering both commits; CI green on all jobs; and a live before/after on `ersatztv-test` — the identical 3-way simultaneous burst gave **3 of 3** channels unthrottled on the pre-fix image versus **1** (the configured limit) on the merged one. **Deferred:** #539 — three Low findings (a residual transient over-admission window a never-negative `Release` would close, the counter's floor-triggered under-count, and `UnbalancedReleases` not surfaced in any log). All require an unbalanced release, unreachable while there is one guarded release site. **Docs updated:** `docs/decisions.md` (+ regenerated `docs/decisions/README.md`). No API/route/SPA surface changed, so no OpenAPI regen or parity-doc update was owed.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#536