test(444): deterministic functional-E2E for the playout-build lock 409 + isLocked projection #470

2026-07-19T17:38:28+02:00

timothy commented

2026-07-19 17:38:28 +02:00

Closes #444 — the last deferred lock-contention flow from #363.

What

Adds Flow C to scripts/e2e-functional.sh: a deterministic functional-E2E assertion for the playout-build lock 409 + isLocked projection (#215).

The hard part (why #363 deferred it): a playout build is enqueued onto the single-consumer WorkerService channel and the trigger returns before BuildPlayoutHandler acquires the lock (released in its finally), so an accepted trigger does not prove the lock is held — there's a real enqueue→dequeue race.

Flow C makes it race-free:

Seed a few short ffmpeg episodes → a real Collection → a Classic Flood schedule → a Classic playout (creating it enqueues a Reset build).
Crank PlayoutDaysToBuild=5 (config playout.days_to_build) so a single build is wide enough to observe (~43k playout items ≈ ~1s here; CI runners are slower, so the window only widens).
Poll GET /playouts/{id} until isLocked:true (never a sleep), then assert while provably locked:
- PUT /playouts/{id} → 409
- POST /channels/{id}/playout/reset → 409
- GET /playouts list projection shows isLocked:true
After the build: isLocked:false and the same PUT → 200 (proves the 409 is lock-specific, not an always-failing call).

Each racing assertion is guarded: if the build finishes mid-flight it degrades to an advisory skip, never a false red. The whole flow self-skips without ffmpeg, or if the build is never observed locked.

Sizing (measured, not guessed)

On a fresh playout the build time is window / item-duration, linear in the day count. Measured: 2d→0.36s/17k, 5d→~1s/43k, 30d→2.5s/259k. Going wider is counter-productive — a 777k-item build saturates the single worker with its post-build gap/overlap jobs and starves later builds. 5 days is the lightest setting with a comfortable, reliable window.

Verification

Green across 6 fresh-instance runs (3 isolated Flow C + 3 full-harness), all 45/45.
Cold-context adversarial review: MERGEABLE. Its one non-blocking finding (racing assertions hard-failed on a lost race) is fixed here — the guarded expect_409_while_locked / list re-check now downgrade a mid-flight release to an advisory skip.
No .cs touched; no API/route/SPA change → no OpenAPI regen. Docs updated: docs/e2e-local.md + docs/ci-cd.md (describe Flow C, drop it from the deferred lists).

Still deferred

Only the UI-interactive Playwright (headless) flows remain out of scope (separate browser-tooling lift).

Closes #444 — the last deferred lock-contention flow from #363. ## What Adds **Flow C** to `scripts/e2e-functional.sh`: a deterministic functional-E2E assertion for the **playout-build lock 409 + `isLocked` projection** (#215). The hard part (why #363 deferred it): a playout build is enqueued onto the single-consumer `WorkerService` channel and the trigger returns **before** `BuildPlayoutHandler` acquires the lock (released in its `finally`), so an accepted trigger does **not** prove the lock is held — there's a real enqueue→dequeue race. Flow C makes it race-free: 1. Seed a few short ffmpeg episodes → a real Collection → a **Classic Flood** schedule → a Classic playout (creating it enqueues a Reset build). 2. Crank `PlayoutDaysToBuild=5` (config `playout.days_to_build`) so a single build is wide enough to observe (~43k playout items ≈ **~1s** here; CI runners are slower, so the window only widens). 3. **Poll `GET /playouts/{id}` until `isLocked:true`** (never a `sleep`), then assert while provably locked: - `PUT /playouts/{id}` → **409** - `POST /channels/{id}/playout/reset` → **409** - `GET /playouts` list projection shows `isLocked:true` 4. After the build: `isLocked:false` and the same `PUT` → **200** (proves the 409 is lock-specific, not an always-failing call). Each racing assertion is **guarded**: if the build finishes mid-flight it degrades to an advisory **skip**, never a false red. The whole flow self-skips without ffmpeg, or if the build is never observed locked. ## Sizing (measured, not guessed) On a fresh playout the build time is `window / item-duration`, linear in the day count. Measured: 2d→0.36s/17k, 5d→~1s/43k, 30d→2.5s/259k. **Going wider is counter-productive** — a 777k-item build saturates the single worker with its post-build gap/overlap jobs and starves later builds. 5 days is the lightest setting with a comfortable, reliable window. ## Verification - Green across **6 fresh-instance runs** (3 isolated Flow C + 3 full-harness), all 45/45. - Cold-context adversarial review: **MERGEABLE**. Its one non-blocking finding (racing assertions hard-failed on a lost race) is **fixed** here — the guarded `expect_409_while_locked` / list re-check now downgrade a mid-flight release to an advisory skip. - No `.cs` touched; no API/route/SPA change → no OpenAPI regen. Docs updated: `docs/e2e-local.md` + `docs/ci-cd.md` (describe Flow C, drop it from the deferred lists). ## Still deferred Only the UI-interactive Playwright (headless) flows remain out of scope (separate browser-tooling lift).

timothy added 1 commit 2026-07-19 17:38:29 +02:00

test(444): deterministic functional-E2E for the playout-build lock 409 + isLocked projection

Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 10s

Details

Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 14s

Details

Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 9s

Details

Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 33s

Details

Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m24s

Details

Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 15m4s

Details

Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 18m25s

Details

Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 18m37s

Details

Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped

Details

038703fe67

Adds "Flow C" to scripts/e2e-functional.sh, the last deferred lock-contention flow from #363.
A playout build is enqueued onto the single-consumer WorkerService channel and the trigger
returns before BuildPlayoutHandler acquires the lock, so an accepted trigger does not prove the
lock is held. Flow C makes it deterministic: seed a Classic Flood schedule over a few short
ffmpeg episodes, crank PlayoutDaysToBuild=5 (~43k items ~= ~1s build), then POLL GET
/playouts/{id} until isLocked:true before firing. Asserts PUT /playouts/{id} -> 409, reset ->
409, and the list-projection isLocked:true while locked; then isLocked:false + PUT -> 200 after
the build (proving the 409 is lock-specific). Each racing assertion is guarded so a build that
finishes mid-flight degrades to an advisory skip, never a false red; the whole flow self-skips
without ffmpeg or if the build is never observed locked.

Sized by measurement on a fresh instance -- going wider is counter-productive (a 777k-item build
saturates the single worker with post-build gap/overlap jobs). Verified green across 6
fresh-instance runs; cold adversarial review MERGEABLE.

Docs: docs/e2e-local.md + docs/ci-cd.md updated to describe Flow C and drop it from the
"deferred" lists.

fixes #444

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

timothy commented

2026-07-19 17:47:40 +02:00

Review-verdict: MERGEABLE @ 038703fe67

Two cold-context adversarial passes, both MERGEABLE:

Initial pass over the full Flow C diff — MERGEABLE with one non-blocking finding: the racing 409 assertions hard-failed if the build released between "observe locked" and "fire the mutation" (a lost race read as a red run, contrary to the harness's "never assert a race we can't prove we won" discipline).
Fix pass over the hardening delta (expect_409_while_locked + the list re-check, in this same commit) — MERGEABLE. Confirmed: a 2xx while still locked still hard-fails (a real guard regression stays red), and only a 2xx with the playout confirmed unlocked downgrades to an advisory skip. set -u clean; happy path unchanged.

Determinism validated by measurement + 6 green fresh-instance runs (3 isolated Flow C + 3 full-harness, 45/45). Remaining reviewer observations (sub-ms theoretical skip window; pageSize=100 list cap unreachable with a handful of playouts) are non-blocking and inherent to the advisory-skip design.

Review-verdict: MERGEABLE @ 038703fe67f9faa5c8f0f24d08d2756e3a8ac494 Two cold-context adversarial passes, both MERGEABLE: 1. **Initial pass** over the full Flow C diff — MERGEABLE with one non-blocking finding: the racing 409 assertions hard-failed if the build released between "observe locked" and "fire the mutation" (a lost race read as a red run, contrary to the harness's "never assert a race we can't prove we won" discipline). 2. **Fix pass** over the hardening delta (`expect_409_while_locked` + the list re-check, in this same commit) — MERGEABLE. Confirmed: a 2xx **while still locked** still hard-fails (a real guard regression stays red), and only a 2xx with the playout **confirmed unlocked** downgrades to an advisory skip. `set -u` clean; happy path unchanged. Determinism validated by measurement + 6 green fresh-instance runs (3 isolated Flow C + 3 full-harness, 45/45). Remaining reviewer observations (sub-ms theoretical skip window; `pageSize=100` list cap unreachable with a handful of playouts) are non-blocking and inherent to the advisory-skip design.

timothy merged commit e8d730359a into main

2026-07-19 17:57:42 +02:00

timothy deleted branch feat/444-buildlock-e2e

2026-07-19 17:57:42 +02:00

timothy referenced this issue from a commit

2026-07-19 17:57:42 +02:00

Merge pull request 'test(444): deterministic functional-E2E for the playout-build lock 409 + isLocked projection' (#470) from feat/444-buildlock-e2e into main

timothy referenced this pull request

2026-07-19 17:58:22 +02:00

functional-E2E: add the playout-build lock 409 + isLocked projection flow (deferred from #363) #444

timothy referenced this pull request

2026-07-19 17:58:54 +02:00