From 038703fe67f9faa5c8f0f24d08d2756e3a8ac494 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 19 Jul 2026 17:37:50 +0200 Subject: [PATCH] test(444): deterministic functional-E2E for the playout-build lock 409 + isLocked projection Adds "Flow C" to scripts/e2e-functional.sh, the last deferred lock-contention flow from #363. A playout build is enqueued onto the single-consumer WorkerService channel and the trigger returns before BuildPlayoutHandler acquires the lock, so an accepted trigger does not prove the lock is held. Flow C makes it deterministic: seed a Classic Flood schedule over a few short ffmpeg episodes, crank PlayoutDaysToBuild=5 (~43k items ~= ~1s build), then POLL GET /playouts/{id} until isLocked:true before firing. Asserts PUT /playouts/{id} -> 409, reset -> 409, and the list-projection isLocked:true while locked; then isLocked:false + PUT -> 200 after the build (proving the 409 is lock-specific). Each racing assertion is guarded so a build that finishes mid-flight degrades to an advisory skip, never a false red; the whole flow self-skips without ffmpeg or if the build is never observed locked. Sized by measurement on a fresh instance -- going wider is counter-productive (a 777k-item build saturates the single worker with post-build gap/overlap jobs). Verified green across 6 fresh-instance runs; cold adversarial review MERGEABLE. Docs: docs/e2e-local.md + docs/ci-cd.md updated to describe Flow C and drop it from the "deferred" lists. fixes #444 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/ci-cd.md | 25 ++++-- docs/e2e-local.md | 26 ++++-- scripts/e2e-functional.sh | 181 +++++++++++++++++++++++++++++++++++--- 3 files changed, 204 insertions(+), 28 deletions(-) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index c3ced0597..2668621bb 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -291,8 +291,8 @@ re-run by hand, turning them into a CI regression net (ersatztv#299). It is the **auth/CSRF/security-stamp** flow (setup-claim → read-gate 401/200 → re-claim 409 → CSRF 403 → login 401/200 → logout 403/204 → post-logout stamp-revocation 401), the **library-scan status contract** (404 unknown / 202 queued / `scan-status` 200), and the **If-Match/412** round-trip on - `rerun-collections`. Since ersatztv#363 it also asserts two **lock-contention 409s** that aren't - curl-only — see below. A `trap` kills the instance on step exit. + `rerun-collections`. Since ersatztv#363 (extended by #444) it also asserts three **lock-contention + 409s** that aren't curl-only — see below. A `trap` kills the instance on step exit. **Lock-contention 409s (ersatztv#363).** The harness now seeds rows the API can't create — a `LibraryPath` and a Jellyfin media-source — directly into the running instance's SQLite DB (via @@ -305,19 +305,26 @@ window is a strict subset of the scan lock's held window), then a second `POST . (deterministic bar a tiny residual TOCTOU gap the multi-second scan covers); (b) the **external-collections "already scanning" 409** — the per-family lock is taken synchronously *before* the 202, so the 202 proves it held, and pointing the seeded source at a -non-routable address keeps the background sync hung so the window stays open. No new CI step or -dependency: ffmpeg ships in the toolchain image, and python3 was already a harness dependency -(`json_field`). The scan flow self-skips (advisory) if ffmpeg is ever absent. +non-routable address keeps the background sync hung so the window stays open; and (c) the +**playout-build "build in progress" 409 + `isLocked` projection** (#215/#444) — a build is enqueued +onto the single-consumer `WorkerService` and the trigger returns *before* the handler locks, so poll +`GET /playouts/{id}` until `isLocked:true` (not the accepted trigger), then fire. Seed a Classic Flood +schedule over a few short episodes and crank `PlayoutDaysToBuild` (`playout.days_to_build`) so one +build is wide enough to observe (~5 days ≈ 43k items ≈ ~1s locally, wider on slower CI); assert `PUT +/playouts/{id}` 409, `POST .../playout/reset` 409, the list `isLocked:true`, then post-build the same +`PUT` 200s. No new CI step or dependency: ffmpeg ships in the toolchain image, and python3 was already +a harness dependency (`json_field`). The scan + build flows self-skip (advisory) if ffmpeg is ever +absent, and the build flow also self-skips if the build is never observed locked (never asserts an +unproven race). **Advisory, by design** (the issue's "keep it a separate job so a functional-E2E flake can't block the unit-test gate"): it is **not** a `needs:` of `build` and **not (yet) a required check**, so a flake blocks nothing. Promote it to a required check / `build` dependency once it's proven reliable — the same staged rollout the `migrations` job used. SQLite is the default provider, so unlike `migrations` it needs **no** DB service container. Runs on PRs and on `main` (regression net); skipped for `v*` tag -builds. Still out of scope (tracked as ersatztv#363 follow-ups): the **playout-build lock 409 + -`isLocked` projection** (#215) — needs a seeded schedule/playout and a build window wide enough to -observe without racing — and the genuinely **UI-interactive Playwright** flows (a separate CI -browser-tooling lift). +builds. Still out of scope (tracked as an ersatztv#363 follow-up): the genuinely **UI-interactive +Playwright** flows (a separate CI browser-tooling lift). The playout-build lock 409 + `isLocked` +projection (#215) landed in #444. ### Docs-only skip (ersatztv#416) diff --git a/docs/e2e-local.md b/docs/e2e-local.md index d883a2b48..0fe60563c 100644 --- a/docs/e2e-local.md +++ b/docs/e2e-local.md @@ -148,7 +148,7 @@ kill "$PID" each assertion even after a failure and prints a pass/fail summary, exiting non-zero if any failed. What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browser); the -**lock-contention** section (added by ersatztv#363) is the exception — it seeds DB rows + media. +**lock-contention** section (added by ersatztv#363, extended by #444) is the exception — it seeds DB rows + media. - **Legacy→SPA redirects**: a sweep of representative `LegacyUiRedirects` routes 302→`/app/*`, plus the `/api` + `/artwork` never-redirect exemption (asserted as "did not 302 to `/app`", since those 4xx from their own handlers — `/api/*` 404s, `/artwork/*` 400s). @@ -156,8 +156,8 @@ What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browse `404`, `scan-status` `200`. - **Optimistic concurrency**: create a collection + a `rerun-collections` targeting it → `GET` emits an `ETag` → `PUT` with a stale `If-Match` `412`, current `200`, malformed `400`. -- **Lock contention (409)** — the two `IEntityLocker` 409s, made **deterministic** by firing the - racing request only once the lock is *provably* held (never a sleep-and-hope). Both seed rows the +- **Lock contention (409)** — the three `IEntityLocker` 409s, made **deterministic** by firing the + racing request only once the lock is *provably* held (never a sleep-and-hope). All seed rows the API can't create straight into the running instance's DB via python3's stdlib `sqlite3` (whose busy-timeout retry serializes behind the app's writer): - **library-scan "already scanning"**: seed ~60 tiny ffmpeg clips into the built-in Shows library @@ -172,15 +172,29 @@ What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browse is taken synchronously *before* the `202`, so the `202` proves it held → `collections-scan-status` shows the `jellyfin` family → a second `POST .../scan-collections` is `409`; unknown source `404`. Needs neither ffmpeg nor the scanner. + - **playout-build "build in progress"** (#215/#444, the hardest): a build is enqueued onto the + single-consumer `WorkerService` channel and the trigger returns *before* `BuildPlayoutHandler` + dequeues and `LockPlayout`s (released in its `finally`), so an accepted trigger does **not** prove + the lock is held — poll `GET /playouts/{id}` until `isLocked:true`, then fire. Seed a few short + ffmpeg episodes → a Collection → a **Classic Flood** schedule → a Classic playout, and crank + `PlayoutDaysToBuild` (config `playout.days_to_build`) so a single build is wide enough to observe: + 5 days ≈ 43k playout items ≈ ~1s locally, wider on slower CI (sized by measurement — item count is + `window / item-duration`, so the flood over a handful of short clips scales with the day count; + going *too* wide is counter-productive — a 777k-item build saturates the single worker with its + post-build gap/overlap jobs). While provably locked: `PUT /playouts/{id}` → 409, `POST + .../playout/reset` → 409, and the `GET /playouts` list projection shows `isLocked:true`; after the + build the lock clears (`isLocked:false`) and the same `PUT` now `200`s (proving the 409 is + lock-specific). Restores the config default afterwards. Self-skips (advisory) without ffmpeg, or if + the build is never observed locked (never asserts a race it can't prove it won). - **Auth/CSRF/security-stamp** (mirrors the #295 manual set; runs **last** — setup-claim is one-shot and logout revokes the session): fresh `auth/config` `setupRequired:true` → read-gate 401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation 403-without-`X-CSRF` → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the same cookie is 401 (rotated security stamp). -**Still deferred** (ersatztv#363 follow-ups): the **playout-build lock 409 + `isLocked` projection** -(#215) — needs a seeded schedule/playout and a build window wide enough to observe without racing — -and the genuinely **UI-interactive Playwright** flows (a separate CI browser-tooling lift). When you +**Still deferred** (ersatztv#363 follow-up): the genuinely **UI-interactive Playwright** flows (a +separate CI browser-tooling lift). The playout-build lock 409 + `isLocked` projection (#215) landed in +#444 — see the lock-contention bullet above. When you extend the harness, add the observed contract here and keep the assertions deterministic (probe the real instance first — the initial cut caught that `/artwork/*` 400s where a guess said 404; the #363 cut measured the real scan window before sizing the seed). diff --git a/scripts/e2e-functional.sh b/scripts/e2e-functional.sh index 39d0ceccb..1b74d7ec8 100755 --- a/scripts/e2e-functional.sh +++ b/scripts/e2e-functional.sh @@ -18,20 +18,18 @@ # 1. Legacy -> SPA redirect sweep (+ the /api,/artwork never-redirect exemption). [curl-only] # 2. Machine-key data-plane: read-gate, library-scan lifecycle (404/202/scan-status), # optimistic-concurrency If-Match/412. [curl-only] -# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress [seeds DB; -# (ersatztv#363) — see that section's header for how each is made deterministic. scan=ffmpeg] +# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress + playout-build [seeds DB; +# (ersatztv#363, #444) — see that section's header for how each is made deterministic. scan=ffmpeg] # 4. Auth / CSRF / security-stamp (setup-claim, read-gate, CSRF gate, login, logout+revoke). # # Section 3 is the first that isn't curl-only: it seeds rows the API can't create (a LibraryPath and # a media-source) straight into the running instance's SQLite DB (via python3's stdlib sqlite3, whose -# busy-timeout retry serializes behind the app's writer) and, for the scan flow, synthesizes media -# with ffmpeg. Both 409s -# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan flow is -# skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner. +# busy-timeout retry serializes behind the app's writer) and, for the scan/build flows, synthesizes +# media with ffmpeg. All three 409s +# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan + build +# flows are skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner. # # Deliberately still OUT of scope (tracked as ersatztv#363 follow-ups): -# - the playout-build lock 409 + isLocked projection (#215) — needs a seeded schedule/playout and a -# build window wide enough to observe without racing, # - the genuinely UI-interactive Playwright (headless) flows — a separate CI browser-tooling lift. # # Exit status: 0 if every assertion passed, 1 if any failed. Assertions keep running after a @@ -179,8 +177,8 @@ expect_status "PUT with the current If-Match -> 200" 200 PUT "$BASE_URL/api/v1/r expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \ -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H 'If-Match: not-a-version' -d "$RC_BODY" -# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress ----------------------- -# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363). Both are made +# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress + playout-build -------- +# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363, #444). All three are made # DETERMINISTIC (no sleep-and-hope) by only firing the racing request once the lock is provably held: # # A. Library-scan "already scanning" 409. A real scan must be in flight, so we seed a Shows library @@ -197,12 +195,21 @@ expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/r # address keeps the background sync hung so the window stays wide open. GET # /media-sources/collections-scan-status corroborates. # +# C. Playout-build "build in progress" 409 + isLocked projection (#215). Hardest of the three: the +# build is enqueued onto the single-consumer WorkerService channel and the trigger returns BEFORE +# BuildPlayoutHandler dequeues + LockPlayout (released in its finally -- BuildPlayoutHandler.cs), so +# an accepted trigger does NOT prove the lock is held. We POLL GET /playouts/{id} until isLocked:true, +# then fire the racing mutations while it is provably held. The build is made observable by seeding a +# Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild (config +# 'playout.days_to_build'): a single 5-day build is ~43k playout items ~= ~1s here, and CI runners are +# slower so the window only widens (#444 measurement). Restores the config default afterwards. +# # Seeding writes rows the API can't create (LibraryPath, media-source) directly into the instance's # SQLite DB via python3's stdlib sqlite3 (already a hard dep of this harness); the connection's # busy-timeout retry serializes this second writer behind the app. Runs on the machine key, before the -# auth section (which claims the admin) -- these never claim it. Flow A is skipped (advisory) without -# ffmpeg; Flow B needs neither ffmpeg nor a scanner. -section "Lock contention (409): scan-in-progress + collections-in-progress" +# auth section (which claims the admin) -- these never claim it. Flows A + C are skipped (advisory) +# without ffmpeg; Flow B needs neither ffmpeg nor a scanner. +section "Lock contention (409): scan-in-progress + collections-in-progress + playout-build" DB="$CONFIG_DIR/ersatztv.sqlite3" @@ -307,6 +314,154 @@ else else bad "could not seed a Jellyfin media source row for the collections-lock flow" fi + + # --- Flow C: playout-build lock 409 + isLocked projection (#215/#444) ----------------------------- + # Unlike Flows A/B, a playout build is enqueued onto the single-consumer WorkerService channel and the + # trigger returns BEFORE BuildPlayoutHandler dequeues and calls entityLocker.LockPlayout (released in its + # finally). So an accepted trigger does NOT prove the lock is held -- there is a real enqueue->dequeue + # race. We make it DETERMINISTIC by POLLING GET /playouts/{id} until isLocked:true (never sleep-and-hope), + # then firing the racing mutations while the lock is provably held. The build is made wide enough to + # observe by seeding a Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild: + # a single 5-day build is ~43k playout items ~= ~1s here (#444 measurement); CI runners are slower, so + # the window only widens. Needs ffmpeg (to synthesize scannable media) -> advisory skip without it. + playout_locked() { api "$BASE_URL/api/v1/playouts/$1" | python3 -c "import json,sys +try: print(json.load(sys.stdin).get('isLocked')) +except Exception: print('')"; } + # isLocked for PLID as seen in the GET /playouts LIST projection (the other place #215 stamps it). + list_locked() { api "$BASE_URL/api/v1/playouts?pageSize=100" | PLID="$1" python3 -c "import json,sys,os +try: + d=json.load(sys.stdin); pid=int(os.environ['PLID']) + print(next((str(p.get('isLocked')) for p in d.get('page',[]) if p.get('id')==pid),'MISSING')) +except Exception: print('ERR')"; } + # PlayoutDaysToBuild is ConfigElementKey 'playout.days_to_build' (default 2); widen it to size the build. + set_days_to_build() { python3 - "$DB" "$1" <<'PY' +import sqlite3, sys +con = sqlite3.connect(sys.argv[1], timeout=10) # busy-timeout retry serializes this second writer +con.execute("DELETE FROM ConfigElement WHERE Key='playout.days_to_build'") +if sys.argv[2] != 'CLEAR': + con.execute("INSERT INTO ConfigElement (Key, Value) VALUES ('playout.days_to_build', ?)", (sys.argv[2],)) +con.commit() +PY + } + # expect_409_while_locked "desc" METHOD URL [curl args...] -- assert a build-locked mutation returns 409, + # but if the build RELEASED before our call landed (a 2xx AND the playout is now confirmed unlocked), + # advisory-skip instead of failing: a lost race is not a bug, and this flow never reds on a race it can't + # prove it won. A 2xx while the playout is STILL locked is a real defect and still fails. ($BL_PL in scope.) + expect_409_while_locked() { + local desc="$1" method="$2" url="$3"; shift 3 + local got; got="$(status_of "$method" "$url" "$@")" + if [ "$got" = "409" ]; then + ok "$desc (=409)" + elif { [ "$got" = "200" ] || [ "$got" = "202" ]; } && [ "$(playout_locked "$BL_PL")" = "False" ]; then + skip "$desc — build released before the racing call landed (got $got, now unlocked) -> advisory" + else + bad "$desc — expected 409, got $got" + fi + } + + if ! command -v ffmpeg >/dev/null 2>&1; then + skip "ffmpeg not found -> skipping the playout-build lock 409 flow (advisory; needs seeded media)" + else + BL_ROOT="$(mktemp -d)/tv"; BL_SEASON="$BL_ROOT/Build Lock E2E/Season 01"; mkdir -p "$BL_SEASON" + BL_SRC="$(mktemp -d)/bl.mkv" + if ffmpeg -y -loglevel error -f lavfi -i "testsrc=duration=10:size=160x120:rate=5" \ + -c:v libx264 -pix_fmt yuv420p "$BL_SRC" >/dev/null 2>&1; then + # A few 10s episodes: the flood item repeats the collection to fill the whole build window, so item + # count (hence build duration) is window/item-duration -- 3 distinct items are plenty to flood with. + for n in 1 2 3; do cp "$BL_SRC" "$BL_SEASON/Build Lock E2E - s01e0$n.mkv"; done + if [ -z "$(seed_library_path "$BL_ROOT" 2)" ]; then + bad "could not seed a LibraryPath row for the build-lock flow (library 2)" + else + # Wait for any in-progress library-2 scan (Flow A's) to clear so our scan won't 409, then scan and + # wait for OUR titled episodes to land (filtered by title, so Flow A's 'Show Lock' items don't leak in). + for _ in $(seq 1 200); do + api "$BASE_URL/api/v1/libraries/scan-status" | grep -q '"libraryId":2,' || break + sleep 0.1 + done + api -X POST "$BASE_URL/api/v1/libraries/2/scan" -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' -o /dev/null -w '' + BL_EPIDS="" + for _ in $(seq 1 200); do + BL_EPIDS=$(api "$BASE_URL/api/v1/library/browse?mediaType=Episode&pageSize=200" | python3 -c "import json,sys +d=json.load(sys.stdin) +print(','.join(str(i['mediaItemId']) for i in d.get('page',[]) if 'Build Lock E2E' in (i.get('title') or '')))") + [ -n "$BL_EPIDS" ] && break; sleep 0.15 + done + if [ -z "$BL_EPIDS" ]; then + skip "build-lock flow: seeded episodes never scanned in within ~30s -> not asserting (advisory)" + else + ok "seeded + scanned build-lock episodes (ids: $BL_EPIDS)" + # Collection over the episodes -> a Classic Flood schedule item -> a Classic playout on a channel. + BL_COLL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/collections" -d '{"name":"E2E BuildLock Collection"}' -o "$BL_COLL_JSON" -w '' + BL_COLL="$(json_field "$BL_COLL_JSON" id)" + api -X POST "$BASE_URL/api/v1/collections/$BL_COLL/items" -d "{\"episodeIds\":[$BL_EPIDS]}" -o /dev/null -w '' + BL_SCH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/schedules" -d '{"name":"E2E BuildLock Schedule"}' -o "$BL_SCH_JSON" -w '' + BL_SCH="$(json_field "$BL_SCH_JSON" id)" + api -X PUT "$BASE_URL/api/v1/schedules/$BL_SCH/items" -o /dev/null -w '' -d "{\"items\":[{ + \"startType\":\"Dynamic\",\"playoutMode\":\"Flood\",\"collectionType\":\"Collection\",\"collectionId\":$BL_COLL, + \"playbackOrder\":\"Chronological\",\"marathonGroupBy\":\"None\",\"marathonShuffleGroups\":false, + \"marathonShuffleItems\":false,\"fillWithGroupMode\":\"None\",\"multipleMode\":\"Count\",\"multipleCount\":\"\", + \"tailMode\":\"None\",\"customTitle\":\"\",\"guideMode\":\"Normal\",\"searchTitle\":\"\",\"searchQuery\":\"\", + \"preferredAudioLanguageCode\":\"\",\"preferredAudioTitle\":\"\",\"preferredSubtitleLanguageCode\":\"\", + \"watermarkIds\":[],\"graphicsElementIds\":[]}]}" + BL_CH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/channels" -o "$BL_CH_JSON" -w '' -d '{ + "name":"E2E BuildLock Channel","number":"79999","group":"E2E","categories":"","ffmpegProfileId":1, + "streamSelectorMode":"Default","streamSelector":"","playoutSource":"Generated","playoutMode":"Continuous", + "streamingMode":"TransportStream","subtitleMode":"None","musicVideoCreditsMode":"None","songVideoMode":"Default", + "transcodeMode":"OnDemand","idleBehavior":"StopOnDisconnect","isEnabled":true,"showInEpg":true}' + BL_CH="$(json_field "$BL_CH_JSON" id)" + set_days_to_build 5 + # Creating a Classic playout enqueues a Reset build (CreateClassicPlayoutHandler) -- the build we observe. + BL_PL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/playouts" -o "$BL_PL_JSON" -w '' \ + -d "{\"channelId\":$BL_CH,\"scheduleKind\":\"Classic\",\"programScheduleId\":$BL_SCH}" + BL_PL="$(json_field "$BL_PL_JSON" id)" + if [ -z "$BL_PL" ]; then + bad "could not create the Classic playout for the build-lock flow (body: $(cat "$BL_PL_JSON"))" + else + bl_locked="" + for _ in $(seq 1 300); do + [ "$(playout_locked "$BL_PL")" = "True" ] && { bl_locked=1; break; } + sleep 0.05 + done + if [ -z "$bl_locked" ]; then + # Never observed locked (build finished faster than we sampled): don't assert a race we can't + # prove we won -- advisory skip, matching Flow A's discipline. + skip "playout build never observed locked within ~15s -> not asserting the 409 (advisory; build too fast)" + else + ok "playout $BL_PL build observed locked (GET /playouts/{id} isLocked:true projection)" + # While the lock is provably held: every id-keyed mutation 409s and both isLocked projections + # agree. Each racing call is guarded (expect_409_while_locked / the list re-check below) so a + # build that finishes mid-flight degrades to an advisory skip, never a false red. + expect_409_while_locked "PUT /playouts/$BL_PL while build in progress -> 409" PUT "$BASE_URL/api/v1/playouts/$BL_PL" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}' + expect_409_while_locked "POST /channels/$BL_CH/playout/reset while build in progress -> 409" POST "$BASE_URL/api/v1/channels/$BL_CH/playout/reset?mode=Reset" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' + bl_list="$(list_locked "$BL_PL")" + if [ "$bl_list" = "True" ]; then + ok "GET /playouts list projection shows isLocked:true for $BL_PL" + elif [ "$bl_list" = "False" ]; then + skip "GET /playouts list projection already isLocked:false (build released mid-flight) -> advisory" + else + bad "GET /playouts list projection error for $BL_PL (got '$bl_list')" + fi + # Once the build completes the lock clears: isLocked flips false and the SAME PUT now succeeds, + # proving the 409 above was lock-specific (not an always-failing call). + for _ in $(seq 1 600); do [ "$(playout_locked "$BL_PL")" = "False" ] && break; sleep 0.05; done + if [ "$(playout_locked "$BL_PL")" = "False" ]; then + ok "playout $BL_PL isLocked:false after the build completes" + else + bad "playout $BL_PL never unlocked after the build" + fi + expect_status "PUT /playouts/$BL_PL after the build (unlocked) -> 200" 200 PUT "$BASE_URL/api/v1/playouts/$BL_PL" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}' + fi + fi + set_days_to_build CLEAR # restore the default build horizon + fi + fi + else + bad "ffmpeg present but failed to synthesize the seed clip for the build-lock flow" + fi + fi fi # ---- 4. Auth / CSRF / security-stamp flow ------------------------------------------------------- -- 2.47.3