test(444): deterministic functional-E2E for the playout-build lock 409 + isLocked projection #470

Merged
timothy merged 1 commits from feat/444-buildlock-e2e into main 2026-07-19 17:57:42 +02:00
3 changed files with 204 additions and 28 deletions
+16 -9
View File
@@ -291,8 +291,8 @@ re-run by hand, turning them into a CI regression net (ersatztv#299). It is the
**auth/CSRF/security-stamp** flow (setup-claim → read-gate 401/200 → re-claim 409 → CSRF 403 →
login 401/200 → logout 403/204 → post-logout stamp-revocation 401), the **library-scan status
contract** (404 unknown / 202 queued / `scan-status` 200), and the **If-Match/412** round-trip on
`rerun-collections`. Since ersatztv#363 it also asserts two **lock-contention 409s** that aren't
curl-only — see below. A `trap` kills the instance on step exit.
`rerun-collections`. Since ersatztv#363 (extended by #444) it also asserts three **lock-contention
409s** that aren't curl-only — see below. A `trap` kills the instance on step exit.
**Lock-contention 409s (ersatztv#363).** The harness now seeds rows the API can't create — a
`LibraryPath` and a Jellyfin media-source — directly into the running instance's SQLite DB (via
@@ -305,19 +305,26 @@ window is a strict subset of the scan lock's held window), then a second `POST .
(deterministic bar a tiny residual TOCTOU gap the multi-second scan covers); (b) the
**external-collections "already scanning" 409** — the per-family lock is taken
synchronously *before* the 202, so the 202 proves it held, and pointing the seeded source at a
non-routable address keeps the background sync hung so the window stays open. No new CI step or
dependency: ffmpeg ships in the toolchain image, and python3 was already a harness dependency
(`json_field`). The scan flow self-skips (advisory) if ffmpeg is ever absent.
non-routable address keeps the background sync hung so the window stays open; and (c) the
**playout-build "build in progress" 409 + `isLocked` projection** (#215/#444) — a build is enqueued
onto the single-consumer `WorkerService` and the trigger returns *before* the handler locks, so poll
`GET /playouts/{id}` until `isLocked:true` (not the accepted trigger), then fire. Seed a Classic Flood
schedule over a few short episodes and crank `PlayoutDaysToBuild` (`playout.days_to_build`) so one
build is wide enough to observe (~5 days ≈ 43k items ≈ ~1s locally, wider on slower CI); assert `PUT
/playouts/{id}` 409, `POST .../playout/reset` 409, the list `isLocked:true`, then post-build the same
`PUT` 200s. No new CI step or dependency: ffmpeg ships in the toolchain image, and python3 was already
a harness dependency (`json_field`). The scan + build flows self-skip (advisory) if ffmpeg is ever
absent, and the build flow also self-skips if the build is never observed locked (never asserts an
unproven race).
**Advisory, by design** (the issue's "keep it a separate job so a functional-E2E flake can't block the
unit-test gate"): it is **not** a `needs:` of `build` and **not (yet) a required check**, so a flake
blocks nothing. Promote it to a required check / `build` dependency once it's proven reliable — the
same staged rollout the `migrations` job used. SQLite is the default provider, so unlike `migrations`
it needs **no** DB service container. Runs on PRs and on `main` (regression net); skipped for `v*` tag
builds. Still out of scope (tracked as ersatztv#363 follow-ups): the **playout-build lock 409 +
`isLocked` projection** (#215) — needs a seeded schedule/playout and a build window wide enough to
observe without racing — and the genuinely **UI-interactive Playwright** flows (a separate CI
browser-tooling lift).
builds. Still out of scope (tracked as an ersatztv#363 follow-up): the genuinely **UI-interactive
Playwright** flows (a separate CI browser-tooling lift). The playout-build lock 409 + `isLocked`
projection (#215) landed in #444.
### Docs-only skip (ersatztv#416)
+20 -6
View File
@@ -148,7 +148,7 @@ kill "$PID"
each assertion even after a failure and prints a pass/fail summary, exiting non-zero if any failed.
What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browser); the
**lock-contention** section (added by ersatztv#363) is the exception — it seeds DB rows + media.
**lock-contention** section (added by ersatztv#363, extended by #444) is the exception — it seeds DB rows + media.
- **Legacy→SPA redirects**: a sweep of representative `LegacyUiRedirects` routes 302→`/app/*`, plus
the `/api` + `/artwork` never-redirect exemption (asserted as "did not 302 to `/app`", since those
4xx from their own handlers — `/api/*` 404s, `/artwork/*` 400s).
@@ -156,8 +156,8 @@ What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browse
`404`, `scan-status` `200`.
- **Optimistic concurrency**: create a collection + a `rerun-collections` targeting it → `GET` emits
an `ETag` → `PUT` with a stale `If-Match` `412`, current `200`, malformed `400`.
- **Lock contention (409)** — the two `IEntityLocker` 409s, made **deterministic** by firing the
racing request only once the lock is *provably* held (never a sleep-and-hope). Both seed rows the
- **Lock contention (409)** — the three `IEntityLocker` 409s, made **deterministic** by firing the
racing request only once the lock is *provably* held (never a sleep-and-hope). All seed rows the
API can't create straight into the running instance's DB via python3's stdlib `sqlite3` (whose
busy-timeout retry serializes behind the app's writer):
- **library-scan "already scanning"**: seed ~60 tiny ffmpeg clips into the built-in Shows library
@@ -172,15 +172,29 @@ What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browse
is taken synchronously *before* the `202`, so the `202` proves it held → `collections-scan-status`
shows the `jellyfin` family → a second `POST .../scan-collections` is `409`; unknown source `404`.
Needs neither ffmpeg nor the scanner.
- **playout-build "build in progress"** (#215/#444, the hardest): a build is enqueued onto the
single-consumer `WorkerService` channel and the trigger returns *before* `BuildPlayoutHandler`
dequeues and `LockPlayout`s (released in its `finally`), so an accepted trigger does **not** prove
the lock is held — poll `GET /playouts/{id}` until `isLocked:true`, then fire. Seed a few short
ffmpeg episodes → a Collection → a **Classic Flood** schedule → a Classic playout, and crank
`PlayoutDaysToBuild` (config `playout.days_to_build`) so a single build is wide enough to observe:
5 days ≈ 43k playout items ≈ ~1s locally, wider on slower CI (sized by measurement — item count is
`window / item-duration`, so the flood over a handful of short clips scales with the day count;
going *too* wide is counter-productive — a 777k-item build saturates the single worker with its
post-build gap/overlap jobs). While provably locked: `PUT /playouts/{id}` → 409, `POST
.../playout/reset` → 409, and the `GET /playouts` list projection shows `isLocked:true`; after the
build the lock clears (`isLocked:false`) and the same `PUT` now `200`s (proving the 409 is
lock-specific). Restores the config default afterwards. Self-skips (advisory) without ffmpeg, or if
the build is never observed locked (never asserts a race it can't prove it won).
- **Auth/CSRF/security-stamp** (mirrors the #295 manual set; runs **last** — setup-claim is one-shot
and logout revokes the session): fresh `auth/config` `setupRequired:true` → read-gate
401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation
403-without-`X-CSRF` → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the
same cookie is 401 (rotated security stamp).
**Still deferred** (ersatztv#363 follow-ups): the **playout-build lock 409 + `isLocked` projection**
(#215) — needs a seeded schedule/playout and a build window wide enough to observe without racing —
and the genuinely **UI-interactive Playwright** flows (a separate CI browser-tooling lift). When you
**Still deferred** (ersatztv#363 follow-up): the genuinely **UI-interactive Playwright** flows (a
separate CI browser-tooling lift). The playout-build lock 409 + `isLocked` projection (#215) landed in
#444 — see the lock-contention bullet above. When you
extend the harness, add the observed contract here and keep the assertions deterministic (probe the
real instance first — the initial cut caught that `/artwork/*` 400s where a guess said 404; the #363
cut measured the real scan window before sizing the seed).
+168 -13
View File
@@ -18,20 +18,18 @@
# 1. Legacy -> SPA redirect sweep (+ the /api,/artwork never-redirect exemption). [curl-only]
# 2. Machine-key data-plane: read-gate, library-scan lifecycle (404/202/scan-status),
# optimistic-concurrency If-Match/412. [curl-only]
# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress [seeds DB;
# (ersatztv#363) — see that section's header for how each is made deterministic. scan=ffmpeg]
# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress + playout-build [seeds DB;
# (ersatztv#363, #444) — see that section's header for how each is made deterministic. scan=ffmpeg]
# 4. Auth / CSRF / security-stamp (setup-claim, read-gate, CSRF gate, login, logout+revoke).
#
# Section 3 is the first that isn't curl-only: it seeds rows the API can't create (a LibraryPath and
# a media-source) straight into the running instance's SQLite DB (via python3's stdlib sqlite3, whose
# busy-timeout retry serializes behind the app's writer) and, for the scan flow, synthesizes media
# with ffmpeg. Both 409s
# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan flow is
# skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner.
# busy-timeout retry serializes behind the app's writer) and, for the scan/build flows, synthesizes
# media with ffmpeg. All three 409s
# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan + build
# flows are skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner.
#
# Deliberately still OUT of scope (tracked as ersatztv#363 follow-ups):
# - the playout-build lock 409 + isLocked projection (#215) — needs a seeded schedule/playout and a
# build window wide enough to observe without racing,
# - the genuinely UI-interactive Playwright (headless) flows — a separate CI browser-tooling lift.
#
# Exit status: 0 if every assertion passed, 1 if any failed. Assertions keep running after a
@@ -179,8 +177,8 @@ expect_status "PUT with the current If-Match -> 200" 200 PUT "$BASE_URL/api/v1/r
expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H 'If-Match: not-a-version' -d "$RC_BODY"
# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress -----------------------
# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363). Both are made
# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress + playout-build --------
# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363, #444). All three are made
# DETERMINISTIC (no sleep-and-hope) by only firing the racing request once the lock is provably held:
#
# A. Library-scan "already scanning" 409. A real scan must be in flight, so we seed a Shows library
@@ -197,12 +195,21 @@ expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/r
# address keeps the background sync hung so the window stays wide open. GET
# /media-sources/collections-scan-status corroborates.
#
# C. Playout-build "build in progress" 409 + isLocked projection (#215). Hardest of the three: the
# build is enqueued onto the single-consumer WorkerService channel and the trigger returns BEFORE
# BuildPlayoutHandler dequeues + LockPlayout (released in its finally -- BuildPlayoutHandler.cs), so
# an accepted trigger does NOT prove the lock is held. We POLL GET /playouts/{id} until isLocked:true,
# then fire the racing mutations while it is provably held. The build is made observable by seeding a
# Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild (config
# 'playout.days_to_build'): a single 5-day build is ~43k playout items ~= ~1s here, and CI runners are
# slower so the window only widens (#444 measurement). Restores the config default afterwards.
#
# Seeding writes rows the API can't create (LibraryPath, media-source) directly into the instance's
# SQLite DB via python3's stdlib sqlite3 (already a hard dep of this harness); the connection's
# busy-timeout retry serializes this second writer behind the app. Runs on the machine key, before the
# auth section (which claims the admin) -- these never claim it. Flow A is skipped (advisory) without
# ffmpeg; Flow B needs neither ffmpeg nor a scanner.
section "Lock contention (409): scan-in-progress + collections-in-progress"
# auth section (which claims the admin) -- these never claim it. Flows A + C are skipped (advisory)
# without ffmpeg; Flow B needs neither ffmpeg nor a scanner.
section "Lock contention (409): scan-in-progress + collections-in-progress + playout-build"
DB="$CONFIG_DIR/ersatztv.sqlite3"
@@ -307,6 +314,154 @@ else
else
bad "could not seed a Jellyfin media source row for the collections-lock flow"
fi
# --- Flow C: playout-build lock 409 + isLocked projection (#215/#444) -----------------------------
# Unlike Flows A/B, a playout build is enqueued onto the single-consumer WorkerService channel and the
# trigger returns BEFORE BuildPlayoutHandler dequeues and calls entityLocker.LockPlayout (released in its
# finally). So an accepted trigger does NOT prove the lock is held -- there is a real enqueue->dequeue
# race. We make it DETERMINISTIC by POLLING GET /playouts/{id} until isLocked:true (never sleep-and-hope),
# then firing the racing mutations while the lock is provably held. The build is made wide enough to
# observe by seeding a Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild:
# a single 5-day build is ~43k playout items ~= ~1s here (#444 measurement); CI runners are slower, so
# the window only widens. Needs ffmpeg (to synthesize scannable media) -> advisory skip without it.
playout_locked() { api "$BASE_URL/api/v1/playouts/$1" | python3 -c "import json,sys
try: print(json.load(sys.stdin).get('isLocked'))
except Exception: print('')"; }
# isLocked for PLID as seen in the GET /playouts LIST projection (the other place #215 stamps it).
list_locked() { api "$BASE_URL/api/v1/playouts?pageSize=100" | PLID="$1" python3 -c "import json,sys,os
try:
d=json.load(sys.stdin); pid=int(os.environ['PLID'])
print(next((str(p.get('isLocked')) for p in d.get('page',[]) if p.get('id')==pid),'MISSING'))
except Exception: print('ERR')"; }
# PlayoutDaysToBuild is ConfigElementKey 'playout.days_to_build' (default 2); widen it to size the build.
set_days_to_build() { python3 - "$DB" "$1" <<'PY'
import sqlite3, sys
con = sqlite3.connect(sys.argv[1], timeout=10) # busy-timeout retry serializes this second writer
con.execute("DELETE FROM ConfigElement WHERE Key='playout.days_to_build'")
if sys.argv[2] != 'CLEAR':
con.execute("INSERT INTO ConfigElement (Key, Value) VALUES ('playout.days_to_build', ?)", (sys.argv[2],))
con.commit()
PY
}
# expect_409_while_locked "desc" METHOD URL [curl args...] -- assert a build-locked mutation returns 409,
# but if the build RELEASED before our call landed (a 2xx AND the playout is now confirmed unlocked),
# advisory-skip instead of failing: a lost race is not a bug, and this flow never reds on a race it can't
# prove it won. A 2xx while the playout is STILL locked is a real defect and still fails. ($BL_PL in scope.)
expect_409_while_locked() {
local desc="$1" method="$2" url="$3"; shift 3
local got; got="$(status_of "$method" "$url" "$@")"
if [ "$got" = "409" ]; then
ok "$desc (=409)"
elif { [ "$got" = "200" ] || [ "$got" = "202" ]; } && [ "$(playout_locked "$BL_PL")" = "False" ]; then
skip "$desc — build released before the racing call landed (got $got, now unlocked) -> advisory"
else
bad "$desc — expected 409, got $got"
fi
}
if ! command -v ffmpeg >/dev/null 2>&1; then
skip "ffmpeg not found -> skipping the playout-build lock 409 flow (advisory; needs seeded media)"
else
BL_ROOT="$(mktemp -d)/tv"; BL_SEASON="$BL_ROOT/Build Lock E2E/Season 01"; mkdir -p "$BL_SEASON"
BL_SRC="$(mktemp -d)/bl.mkv"
if ffmpeg -y -loglevel error -f lavfi -i "testsrc=duration=10:size=160x120:rate=5" \
-c:v libx264 -pix_fmt yuv420p "$BL_SRC" >/dev/null 2>&1; then
# A few 10s episodes: the flood item repeats the collection to fill the whole build window, so item
# count (hence build duration) is window/item-duration -- 3 distinct items are plenty to flood with.
for n in 1 2 3; do cp "$BL_SRC" "$BL_SEASON/Build Lock E2E - s01e0$n.mkv"; done
if [ -z "$(seed_library_path "$BL_ROOT" 2)" ]; then
bad "could not seed a LibraryPath row for the build-lock flow (library 2)"
else
# Wait for any in-progress library-2 scan (Flow A's) to clear so our scan won't 409, then scan and
# wait for OUR titled episodes to land (filtered by title, so Flow A's 'Show Lock' items don't leak in).
for _ in $(seq 1 200); do
api "$BASE_URL/api/v1/libraries/scan-status" | grep -q '"libraryId":2,' || break
sleep 0.1
done
api -X POST "$BASE_URL/api/v1/libraries/2/scan" -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' -o /dev/null -w ''
BL_EPIDS=""
for _ in $(seq 1 200); do
BL_EPIDS=$(api "$BASE_URL/api/v1/library/browse?mediaType=Episode&pageSize=200" | python3 -c "import json,sys
d=json.load(sys.stdin)
print(','.join(str(i['mediaItemId']) for i in d.get('page',[]) if 'Build Lock E2E' in (i.get('title') or '')))")
[ -n "$BL_EPIDS" ] && break; sleep 0.15
done
if [ -z "$BL_EPIDS" ]; then
skip "build-lock flow: seeded episodes never scanned in within ~30s -> not asserting (advisory)"
else
ok "seeded + scanned build-lock episodes (ids: $BL_EPIDS)"
# Collection over the episodes -> a Classic Flood schedule item -> a Classic playout on a channel.
BL_COLL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/collections" -d '{"name":"E2E BuildLock Collection"}' -o "$BL_COLL_JSON" -w ''
BL_COLL="$(json_field "$BL_COLL_JSON" id)"
api -X POST "$BASE_URL/api/v1/collections/$BL_COLL/items" -d "{\"episodeIds\":[$BL_EPIDS]}" -o /dev/null -w ''
BL_SCH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/schedules" -d '{"name":"E2E BuildLock Schedule"}' -o "$BL_SCH_JSON" -w ''
BL_SCH="$(json_field "$BL_SCH_JSON" id)"
api -X PUT "$BASE_URL/api/v1/schedules/$BL_SCH/items" -o /dev/null -w '' -d "{\"items\":[{
\"startType\":\"Dynamic\",\"playoutMode\":\"Flood\",\"collectionType\":\"Collection\",\"collectionId\":$BL_COLL,
\"playbackOrder\":\"Chronological\",\"marathonGroupBy\":\"None\",\"marathonShuffleGroups\":false,
\"marathonShuffleItems\":false,\"fillWithGroupMode\":\"None\",\"multipleMode\":\"Count\",\"multipleCount\":\"\",
\"tailMode\":\"None\",\"customTitle\":\"\",\"guideMode\":\"Normal\",\"searchTitle\":\"\",\"searchQuery\":\"\",
\"preferredAudioLanguageCode\":\"\",\"preferredAudioTitle\":\"\",\"preferredSubtitleLanguageCode\":\"\",
\"watermarkIds\":[],\"graphicsElementIds\":[]}]}"
BL_CH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/channels" -o "$BL_CH_JSON" -w '' -d '{
"name":"E2E BuildLock Channel","number":"79999","group":"E2E","categories":"","ffmpegProfileId":1,
"streamSelectorMode":"Default","streamSelector":"","playoutSource":"Generated","playoutMode":"Continuous",
"streamingMode":"TransportStream","subtitleMode":"None","musicVideoCreditsMode":"None","songVideoMode":"Default",
"transcodeMode":"OnDemand","idleBehavior":"StopOnDisconnect","isEnabled":true,"showInEpg":true}'
BL_CH="$(json_field "$BL_CH_JSON" id)"
set_days_to_build 5
# Creating a Classic playout enqueues a Reset build (CreateClassicPlayoutHandler) -- the build we observe.
BL_PL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/playouts" -o "$BL_PL_JSON" -w '' \
-d "{\"channelId\":$BL_CH,\"scheduleKind\":\"Classic\",\"programScheduleId\":$BL_SCH}"
BL_PL="$(json_field "$BL_PL_JSON" id)"
if [ -z "$BL_PL" ]; then
bad "could not create the Classic playout for the build-lock flow (body: $(cat "$BL_PL_JSON"))"
else
bl_locked=""
for _ in $(seq 1 300); do
[ "$(playout_locked "$BL_PL")" = "True" ] && { bl_locked=1; break; }
sleep 0.05
done
if [ -z "$bl_locked" ]; then
# Never observed locked (build finished faster than we sampled): don't assert a race we can't
# prove we won -- advisory skip, matching Flow A's discipline.
skip "playout build never observed locked within ~15s -> not asserting the 409 (advisory; build too fast)"
else
ok "playout $BL_PL build observed locked (GET /playouts/{id} isLocked:true projection)"
# While the lock is provably held: every id-keyed mutation 409s and both isLocked projections
# agree. Each racing call is guarded (expect_409_while_locked / the list re-check below) so a
# build that finishes mid-flight degrades to an advisory skip, never a false red.
expect_409_while_locked "PUT /playouts/$BL_PL while build in progress -> 409" PUT "$BASE_URL/api/v1/playouts/$BL_PL" \
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}'
expect_409_while_locked "POST /channels/$BL_CH/playout/reset while build in progress -> 409" POST "$BASE_URL/api/v1/channels/$BL_CH/playout/reset?mode=Reset" \
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json'
bl_list="$(list_locked "$BL_PL")"
if [ "$bl_list" = "True" ]; then
ok "GET /playouts list projection shows isLocked:true for $BL_PL"
elif [ "$bl_list" = "False" ]; then
skip "GET /playouts list projection already isLocked:false (build released mid-flight) -> advisory"
else
bad "GET /playouts list projection error for $BL_PL (got '$bl_list')"
fi
# Once the build completes the lock clears: isLocked flips false and the SAME PUT now succeeds,
# proving the 409 above was lock-specific (not an always-failing call).
for _ in $(seq 1 600); do [ "$(playout_locked "$BL_PL")" = "False" ] && break; sleep 0.05; done
if [ "$(playout_locked "$BL_PL")" = "False" ]; then
ok "playout $BL_PL isLocked:false after the build completes"
else
bad "playout $BL_PL never unlocked after the build"
fi
expect_status "PUT /playouts/$BL_PL after the build (unlocked) -> 200" 200 PUT "$BASE_URL/api/v1/playouts/$BL_PL" \
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}'
fi
fi
set_days_to_build CLEAR # restore the default build horizon
fi
fi
else
bad "ffmpeg present but failed to synthesize the seed clip for the build-lock flow"
fi
fi
fi
# ---- 4. Auth / CSRF / security-stamp flow -------------------------------------------------------