From 8a85f9ddb502eff78f150ac175e3b5815219da26 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sat, 18 Jul 2026 15:17:21 +0200 Subject: [PATCH] =?UTF-8?q?test(363):=20functional-E2E=20harness=20?= =?UTF-8?q?=E2=80=94=20add=20deterministic=20scan-lock=20+=20collections-l?= =?UTF-8?q?ock=20409=20flows?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends scripts/e2e-functional.sh with the two IEntityLocker 409 contracts the first cut (ersatztv#299) deferred as "racy", made DETERMINISTIC by firing the racing request only once the lock is provably held (no sleep-and-hope): - library-scan "already scanning" 409: seed ~60 tiny ffmpeg clips into the built-in Shows library so the scanner subprocess runs a few seconds, poll GET /libraries/scan-status until the library is active (that window is a strict subset of the scan lock's held window — StartScan after LockLibrary, EndScan before UnlockLibrary), then a second POST .../scan is a guaranteed 409. Self-skips (advisory) when ffmpeg is absent. - external-collections "already scanning" 409: seed a Jellyfin media-source row pointing at a non-routable address so the background sync hangs and the per-family lock stays held; the lock is taken synchronously before the 202, so the 202 proves it held. collections-scan-status corroborates; unknown source 404. Seeding uses python3's stdlib sqlite3 (already a harness dep) to insert rows the API can't create (LibraryPath, media-source); WAL mode tolerates the second writer. No new CI step/dependency — ffmpeg ships in the toolchain image. Verified: 4/4 fresh-instance runs green (38/38), lock section deterministic. Still deferred to #363 follow-ups: the playout-build lock 409 + isLocked projection (#215) and the UI-interactive Playwright flows. Docs updated same PR: docs/e2e-local.md, docs/ci-cd.md, the functional-e2e job comment in .gitea/workflows/docker-build.yml. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitea/workflows/docker-build.yml | 20 +++-- docs/ci-cd.md | 35 +++++--- docs/e2e-local.md | 38 +++++--- scripts/e2e-functional.sh | 141 +++++++++++++++++++++++++++--- 4 files changed, 194 insertions(+), 40 deletions(-) diff --git a/.gitea/workflows/docker-build.yml b/.gitea/workflows/docker-build.yml index d6fb06356..45278c9dd 100644 --- a/.gitea/workflows/docker-build.yml +++ b/.gitea/workflows/docker-build.yml @@ -423,12 +423,13 @@ jobs: name: Functional E2E (curl contracts) runs-on: ubuntu-latest # Advisory gate (ersatztv#299): boots the app from source and drives the manual live-E2E - # curl flows (legacy->SPA redirects, auth/CSRF/security-stamp, library-scan status contract, - # If-Match/412) that sessions have been re-running by hand. Deliberately NOT a `needs:` of - # `build` and not (yet) a required check, so a functional-E2E flake can't block image builds or - # the unit-test gate — promote it to a required check / build dependency once it's proven - # reliable (same rollout the `migrations` job used). SQLite default provider -> no DB service. - # Runs on PRs and on main (regression net); skipped for v* tag builds. + # flows (legacy->SPA redirects, auth/CSRF/security-stamp, library-scan status contract, + # If-Match/412, and since ersatztv#363 two lock-contention 409s) that sessions have been + # re-running by hand. Deliberately NOT a `needs:` of `build` and not (yet) a required check, so a + # functional-E2E flake can't block image builds or the unit-test gate — promote it to a required + # check / build dependency once it's proven reliable (same rollout the `migrations` job used). + # SQLite default provider -> no DB service. Runs on PRs and on main (regression net); skipped for + # v* tag builds. if: github.event_name == 'pull_request' || github.ref == 'refs/heads/main' container: image: 192.168.1.95:3000/timothy/ersatztv-ci:07048b8 @@ -482,9 +483,10 @@ jobs: # The old `command -v ffmpeg || sudo apt-get install ffmpeg` step is gone (ersatztv#390): # the toolchain image ships the same ffmpeg build prod runs, so the binary is already here. - # That step also cost 110s of every run to install a dependency this harness never uses — - # it boots the app (whose only ffmpeg touch at startup is a LogWarning from - # FFmpegLocatorService) and drives curl-only contracts that never transcode. + # That step also cost 110s of every run. The harness never *transcodes*, but since ersatztv#363 + # it does use ffmpeg to synthesize ~60 tiny testsrc clips to seed the scan-lock 409 flow (and + # python3's stdlib sqlite3 to seed the DB rows the API can't create) — both already present in + # the image, so still no per-run install. The scan flow self-skips if ffmpeg is ever absent. - name: Boot instance and run functional-E2E harness if: steps.detect.outputs.docs_only != 'true' diff --git a/docs/ci-cd.md b/docs/ci-cd.md index c62b3003c..0918f3eca 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -286,22 +286,37 @@ re-run by hand, turning them into a CI regression net (ersatztv#299). It is the 2. `ETV_BUILD_CONFIG=Release scripts/e2e-local.sh ` — copies `wwwroot`, launches `dotnet ErsatzTV.dll` in the background (logging to a file so the launch step returns once the app is ready), prints `PID`/`CONFIG_DIR`. -3. `scripts/e2e-functional.sh http://localhost:8409 ` — asserts, all curl-only and - deterministic (no seeded media, no browser): the **legacy→SPA redirect sweep** (+ the `/api`, - `/artwork` never-redirect exemption), the **auth/CSRF/security-stamp** flow (setup-claim → - read-gate 401/200 → re-claim 409 → CSRF 403 → login 401/200 → logout 403/204 → post-logout - stamp-revocation 401), the **library-scan status contract** (404 unknown / 202 queued / - `scan-status` 200), and the **If-Match/412** round-trip on `rerun-collections`. A `trap` kills the - instance on step exit. +3. `scripts/e2e-functional.sh http://localhost:8409 ` — asserts. Mostly curl-only: the + **legacy→SPA redirect sweep** (+ the `/api`, `/artwork` never-redirect exemption), the + **auth/CSRF/security-stamp** flow (setup-claim → read-gate 401/200 → re-claim 409 → CSRF 403 → + login 401/200 → logout 403/204 → post-logout stamp-revocation 401), the **library-scan status + contract** (404 unknown / 202 queued / `scan-status` 200), and the **If-Match/412** round-trip on + `rerun-collections`. Since ersatztv#363 it also asserts two **lock-contention 409s** that aren't + curl-only — see below. A `trap` kills the instance on step exit. + +**Lock-contention 409s (ersatztv#363).** The harness now seeds rows the API can't create — a +`LibraryPath` and a Jellyfin media-source — directly into the running instance's SQLite DB (via +python3's stdlib `sqlite3`; WAL mode tolerates the second writer), and synthesizes media with the +image's ffmpeg, to exercise two `IEntityLocker` contracts **deterministically** (it only fires the +racing request once the lock is *provably* held, never a sleep-and-hope): (a) the **library-scan +"already scanning" 409** — seed ~60 tiny clips into the built-in Shows library so the scanner +subprocess runs a few seconds, poll `GET /libraries/scan-status` until the library shows active (that +window is a strict subset of the scan lock's held window), then a second `POST .../scan` is a +guaranteed 409; (b) the **external-collections "already scanning" 409** — the per-family lock is taken +synchronously *before* the 202, so the 202 proves it held, and pointing the seeded source at a +non-routable address keeps the background sync hung so the window stays open. No new CI step or +dependency: ffmpeg ships in the toolchain image, and python3 was already a harness dependency +(`json_field`). The scan flow self-skips (advisory) if ffmpeg is ever absent. **Advisory, by design** (the issue's "keep it a separate job so a functional-E2E flake can't block the unit-test gate"): it is **not** a `needs:` of `build` and **not (yet) a required check**, so a flake blocks nothing. Promote it to a required check / `build` dependency once it's proven reliable — the same staged rollout the `migrations` job used. SQLite is the default provider, so unlike `migrations` it needs **no** DB service container. Runs on PRs and on `main` (regression net); skipped for `v*` tag -builds. Out of scope for this first cut (need the scanner subprocess + seeded media, or a browser, to -be deterministic — tracked as ersatztv#299 follow-ups): the racy 409 "already-scanning" re-trigger, -the playout-build lock 409, and the genuinely UI-interactive Playwright flows. +builds. Still out of scope (tracked as ersatztv#363 follow-ups): the **playout-build lock 409 + +`isLocked` projection** (#215) — needs a seeded schedule/playout and a build window wide enough to +observe without racing — and the genuinely **UI-interactive Playwright** flows (a separate CI +browser-tooling lift). ### Docs-only skip (ersatztv#416) diff --git a/docs/e2e-local.md b/docs/e2e-local.md index 923c394ba..4d4d4fa8a 100644 --- a/docs/e2e-local.md +++ b/docs/e2e-local.md @@ -147,24 +147,42 @@ kill "$PID" `$CONFIG_DIR/api.key` and sends `X-Api-Key` on every `/api` call (the surface is fail-closed). It runs each assertion even after a failure and prints a pass/fail summary, exiting non-zero if any failed. -What it covers (all curl-only, deterministic, no seeded media / ffmpeg-transcode / browser needed): +What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browser); the +**lock-contention** section (added by ersatztv#363) is the exception — it seeds DB rows + media. - **Legacy→SPA redirects**: a sweep of representative `LegacyUiRedirects` routes 302→`/app/*`, plus the `/api` + `/artwork` never-redirect exemption (asserted as "did not 302 to `/app`", since those 4xx from their own handlers — `/api/*` 404s, `/artwork/*` 400s). -- **Auth/CSRF/security-stamp** (mirrors the #295 manual set): fresh `auth/config` `setupRequired:true` - → read-gate 401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation - 403-without-`X-CSRF` → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the - same cookie is 401 (rotated security stamp). - **Library-scan status contract**: create an empty local library → scan `202`, unknown-library scan `404`, `scan-status` `200`. - **Optimistic concurrency**: create a collection + a `rerun-collections` targeting it → `GET` emits an `ETag` → `PUT` with a stale `If-Match` `412`, current `200`, malformed `400`. +- **Lock contention (409)** — the two `IEntityLocker` 409s, made **deterministic** by firing the + racing request only once the lock is *provably* held (never a sleep-and-hope). Both seed rows the + API can't create straight into the running instance's DB via python3's stdlib `sqlite3` (WAL mode + tolerates the second writer): + - **library-scan "already scanning"**: seed ~60 tiny ffmpeg clips into the built-in Shows library + (Id=2) so the scanner subprocess runs a few seconds → `POST .../scan?deep=true` `202` → poll + `GET /libraries/scan-status` until library 2 shows active (that window is a strict subset of the + scan lock's held window — `ScannerProxyService.StartScan` fires *after* `LockLibrary`, `EndScan` + *before* `UnlockLibrary`) → a second `POST .../scan` is a guaranteed `409`. Self-skips (advisory) + if ffmpeg is absent. + - **external-collections "already scanning"**: seed a Jellyfin media-source row pointing at a + non-routable address (so the background sync hangs and the per-family lock stays held). The lock + is taken synchronously *before* the `202`, so the `202` proves it held → `collections-scan-status` + shows the `jellyfin` family → a second `POST .../scan-collections` is `409`; unknown source `404`. + Needs neither ffmpeg nor the scanner. +- **Auth/CSRF/security-stamp** (mirrors the #295 manual set; runs **last** — setup-claim is one-shot + and logout revokes the session): fresh `auth/config` `setupRequired:true` → read-gate + 401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation + 403-without-`X-CSRF` → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the + same cookie is 401 (rotated security stamp). -**Deliberately deferred** (need the scanner subprocess + seeded media, or a browser, to be -deterministic — ersatztv#299 follow-ups): the 409 "already-scanning" re-trigger, the playout-build -lock 409, and the genuinely UI-interactive Playwright flows. When you extend the harness, add the -observed contract here and keep the assertions deterministic (probe the real instance first — the -initial cut caught that `/artwork/*` 400s where a guess said 404). +**Still deferred** (ersatztv#363 follow-ups): the **playout-build lock 409 + `isLocked` projection** +(#215) — needs a seeded schedule/playout and a build window wide enough to observe without racing — +and the genuinely **UI-interactive Playwright** flows (a separate CI browser-tooling lift). When you +extend the harness, add the observed contract here and keep the assertions deterministic (probe the +real instance first — the initial cut caught that `/artwork/*` 400s where a guess said 404; the #363 +cut measured the real scan window before sizing the seed). ## Seeding a local TV library for E2E diff --git a/scripts/e2e-functional.sh b/scripts/e2e-functional.sh index 515c80434..ac3c847bb 100755 --- a/scripts/e2e-functional.sh +++ b/scripts/e2e-functional.sh @@ -14,17 +14,24 @@ # ($CONFIG_DIR/api.key). The /api surface is fail-closed (Api:RequireKeyForReads # defaults true), so every /api call sends X-Api-Key. # -# Scope (all curl-only, deterministic, no seeded media / ffmpeg / browser needed): -# 1. Legacy -> SPA redirect sweep (+ the /api,/artwork never-redirect exemption). -# 2. Auth / CSRF / security-stamp flow (setup-claim, read-gate, CSRF gate, login, logout+revoke). -# 3. Library scan lifecycle status-code contract (404 / 202 / scan-status). -# 4. Optimistic-concurrency If-Match / 412 round-trip. +# Scope: +# 1. Legacy -> SPA redirect sweep (+ the /api,/artwork never-redirect exemption). [curl-only] +# 2. Machine-key data-plane: read-gate, library-scan lifecycle (404/202/scan-status), +# optimistic-concurrency If-Match/412. [curl-only] +# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress [seeds DB; +# (ersatztv#363) — see that section's header for how each is made deterministic. scan=ffmpeg] +# 4. Auth / CSRF / security-stamp (setup-claim, read-gate, CSRF gate, login, logout+revoke). # -# Deliberately OUT of scope for this first cut (need seeded media + the scanner subprocess, or a -# browser, to be deterministic — tracked as ersatztv#299 follow-ups): -# - the 409 "already scanning" re-trigger (racy without a long-running scan), -# - the playout-build lock 409, -# - the genuinely UI-interactive Playwright flows. +# Section 3 is the first that isn't curl-only: it seeds rows the API can't create (a LibraryPath and +# a media-source) straight into the running instance's SQLite DB (via python3's stdlib sqlite3 — WAL +# mode tolerates a second writer) and, for the scan flow, synthesizes media with ffmpeg. Both 409s +# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan flow is +# skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner. +# +# Deliberately still OUT of scope (tracked as ersatztv#363 follow-ups): +# - the playout-build lock 409 + isLocked projection (#215) — needs a seeded schedule/playout and a +# build window wide enough to observe without racing, +# - the genuinely UI-interactive Playwright (headless) flows — a separate CI browser-tooling lift. # # Exit status: 0 if every assertion passed, 1 if any failed. Assertions keep running after a # failure so one run reports the full picture. @@ -171,7 +178,119 @@ expect_status "PUT with the current If-Match -> 200" 200 PUT "$BASE_URL/api/v1/r expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \ -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H 'If-Match: not-a-version' -d "$RC_BODY" -# ---- 3. Auth / CSRF / security-stamp flow ------------------------------------------------------- +# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress ----------------------- +# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363). Both are made +# DETERMINISTIC (no sleep-and-hope) by only firing the racing request once the lock is provably held: +# +# A. Library-scan "already scanning" 409. A real scan must be in flight, so we seed a Shows library +# with enough media that the scanner subprocess (out-of-process, one ffprobe per file) runs for a +# few seconds. The scan-status "active" window is a strict SUBSET of the scan lock's held window +# (ScannerProxyService.StartScan happens after locker.LockLibrary, EndScan before UnlockLibrary -- +# ErsatzTV/Services/ScannerService.cs), so once GET /libraries/scan-status shows the library +# active, the lock is provably held and a second POST .../scan is a guaranteed 409. +# +# B. External-collections "already scanning" 409 (per-family lock). The controller takes the lock +# SYNCHRONOUSLY before returning 202 (JellyfinMediaSourcesController), so the 202 itself proves +# the lock is held; pointing the seeded source at a non-routable address keeps the background +# sync hung so the window stays wide open. GET /media-sources/collections-scan-status corroborates. +# +# Seeding writes rows the API can't create (LibraryPath, media-source) directly into the instance's +# SQLite DB via python3's stdlib sqlite3 (already a hard dep of this harness); WAL mode tolerates the +# second writer. Runs on the machine key, before the auth section (which claims the admin) -- these +# never claim it. Flow A is skipped (advisory) without ffmpeg; Flow B needs neither ffmpeg nor a scanner. +section "Lock contention (409): scan-in-progress + collections-in-progress" + +DB="$CONFIG_DIR/ersatztv.sqlite3" + +# seed_library_path PATH LIBRARY_ID -- attach a folder to a built-in local library (parameterized). +seed_library_path() { + python3 - "$DB" "$1" "$2" <<'PY' +import sqlite3, sys +con = sqlite3.connect(sys.argv[1], timeout=10) +con.execute("INSERT INTO LibraryPath (Path, LibraryId) VALUES (?, ?)", (sys.argv[2], int(sys.argv[3]))) +con.commit() +PY +} + +# seed_jellyfin_source ADDRESS -- create MediaSource + JellyfinMediaSource + JellyfinConnection rows, +# print the new media-source id. The row only needs to EXIST for the {id} to resolve (404 otherwise); +# a non-routable ADDRESS makes the background collections sync hang so the family lock stays held. +seed_jellyfin_source() { + python3 - "$DB" "$1" <<'PY' +import sqlite3, sys +con = sqlite3.connect(sys.argv[1], timeout=10) +cur = con.cursor() +cur.execute("INSERT INTO MediaSource DEFAULT VALUES") +msid = cur.lastrowid +cur.execute("INSERT INTO JellyfinMediaSource (Id, ServerName, OperatingSystem) VALUES (?, 'E2E-Lock-Jellyfin', 'linux')", (msid,)) +cur.execute("INSERT INTO JellyfinConnection (Address, JellyfinMediaSourceId) VALUES (?, ?)", (sys.argv[2], msid)) +con.commit() +print(msid) +PY +} + +skip() { printf ' \033[33mSKIP\033[0m %s\n' "$1"; } + +if [ ! -f "$DB" ]; then + bad "expected the instance SQLite DB at $DB (needed to seed the lock-flow rows)" +else + # --- Flow A: library-scan already-scanning 409 (needs seeded media + the scanner subprocess) --- + if command -v ffmpeg >/dev/null 2>&1; then + SEED_ROOT="$(mktemp -d)/tv" + SEED_SEASON="$SEED_ROOT/Show Lock/Season 01" + mkdir -p "$SEED_SEASON" + SEED_SRC="$(mktemp -d)/seed.mkv" + if ffmpeg -y -loglevel error -f lavfi -i testsrc=duration=1:size=160x120:rate=5 \ + -c:v libx264 -pix_fmt yuv420p "$SEED_SRC" >/dev/null 2>&1; then + # 60 distinct episode files (the scanner ffprobes each) -> a multi-second scan even on fast CI + # I/O, so the poll->fire gap (one HTTP round-trip) is comfortably inside the lock window. + for n in $(seq -w 1 60); do cp "$SEED_SRC" "$SEED_SEASON/Show Lock - s01e$n.mkv"; done + seed_library_path "$SEED_ROOT" 2 # Shows is the built-in library Id=2 (docs/e2e-local.md recipe) + expect_status "POST /libraries/2/scan?deep=true -> 202 (queued)" 202 POST "$BASE_URL/api/v1/libraries/2/scan?deep=true" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' + # Poll scan-status until the scan is provably in flight, then fire the racing scan. + scan_active="" + for _ in $(seq 1 100); do + if api "$BASE_URL/api/v1/libraries/scan-status" | grep -q '"libraryId":2'; then scan_active=1; break; fi + sleep 0.1 + done + if [ -n "$scan_active" ]; then + ok "scan-status shows library 2 active (scan lock provably held)" + expect_status "second POST /libraries/2/scan while scanning -> 409" 409 POST "$BASE_URL/api/v1/libraries/2/scan" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' + else + # Never observed active within 10s: don't assert a race we can't prove we won (advisory skip). + skip "library-2 scan never observed active within 10s -> not asserting the 409 (seed=60 files)" + fi + else + bad "ffmpeg present but failed to synthesize the seed clip for the scan-lock flow" + fi + else + skip "ffmpeg not found -> skipping the library-scan 409 flow (advisory; needs seeded media + scanner)" + fi + + # --- Flow B: external-collections already-scanning 409 (per-family lock; no ffmpeg/scanner) --- + MSID="$(seed_jellyfin_source 'http://10.255.255.1:8096')" + if [ -n "$MSID" ]; then + ok "seeded Jellyfin media source id=$MSID (non-routable -> sync hangs, lock stays held)" + expect_status "POST /media-sources/jellyfin/$MSID/scan-collections -> 202" 202 POST "$BASE_URL/api/v1/media-sources/jellyfin/$MSID/scan-collections" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' + # The lock is taken synchronously before the 202, so it is provably held now; corroborate via status. + if api "$BASE_URL/api/v1/media-sources/collections-scan-status" | grep -q '"family":"jellyfin"'; then + ok "collections-scan-status shows the jellyfin family locked" + else + bad "collections-scan-status did not show the jellyfin family locked after a 202" + fi + expect_status "second POST .../jellyfin/$MSID/scan-collections while scanning -> 409" 409 POST "$BASE_URL/api/v1/media-sources/jellyfin/$MSID/scan-collections" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' + expect_status "POST .../jellyfin/99999/scan-collections (unknown source) -> 404" 404 POST "$BASE_URL/api/v1/media-sources/jellyfin/99999/scan-collections" \ + -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' + else + bad "could not seed a Jellyfin media source row for the collections-lock flow" + fi +fi + +# ---- 4. Auth / CSRF / security-stamp flow ------------------------------------------------------- # Runs last: setup-claim is a one-shot on a fresh config and logout revokes the session. All prior # sections used the machine key, which does not claim the admin, so setupRequired is still true here. section "Auth: setup-claim, CSRF gate, login, logout + stamp revocation"