timothyandClaude Opus 4.8
038703fe67
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 10s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 14s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 9s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 33s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m24s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 15m4s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 18m25s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 18m37s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Adds "Flow C" to scripts/e2e-functional.sh, the last deferred lock-contention flow from #363. A playout build is enqueued onto the single-consumer WorkerService channel and the trigger returns before BuildPlayoutHandler acquires the lock, so an accepted trigger does not prove the lock is held. Flow C makes it deterministic: seed a Classic Flood schedule over a few short ffmpeg episodes, crank PlayoutDaysToBuild=5 (~43k items ~= ~1s build), then POLL GET /playouts/{id} until isLocked:true before firing. Asserts PUT /playouts/{id} -> 409, reset -> 409, and the list-projection isLocked:true while locked; then isLocked:false + PUT -> 200 after the build (proving the 409 is lock-specific). Each racing assertion is guarded so a build that finishes mid-flight degrades to an advisory skip, never a false red; the whole flow self-skips without ffmpeg or if the build is never observed locked. Sized by measurement on a fresh instance -- going wider is counter-productive (a 777k-item build saturates the single worker with post-build gap/overlap jobs). Verified green across 6 fresh-instance runs; cold adversarial review MERGEABLE. Docs: docs/e2e-local.md + docs/ci-cd.md updated to describe Flow C and drop it from the "deferred" lists. fixes #444 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
511 lines
32 KiB
Bash
Executable File
511 lines
32 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# scripts/e2e-functional.sh — drive the manual live-E2E functional flows against a RUNNING
|
|
# ErsatzTV instance and assert their HTTP contracts. This is the codified, automatable half of the
|
|
# ad-hoc curl scenarios sessions have been running by hand (ersatztv#299); it runs both locally and
|
|
# in CI (the `functional-e2e` job in .gitea/workflows/docker-build.yml).
|
|
#
|
|
# It does NOT boot the app — pair it with scripts/e2e-local.sh (which builds/launches and prints the
|
|
# CONFIG_DIR), or point it at any already-running instance:
|
|
#
|
|
# scripts/e2e-functional.sh [BASE_URL] [CONFIG_DIR]
|
|
#
|
|
# BASE_URL default http://localhost:8409 (the app's UI+API port; ETV_UI_PORT)
|
|
# CONFIG_DIR the instance's config folder — REQUIRED, so we can read its machine api key
|
|
# ($CONFIG_DIR/api.key). The /api surface is fail-closed (Api:RequireKeyForReads
|
|
# defaults true), so every /api call sends X-Api-Key.
|
|
#
|
|
# Scope:
|
|
# 1. Legacy -> SPA redirect sweep (+ the /api,/artwork never-redirect exemption). [curl-only]
|
|
# 2. Machine-key data-plane: read-gate, library-scan lifecycle (404/202/scan-status),
|
|
# optimistic-concurrency If-Match/412. [curl-only]
|
|
# 3. Lock-contention 409s: scan-in-progress + external-collections-in-progress + playout-build [seeds DB;
|
|
# (ersatztv#363, #444) — see that section's header for how each is made deterministic. scan=ffmpeg]
|
|
# 4. Auth / CSRF / security-stamp (setup-claim, read-gate, CSRF gate, login, logout+revoke).
|
|
#
|
|
# Section 3 is the first that isn't curl-only: it seeds rows the API can't create (a LibraryPath and
|
|
# a media-source) straight into the running instance's SQLite DB (via python3's stdlib sqlite3, whose
|
|
# busy-timeout retry serializes behind the app's writer) and, for the scan/build flows, synthesizes
|
|
# media with ffmpeg. All three 409s
|
|
# are made DETERMINISTIC (no racy sleep-and-hope) — see the section header for how. The scan + build
|
|
# flows are skipped (advisory) when ffmpeg is absent; the collections flow needs neither ffmpeg nor a scanner.
|
|
#
|
|
# Deliberately still OUT of scope (tracked as ersatztv#363 follow-ups):
|
|
# - the genuinely UI-interactive Playwright (headless) flows — a separate CI browser-tooling lift.
|
|
#
|
|
# Exit status: 0 if every assertion passed, 1 if any failed. Assertions keep running after a
|
|
# failure so one run reports the full picture.
|
|
|
|
set -uo pipefail
|
|
|
|
BASE_URL="${1:-http://localhost:8409}"
|
|
CONFIG_DIR="${2:-}"
|
|
|
|
if [ -z "$CONFIG_DIR" ]; then
|
|
echo "error: CONFIG_DIR (arg 2) is required — needed to read the machine api key ($CONFIG_DIR/api.key)." >&2
|
|
echo "usage: scripts/e2e-functional.sh [BASE_URL] CONFIG_DIR" >&2
|
|
exit 2
|
|
fi
|
|
|
|
KEY_FILE="$CONFIG_DIR/api.key"
|
|
if [ ! -f "$KEY_FILE" ]; then
|
|
echo "error: $KEY_FILE not found. Is the instance running with ETV_CONFIG_FOLDER=$CONFIG_DIR?" >&2
|
|
exit 2
|
|
fi
|
|
API_KEY="$(cat "$KEY_FILE")"
|
|
|
|
PASS=0
|
|
FAIL=0
|
|
FAILURES=()
|
|
|
|
# ---- assertion helpers -------------------------------------------------------------------------
|
|
|
|
# ok/bad: record a single assertion result.
|
|
ok() { PASS=$((PASS + 1)); printf ' \033[32mPASS\033[0m %s\n' "$1"; }
|
|
bad() { FAIL=$((FAIL + 1)); FAILURES+=("$1"); printf ' \033[31mFAIL\033[0m %s\n' "$1"; }
|
|
|
|
# status_of METHOD URL [extra curl args...] -> prints the HTTP status code.
|
|
status_of() {
|
|
local method="$1" url="$2"; shift 2
|
|
curl -s -o /dev/null -w '%{http_code}' -X "$method" "$@" "$url"
|
|
}
|
|
|
|
# expect_status "desc" EXPECTED METHOD URL [extra curl args...]
|
|
expect_status() {
|
|
local desc="$1" expected="$2" method="$3" url="$4"; shift 4
|
|
local got; got="$(status_of "$method" "$url" "$@")"
|
|
if [ "$got" = "$expected" ]; then ok "$desc (=$expected)"; else bad "$desc — expected $expected, got $got [$method $url]"; fi
|
|
}
|
|
|
|
# api(): curl against the /api surface with the machine key + JSON content type.
|
|
api() { curl -s -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' "$@"; }
|
|
|
|
# json_field FILE FIELD -> value of a top-level JSON field (via python3, always present here).
|
|
json_field() { python3 -c "import json,sys; print(json.load(open('$1')).get('$2',''))"; }
|
|
|
|
section() { printf '\n\033[1m== %s ==\033[0m\n' "$1"; }
|
|
|
|
echo "Functional E2E against $BASE_URL (config: $CONFIG_DIR)"
|
|
|
|
# ---- 1. Legacy -> SPA redirect sweep ------------------------------------------------------------
|
|
# The legacy branch 302s GET requests to their /app equivalent; /api, /artwork, /docs, /openapi are
|
|
# NEVER redirected (they 4xx from their own handlers/fallback). See ErsatzTV/LegacyUiRedirects.cs +
|
|
# Startup.cs MapFallback, and docs/blazor-route-parity.md.
|
|
section "Legacy -> SPA redirects"
|
|
|
|
# redirects_to "src" "expected /app location suffix"
|
|
redirects_to() {
|
|
local src="$1" want="$2"
|
|
local code loc
|
|
code="$(curl -s -o /dev/null -w '%{http_code}' "$BASE_URL$src")"
|
|
loc="$(curl -s -o /dev/null -w '%{redirect_url}' "$BASE_URL$src")"
|
|
if [ "$code" = "302" ] && [ "$loc" = "$BASE_URL$want" ]; then
|
|
ok "$src -> 302 $want"
|
|
else
|
|
bad "$src — expected 302 -> $BASE_URL$want, got $code -> ${loc:-<none>}"
|
|
fi
|
|
}
|
|
redirects_to "/" "/app"
|
|
redirects_to "/channels" "/app/channels"
|
|
redirects_to "/channels/5" "/app/edit-channel/5"
|
|
redirects_to "/settings/ffmpeg" "/app/settings/streaming"
|
|
redirects_to "/media/movies" "/app/media?kind=movies"
|
|
redirects_to "/media/sources/plex/3/libraries" "/app/libraries/plex/3/sync"
|
|
|
|
# The exempt prefixes must NOT redirect to /app — they 4xx from their own handlers (an unknown
|
|
# /api path 404s; /artwork rejects a non-artwork path 400). Assert only "did not 302 to /app".
|
|
never_redirects() {
|
|
local src="$1"
|
|
local code loc
|
|
code="$(curl -s -o /dev/null -w '%{http_code}' "$BASE_URL$src")"
|
|
loc="$(curl -s -o /dev/null -w '%{redirect_url}' "$BASE_URL$src")"
|
|
if [ "$code" != "302" ] && [ -z "$loc" ] && [ "${code:0:1}" = "4" ]; then
|
|
ok "$src -> not redirected (${code})"
|
|
else
|
|
bad "$src — expected a non-redirect 4xx, got $code -> ${loc:-<none>}"
|
|
fi
|
|
}
|
|
never_redirects "/api/does-not-exist"
|
|
never_redirects "/artwork/does-not-exist"
|
|
|
|
# ---- 2. Machine-key data-plane contracts (scan lifecycle + If-Match/412) ------------------------
|
|
# These use the machine key (X-Api-Key), which is independent of the session and does NOT claim the
|
|
# local admin — so the fresh-config setup-claim flow in section 3 still sees setupRequired:true.
|
|
section "Read gate (Api:RequireKeyForReads = true)"
|
|
expect_status "GET /api/v1/channels without a key -> 401" 401 GET "$BASE_URL/api/v1/channels"
|
|
expect_status "GET /api/v1/channels with the machine key -> 200" 200 GET "$BASE_URL/api/v1/channels" \
|
|
-H "X-Api-Key: $API_KEY"
|
|
|
|
section "Library scan lifecycle contract"
|
|
# Create an empty local library (no LibraryPath) — enough to exercise the enqueue/status contract
|
|
# without any media on disk (QueueLibraryScanByLibraryIdHandler never inspects paths).
|
|
LIB_JSON="$(mktemp)"
|
|
api -X POST "$BASE_URL/api/v1/libraries/local" -d '{"name":"E2E Functional Library"}' -o "$LIB_JSON" -w ''
|
|
LIB_ID="$(json_field "$LIB_JSON" id)"
|
|
if [ -n "$LIB_ID" ]; then ok "created local library id=$LIB_ID"; else bad "could not create local library (body: $(cat "$LIB_JSON"))"; fi
|
|
expect_status "POST /api/v1/libraries/$LIB_ID/scan -> 202 (queued)" 202 POST "$BASE_URL/api/v1/libraries/$LIB_ID/scan" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
expect_status "POST /api/v1/libraries/99999/scan -> 404 (unknown library)" 404 POST "$BASE_URL/api/v1/libraries/99999/scan" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
expect_status "GET /api/v1/libraries/scan-status -> 200" 200 GET "$BASE_URL/api/v1/libraries/scan-status" \
|
|
-H "X-Api-Key: $API_KEY"
|
|
|
|
section "Optimistic concurrency (If-Match / 412)"
|
|
# rerun-collections is the reference concurrency-aware editor: GET emits an ETag, PUT enforces
|
|
# If-Match. It needs an existing collection to target, so create one first.
|
|
COLL_JSON="$(mktemp)"
|
|
api -X POST "$BASE_URL/api/v1/collections" -d '{"name":"E2E Functional Collection"}' -o "$COLL_JSON" -w ''
|
|
COLL_ID="$(json_field "$COLL_JSON" id)"
|
|
RC_JSON="$(mktemp)"
|
|
api -X POST "$BASE_URL/api/v1/rerun-collections" -o "$RC_JSON" -w '' -d "{
|
|
\"name\":\"E2E Functional Rerun\",\"collectionType\":\"Collection\",\"selectedId\":$COLL_ID,
|
|
\"firstRunPlaybackOrder\":\"Chronological\",\"rerunPlaybackOrder\":\"Chronological\"}"
|
|
RC_ID="$(json_field "$RC_JSON" id)"
|
|
if [ -n "$RC_ID" ]; then ok "created rerun-collection id=$RC_ID (targets collection $COLL_ID)"; else bad "could not create rerun-collection (body: $(cat "$RC_JSON"))"; fi
|
|
|
|
# GET must surface the current version as a quoted-integer ETag.
|
|
ETAG="$(api -D - -o /dev/null "$BASE_URL/api/v1/rerun-collections/$RC_ID" | awk 'tolower($1)=="etag:"{print $2}' | tr -d '\r')"
|
|
if [ -n "$ETAG" ]; then ok "GET rerun-collection $RC_ID surfaces ETag $ETAG"; else bad "GET rerun-collection $RC_ID has no ETag header"; fi
|
|
|
|
RC_BODY="{\"name\":\"E2E Functional Rerun v2\",\"selectedId\":$COLL_ID,\"firstRunPlaybackOrder\":\"Chronological\",\"rerunPlaybackOrder\":\"Chronological\"}"
|
|
# A well-formed but non-current version must be rejected 412 (999999 can never be the live version here).
|
|
expect_status "PUT with a stale If-Match -> 412" 412 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H 'If-Match: "999999"' -d "$RC_BODY"
|
|
# The current ETag must be accepted.
|
|
expect_status "PUT with the current If-Match -> 200" 200 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H "If-Match: $ETAG" -d "$RC_BODY"
|
|
# A malformed (non-integer) If-Match is a client error.
|
|
expect_status "PUT with a malformed If-Match -> 400" 400 PUT "$BASE_URL/api/v1/rerun-collections/$RC_ID" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -H 'If-Match: not-a-version' -d "$RC_BODY"
|
|
|
|
# ---- 3. Lock contention (409): scan-in-progress + collections-in-progress + playout-build --------
|
|
# The IEntityLocker 409 contracts the first cut deferred (ersatztv#299 -> #363, #444). All three are made
|
|
# DETERMINISTIC (no sleep-and-hope) by only firing the racing request once the lock is provably held:
|
|
#
|
|
# A. Library-scan "already scanning" 409. A real scan must be in flight, so we seed a Shows library
|
|
# with enough media that the scanner subprocess (out-of-process, one ffprobe per file) runs for a
|
|
# few seconds. The scan-status "active" window is a strict SUBSET of the scan lock's held window
|
|
# (ScannerProxyService.StartScan happens after locker.LockLibrary, EndScan before UnlockLibrary --
|
|
# ErsatzTV/Services/ScannerService.cs), so once GET /libraries/scan-status shows the library
|
|
# active the lock is provably held and a second POST .../scan is a 409 -- deterministic bar a
|
|
# tiny residual TOCTOU gap (observe-active -> 2nd POST lands) that the multi-second scan covers.
|
|
#
|
|
# B. External-collections "already scanning" 409 (per-family lock). The controller takes the lock
|
|
# SYNCHRONOUSLY before returning 202 (JellyfinMediaSourcesController), so the 202 itself proves
|
|
# the lock is held (race-free by construction); pointing the seeded source at a non-routable
|
|
# address keeps the background sync hung so the window stays wide open. GET
|
|
# /media-sources/collections-scan-status corroborates.
|
|
#
|
|
# C. Playout-build "build in progress" 409 + isLocked projection (#215). Hardest of the three: the
|
|
# build is enqueued onto the single-consumer WorkerService channel and the trigger returns BEFORE
|
|
# BuildPlayoutHandler dequeues + LockPlayout (released in its finally -- BuildPlayoutHandler.cs), so
|
|
# an accepted trigger does NOT prove the lock is held. We POLL GET /playouts/{id} until isLocked:true,
|
|
# then fire the racing mutations while it is provably held. The build is made observable by seeding a
|
|
# Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild (config
|
|
# 'playout.days_to_build'): a single 5-day build is ~43k playout items ~= ~1s here, and CI runners are
|
|
# slower so the window only widens (#444 measurement). Restores the config default afterwards.
|
|
#
|
|
# Seeding writes rows the API can't create (LibraryPath, media-source) directly into the instance's
|
|
# SQLite DB via python3's stdlib sqlite3 (already a hard dep of this harness); the connection's
|
|
# busy-timeout retry serializes this second writer behind the app. Runs on the machine key, before the
|
|
# auth section (which claims the admin) -- these never claim it. Flows A + C are skipped (advisory)
|
|
# without ffmpeg; Flow B needs neither ffmpeg nor a scanner.
|
|
section "Lock contention (409): scan-in-progress + collections-in-progress + playout-build"
|
|
|
|
DB="$CONFIG_DIR/ersatztv.sqlite3"
|
|
|
|
# seed_library_path PATH LIBRARY_ID -- attach a folder to a built-in local library (parameterized);
|
|
# prints the new LibraryPath row id so the caller can detect a silent seeding failure.
|
|
seed_library_path() {
|
|
python3 - "$DB" "$1" "$2" <<'PY'
|
|
import sqlite3, sys
|
|
con = sqlite3.connect(sys.argv[1], timeout=10) # busy-timeout retry serializes this second writer
|
|
cur = con.cursor()
|
|
cur.execute("INSERT INTO LibraryPath (Path, LibraryId) VALUES (?, ?)", (sys.argv[2], int(sys.argv[3])))
|
|
con.commit()
|
|
print(cur.lastrowid)
|
|
PY
|
|
}
|
|
|
|
# seed_jellyfin_source ADDRESS -- create MediaSource + JellyfinMediaSource + JellyfinConnection rows,
|
|
# print the new media-source id. The row only needs to EXIST for the {id} to resolve (404 otherwise);
|
|
# a non-routable ADDRESS makes the background collections sync hang so the family lock stays held.
|
|
seed_jellyfin_source() {
|
|
python3 - "$DB" "$1" <<'PY'
|
|
import sqlite3, sys
|
|
con = sqlite3.connect(sys.argv[1], timeout=10) # busy-timeout retry serializes this second writer
|
|
cur = con.cursor()
|
|
cur.execute("INSERT INTO MediaSource DEFAULT VALUES")
|
|
msid = cur.lastrowid
|
|
cur.execute("INSERT INTO JellyfinMediaSource (Id, ServerName, OperatingSystem) VALUES (?, 'E2E-Lock-Jellyfin', 'linux')", (msid,))
|
|
cur.execute("INSERT INTO JellyfinConnection (Address, JellyfinMediaSourceId) VALUES (?, ?)", (sys.argv[2], msid))
|
|
con.commit()
|
|
print(msid)
|
|
PY
|
|
}
|
|
|
|
skip() { printf ' \033[33mSKIP\033[0m %s\n' "$1"; }
|
|
|
|
if [ ! -f "$DB" ]; then
|
|
bad "expected the instance SQLite DB at $DB (needed to seed the lock-flow rows)"
|
|
else
|
|
# --- Flow A: library-scan already-scanning 409 (needs seeded media + the scanner subprocess) ---
|
|
if command -v ffmpeg >/dev/null 2>&1; then
|
|
SEED_ROOT="$(mktemp -d)/tv"
|
|
SEED_SEASON="$SEED_ROOT/Show Lock/Season 01"
|
|
mkdir -p "$SEED_SEASON"
|
|
SEED_SRC="$(mktemp -d)/seed.mkv"
|
|
if ffmpeg -y -loglevel error -f lavfi -i testsrc=duration=1:size=160x120:rate=5 \
|
|
-c:v libx264 -pix_fmt yuv420p "$SEED_SRC" >/dev/null 2>&1; then
|
|
# 60 distinct episode files (the scanner ffprobes each) -> a multi-second scan even on fast CI
|
|
# I/O, so the poll->fire gap (one HTTP round-trip) is comfortably inside the lock window.
|
|
for n in $(seq -w 1 60); do cp "$SEED_SRC" "$SEED_SEASON/Show Lock - s01e$n.mkv"; done
|
|
# Shows is the built-in library Id=2 (docs/e2e-local.md recipe). Check the seed succeeded -- a
|
|
# silent failure would leave library 2 with no paths, the scan a no-op, and the 409 un-asserted.
|
|
if [ -z "$(seed_library_path "$SEED_ROOT" 2)" ]; then
|
|
bad "could not seed a LibraryPath row for the scan-lock flow (library 2)"
|
|
else
|
|
# No ?deep -- local scans always ForceScan (ForceScanLocalLibrary.ForceScan is hardcoded true;
|
|
# `deep` only matters for Plex/Jellyfin/Emby), so a fresh library 2 scan runs unconditionally.
|
|
expect_status "POST /libraries/2/scan -> 202 (queued)" 202 POST "$BASE_URL/api/v1/libraries/2/scan" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
# Poll scan-status until the scan is provably in flight, then fire the racing scan. The active
|
|
# window is a strict subset of the lock's held window, so this is deterministic bar a tiny
|
|
# residual TOCTOU gap (observe-active -> 2nd POST lands) that the multi-second scan covers.
|
|
# Match the id exactly (trailing comma) so "libraryId":2 can't match "libraryId":20 etc.
|
|
scan_active=""
|
|
for _ in $(seq 1 100); do
|
|
if api "$BASE_URL/api/v1/libraries/scan-status" | grep -q '"libraryId":2,'; then scan_active=1; break; fi
|
|
sleep 0.1
|
|
done
|
|
if [ -n "$scan_active" ]; then
|
|
ok "scan-status shows library 2 active (scan lock provably held)"
|
|
expect_status "second POST /libraries/2/scan while scanning -> 409" 409 POST "$BASE_URL/api/v1/libraries/2/scan" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
else
|
|
# Never observed active within 10s: don't assert a race we can't prove we won (advisory skip).
|
|
skip "library-2 scan never observed active within 10s -> not asserting the 409 (seed=60 files)"
|
|
fi
|
|
fi
|
|
else
|
|
bad "ffmpeg present but failed to synthesize the seed clip for the scan-lock flow"
|
|
fi
|
|
else
|
|
skip "ffmpeg not found -> skipping the library-scan 409 flow (advisory; needs seeded media + scanner)"
|
|
fi
|
|
|
|
# --- Flow B: external-collections already-scanning 409 (per-family lock; no ffmpeg/scanner) ---
|
|
# TEST-NET-1 (RFC 5737) is guaranteed non-routable, so the background sync hangs; but Flow B is
|
|
# race-free regardless (the 202 already proves the lock held -- the hang only widens the margin).
|
|
MSID="$(seed_jellyfin_source 'http://192.0.2.1:8096')"
|
|
if [ -n "$MSID" ]; then
|
|
ok "seeded Jellyfin media source id=$MSID (non-routable -> sync hangs, lock stays held)"
|
|
expect_status "POST /media-sources/jellyfin/$MSID/scan-collections -> 202" 202 POST "$BASE_URL/api/v1/media-sources/jellyfin/$MSID/scan-collections" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
# The lock is taken synchronously before the 202, so it is provably held now; corroborate via status.
|
|
if api "$BASE_URL/api/v1/media-sources/collections-scan-status" | grep -q '"family":"jellyfin"'; then
|
|
ok "collections-scan-status shows the jellyfin family locked"
|
|
else
|
|
bad "collections-scan-status did not show the jellyfin family locked after a 202"
|
|
fi
|
|
expect_status "second POST .../jellyfin/$MSID/scan-collections while scanning -> 409" 409 POST "$BASE_URL/api/v1/media-sources/jellyfin/$MSID/scan-collections" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
expect_status "POST .../jellyfin/99999/scan-collections (unknown source) -> 404" 404 POST "$BASE_URL/api/v1/media-sources/jellyfin/99999/scan-collections" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}'
|
|
else
|
|
bad "could not seed a Jellyfin media source row for the collections-lock flow"
|
|
fi
|
|
|
|
# --- Flow C: playout-build lock 409 + isLocked projection (#215/#444) -----------------------------
|
|
# Unlike Flows A/B, a playout build is enqueued onto the single-consumer WorkerService channel and the
|
|
# trigger returns BEFORE BuildPlayoutHandler dequeues and calls entityLocker.LockPlayout (released in its
|
|
# finally). So an accepted trigger does NOT prove the lock is held -- there is a real enqueue->dequeue
|
|
# race. We make it DETERMINISTIC by POLLING GET /playouts/{id} until isLocked:true (never sleep-and-hope),
|
|
# then firing the racing mutations while the lock is provably held. The build is made wide enough to
|
|
# observe by seeding a Classic Flood schedule over a few short episodes and cranking PlayoutDaysToBuild:
|
|
# a single 5-day build is ~43k playout items ~= ~1s here (#444 measurement); CI runners are slower, so
|
|
# the window only widens. Needs ffmpeg (to synthesize scannable media) -> advisory skip without it.
|
|
playout_locked() { api "$BASE_URL/api/v1/playouts/$1" | python3 -c "import json,sys
|
|
try: print(json.load(sys.stdin).get('isLocked'))
|
|
except Exception: print('')"; }
|
|
# isLocked for PLID as seen in the GET /playouts LIST projection (the other place #215 stamps it).
|
|
list_locked() { api "$BASE_URL/api/v1/playouts?pageSize=100" | PLID="$1" python3 -c "import json,sys,os
|
|
try:
|
|
d=json.load(sys.stdin); pid=int(os.environ['PLID'])
|
|
print(next((str(p.get('isLocked')) for p in d.get('page',[]) if p.get('id')==pid),'MISSING'))
|
|
except Exception: print('ERR')"; }
|
|
# PlayoutDaysToBuild is ConfigElementKey 'playout.days_to_build' (default 2); widen it to size the build.
|
|
set_days_to_build() { python3 - "$DB" "$1" <<'PY'
|
|
import sqlite3, sys
|
|
con = sqlite3.connect(sys.argv[1], timeout=10) # busy-timeout retry serializes this second writer
|
|
con.execute("DELETE FROM ConfigElement WHERE Key='playout.days_to_build'")
|
|
if sys.argv[2] != 'CLEAR':
|
|
con.execute("INSERT INTO ConfigElement (Key, Value) VALUES ('playout.days_to_build', ?)", (sys.argv[2],))
|
|
con.commit()
|
|
PY
|
|
}
|
|
# expect_409_while_locked "desc" METHOD URL [curl args...] -- assert a build-locked mutation returns 409,
|
|
# but if the build RELEASED before our call landed (a 2xx AND the playout is now confirmed unlocked),
|
|
# advisory-skip instead of failing: a lost race is not a bug, and this flow never reds on a race it can't
|
|
# prove it won. A 2xx while the playout is STILL locked is a real defect and still fails. ($BL_PL in scope.)
|
|
expect_409_while_locked() {
|
|
local desc="$1" method="$2" url="$3"; shift 3
|
|
local got; got="$(status_of "$method" "$url" "$@")"
|
|
if [ "$got" = "409" ]; then
|
|
ok "$desc (=409)"
|
|
elif { [ "$got" = "200" ] || [ "$got" = "202" ]; } && [ "$(playout_locked "$BL_PL")" = "False" ]; then
|
|
skip "$desc — build released before the racing call landed (got $got, now unlocked) -> advisory"
|
|
else
|
|
bad "$desc — expected 409, got $got"
|
|
fi
|
|
}
|
|
|
|
if ! command -v ffmpeg >/dev/null 2>&1; then
|
|
skip "ffmpeg not found -> skipping the playout-build lock 409 flow (advisory; needs seeded media)"
|
|
else
|
|
BL_ROOT="$(mktemp -d)/tv"; BL_SEASON="$BL_ROOT/Build Lock E2E/Season 01"; mkdir -p "$BL_SEASON"
|
|
BL_SRC="$(mktemp -d)/bl.mkv"
|
|
if ffmpeg -y -loglevel error -f lavfi -i "testsrc=duration=10:size=160x120:rate=5" \
|
|
-c:v libx264 -pix_fmt yuv420p "$BL_SRC" >/dev/null 2>&1; then
|
|
# A few 10s episodes: the flood item repeats the collection to fill the whole build window, so item
|
|
# count (hence build duration) is window/item-duration -- 3 distinct items are plenty to flood with.
|
|
for n in 1 2 3; do cp "$BL_SRC" "$BL_SEASON/Build Lock E2E - s01e0$n.mkv"; done
|
|
if [ -z "$(seed_library_path "$BL_ROOT" 2)" ]; then
|
|
bad "could not seed a LibraryPath row for the build-lock flow (library 2)"
|
|
else
|
|
# Wait for any in-progress library-2 scan (Flow A's) to clear so our scan won't 409, then scan and
|
|
# wait for OUR titled episodes to land (filtered by title, so Flow A's 'Show Lock' items don't leak in).
|
|
for _ in $(seq 1 200); do
|
|
api "$BASE_URL/api/v1/libraries/scan-status" | grep -q '"libraryId":2,' || break
|
|
sleep 0.1
|
|
done
|
|
api -X POST "$BASE_URL/api/v1/libraries/2/scan" -H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{}' -o /dev/null -w ''
|
|
BL_EPIDS=""
|
|
for _ in $(seq 1 200); do
|
|
BL_EPIDS=$(api "$BASE_URL/api/v1/library/browse?mediaType=Episode&pageSize=200" | python3 -c "import json,sys
|
|
d=json.load(sys.stdin)
|
|
print(','.join(str(i['mediaItemId']) for i in d.get('page',[]) if 'Build Lock E2E' in (i.get('title') or '')))")
|
|
[ -n "$BL_EPIDS" ] && break; sleep 0.15
|
|
done
|
|
if [ -z "$BL_EPIDS" ]; then
|
|
skip "build-lock flow: seeded episodes never scanned in within ~30s -> not asserting (advisory)"
|
|
else
|
|
ok "seeded + scanned build-lock episodes (ids: $BL_EPIDS)"
|
|
# Collection over the episodes -> a Classic Flood schedule item -> a Classic playout on a channel.
|
|
BL_COLL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/collections" -d '{"name":"E2E BuildLock Collection"}' -o "$BL_COLL_JSON" -w ''
|
|
BL_COLL="$(json_field "$BL_COLL_JSON" id)"
|
|
api -X POST "$BASE_URL/api/v1/collections/$BL_COLL/items" -d "{\"episodeIds\":[$BL_EPIDS]}" -o /dev/null -w ''
|
|
BL_SCH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/schedules" -d '{"name":"E2E BuildLock Schedule"}' -o "$BL_SCH_JSON" -w ''
|
|
BL_SCH="$(json_field "$BL_SCH_JSON" id)"
|
|
api -X PUT "$BASE_URL/api/v1/schedules/$BL_SCH/items" -o /dev/null -w '' -d "{\"items\":[{
|
|
\"startType\":\"Dynamic\",\"playoutMode\":\"Flood\",\"collectionType\":\"Collection\",\"collectionId\":$BL_COLL,
|
|
\"playbackOrder\":\"Chronological\",\"marathonGroupBy\":\"None\",\"marathonShuffleGroups\":false,
|
|
\"marathonShuffleItems\":false,\"fillWithGroupMode\":\"None\",\"multipleMode\":\"Count\",\"multipleCount\":\"\",
|
|
\"tailMode\":\"None\",\"customTitle\":\"\",\"guideMode\":\"Normal\",\"searchTitle\":\"\",\"searchQuery\":\"\",
|
|
\"preferredAudioLanguageCode\":\"\",\"preferredAudioTitle\":\"\",\"preferredSubtitleLanguageCode\":\"\",
|
|
\"watermarkIds\":[],\"graphicsElementIds\":[]}]}"
|
|
BL_CH_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/channels" -o "$BL_CH_JSON" -w '' -d '{
|
|
"name":"E2E BuildLock Channel","number":"79999","group":"E2E","categories":"","ffmpegProfileId":1,
|
|
"streamSelectorMode":"Default","streamSelector":"","playoutSource":"Generated","playoutMode":"Continuous",
|
|
"streamingMode":"TransportStream","subtitleMode":"None","musicVideoCreditsMode":"None","songVideoMode":"Default",
|
|
"transcodeMode":"OnDemand","idleBehavior":"StopOnDisconnect","isEnabled":true,"showInEpg":true}'
|
|
BL_CH="$(json_field "$BL_CH_JSON" id)"
|
|
set_days_to_build 5
|
|
# Creating a Classic playout enqueues a Reset build (CreateClassicPlayoutHandler) -- the build we observe.
|
|
BL_PL_JSON="$(mktemp)"; api -X POST "$BASE_URL/api/v1/playouts" -o "$BL_PL_JSON" -w '' \
|
|
-d "{\"channelId\":$BL_CH,\"scheduleKind\":\"Classic\",\"programScheduleId\":$BL_SCH}"
|
|
BL_PL="$(json_field "$BL_PL_JSON" id)"
|
|
if [ -z "$BL_PL" ]; then
|
|
bad "could not create the Classic playout for the build-lock flow (body: $(cat "$BL_PL_JSON"))"
|
|
else
|
|
bl_locked=""
|
|
for _ in $(seq 1 300); do
|
|
[ "$(playout_locked "$BL_PL")" = "True" ] && { bl_locked=1; break; }
|
|
sleep 0.05
|
|
done
|
|
if [ -z "$bl_locked" ]; then
|
|
# Never observed locked (build finished faster than we sampled): don't assert a race we can't
|
|
# prove we won -- advisory skip, matching Flow A's discipline.
|
|
skip "playout build never observed locked within ~15s -> not asserting the 409 (advisory; build too fast)"
|
|
else
|
|
ok "playout $BL_PL build observed locked (GET /playouts/{id} isLocked:true projection)"
|
|
# While the lock is provably held: every id-keyed mutation 409s and both isLocked projections
|
|
# agree. Each racing call is guarded (expect_409_while_locked / the list re-check below) so a
|
|
# build that finishes mid-flight degrades to an advisory skip, never a false red.
|
|
expect_409_while_locked "PUT /playouts/$BL_PL while build in progress -> 409" PUT "$BASE_URL/api/v1/playouts/$BL_PL" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}'
|
|
expect_409_while_locked "POST /channels/$BL_CH/playout/reset while build in progress -> 409" POST "$BASE_URL/api/v1/channels/$BL_CH/playout/reset?mode=Reset" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json'
|
|
bl_list="$(list_locked "$BL_PL")"
|
|
if [ "$bl_list" = "True" ]; then
|
|
ok "GET /playouts list projection shows isLocked:true for $BL_PL"
|
|
elif [ "$bl_list" = "False" ]; then
|
|
skip "GET /playouts list projection already isLocked:false (build released mid-flight) -> advisory"
|
|
else
|
|
bad "GET /playouts list projection error for $BL_PL (got '$bl_list')"
|
|
fi
|
|
# Once the build completes the lock clears: isLocked flips false and the SAME PUT now succeeds,
|
|
# proving the 409 above was lock-specific (not an always-failing call).
|
|
for _ in $(seq 1 600); do [ "$(playout_locked "$BL_PL")" = "False" ] && break; sleep 0.05; done
|
|
if [ "$(playout_locked "$BL_PL")" = "False" ]; then
|
|
ok "playout $BL_PL isLocked:false after the build completes"
|
|
else
|
|
bad "playout $BL_PL never unlocked after the build"
|
|
fi
|
|
expect_status "PUT /playouts/$BL_PL after the build (unlocked) -> 200" 200 PUT "$BASE_URL/api/v1/playouts/$BL_PL" \
|
|
-H "X-Api-Key: $API_KEY" -H 'Content-Type: application/json' -d '{"dailyRebuildTime":null,"scheduleFile":null}'
|
|
fi
|
|
fi
|
|
set_days_to_build CLEAR # restore the default build horizon
|
|
fi
|
|
fi
|
|
else
|
|
bad "ffmpeg present but failed to synthesize the seed clip for the build-lock flow"
|
|
fi
|
|
fi
|
|
fi
|
|
|
|
# ---- 4. Auth / CSRF / security-stamp flow -------------------------------------------------------
|
|
# Runs last: setup-claim is a one-shot on a fresh config and logout revokes the session. All prior
|
|
# sections used the machine key, which does not claim the admin, so setupRequired is still true here.
|
|
section "Auth: setup-claim, CSRF gate, login, logout + stamp revocation"
|
|
|
|
CFG_JSON="$(mktemp)"
|
|
curl -s "$BASE_URL/api/v1/auth/config" -o "$CFG_JSON"
|
|
if [ "$(json_field "$CFG_JSON" setupRequired)" = "True" ]; then ok "fresh config: setupRequired=true"; else bad "expected setupRequired=true on a fresh config (body: $(cat "$CFG_JSON"))"; fi
|
|
|
|
JAR="$(mktemp)"
|
|
# Claim the local admin (setup is CSRF-gated like every mutation).
|
|
expect_status "POST /api/v1/auth/setup -> 200 (claims admin, issues session)" 200 POST "$BASE_URL/api/v1/auth/setup" \
|
|
-c "$JAR" -H 'Content-Type: application/json' -H 'X-CSRF: 1' -d '{"username":"admin","password":"e2e-pass-1234"}'
|
|
curl -s "$BASE_URL/api/v1/auth/config" -o "$CFG_JSON"
|
|
if [ "$(json_field "$CFG_JSON" setupRequired)" = "False" ]; then ok "after claim: setupRequired=false"; else bad "expected setupRequired=false after claim (body: $(cat "$CFG_JSON"))"; fi
|
|
expect_status "POST /api/v1/auth/setup again -> 409 (already configured)" 409 POST "$BASE_URL/api/v1/auth/setup" \
|
|
-H 'Content-Type: application/json' -H 'X-CSRF: 1' -d '{"username":"admin2","password":"e2e-pass-1234"}'
|
|
|
|
# CSRF gate on a general /api session mutation: a POST with the session cookie but no X-CSRF is 403.
|
|
expect_status "session mutation without X-CSRF -> 403" 403 POST "$BASE_URL/api/v1/collections" \
|
|
-b "$JAR" -H 'Content-Type: application/json' -d '{"name":"should-be-csrf-blocked"}'
|
|
|
|
# Login: wrong password 401, correct 200 (fresh cookie jar).
|
|
expect_status "login with wrong password -> 401" 401 POST "$BASE_URL/api/v1/auth/login" \
|
|
-H 'Content-Type: application/json' -H 'X-CSRF: 1' -d '{"username":"admin","password":"nope"}'
|
|
JAR2="$(mktemp)"
|
|
expect_status "login with correct password -> 200" 200 POST "$BASE_URL/api/v1/auth/login" \
|
|
-c "$JAR2" -H 'Content-Type: application/json' -H 'X-CSRF: 1' -d '{"username":"admin","password":"e2e-pass-1234"}'
|
|
|
|
# Logout is CSRF-gated; and after it, the same cookie is invalidated by the rotated security stamp.
|
|
expect_status "authed GET before logout -> 200" 200 GET "$BASE_URL/api/v1/auth/machine-key" -b "$JAR2"
|
|
expect_status "logout without X-CSRF -> 403" 403 POST "$BASE_URL/api/v1/auth/logout" -b "$JAR2"
|
|
expect_status "logout with X-CSRF -> 204" 204 POST "$BASE_URL/api/v1/auth/logout" -b "$JAR2" -H 'X-CSRF: 1'
|
|
expect_status "authed GET after logout (stamp revoked) -> 401" 401 GET "$BASE_URL/api/v1/auth/machine-key" -b "$JAR2"
|
|
|
|
# ---- summary ------------------------------------------------------------------------------------
|
|
printf '\n\033[1m== Summary ==\033[0m\n'
|
|
printf 'passed: %d failed: %d\n' "$PASS" "$FAIL"
|
|
if [ "$FAIL" -ne 0 ]; then
|
|
printf '\033[31mfailed assertions:\033[0m\n'
|
|
for f in "${FAILURES[@]}"; do printf ' - %s\n' "$f"; done
|
|
exit 1
|
|
fi
|
|
echo "all functional-E2E assertions passed."
|