ersatztv/docs/e2e-local.md

Local live-E2E recipe

Purpose: how to stand up a real, running instance of this fork locally (dotnet host + built SPA) for manual or Playwright-MCP-driven end-to-end verification — no live Docker/prod dependency. Update this doc (and scripts/e2e-local.sh) in the same PR that changes any convention below.

This is for interactive/agent-driven verification, not CI (docs/ci-cd.md covers the CI pipeline, which never runs the app itself).

When live-E2E is required (not optional)

A live-E2E pass through this recipe is a required step — not a nicety — for any PR that changes an API write-path handler: a POST/PUT/DELETE /api/* command that mutates state and then reloads it through the read path. Reason: this class has a correlated blind spot that unit and characterization tests share. A green fixed-point test passed while a write-path returned a 500 in production because the handler returned a lazy LanguageExt Map the test never enumerated (#229, PR — see docs/decisions.md and api-conventions.md §7); the reload-through-read-path mechanics can throw only when the result is actually materialised, which the SPA does and the test did not. Live driving the real screen (or curling the real endpoint) is the only net that reliably catches it.

Concretely, for a write-path PR: stand up the instance (Steps below), exercise the changed create/update/delete flow against the real endpoint or its SPA screen, and confirm the mutation round-trips through a subsequent read (list/detail/browse) — not just that the write returned 2xx. Pure-SPA/read-only or docs PRs don't need it. State in the PR/close comment that live-E2E ran (or, for a non-write-path change, that it wasn't required) — the same auditable-exemption rule the review skip rubric uses (kickoff workflow lore).

Why the steps are in this order

• ErsatzTV/Startup.cs's /app SPA middleware resolves its static-file root once, at startup (SpaStaticFileRoot(), checked with Directory.Exists and swapped to a NullFileProvider if missing — see Startup.cs around the app.MapWhen(... "/app" ...) block). If wwwroot/app doesn't exist yet when the process starts, the SPA will 404 forever until you restart the process — copying the built files in after the fact does nothing for an already-running instance.
• The dotnet host and the SPA share one port (default 8409, ETV_UI_PORT / SystemEnvironment. UiPort in ErsatzTV.Core/SystemEnvironment.cs) — there's no separate dev server/proxy in this workflow; you're testing the actual production static-hosting path.
• A fresh config folder per run avoids state bleed (leftover channels/schedules/DB) between test sessions corrupting your assertions.

Steps

1. Build the prerequisites (once, or after any source change):

   dotnet build ErsatzTV.sln
   cd web && npm run build && cd ..   # → ErsatzTV/wwwroot/app (vite.config.ts outDir)
   

2. Copy wwwroot into the build output (the dotnet build output directory does not automatically pick up web/'s build artifacts placed directly into the source wwwroot):

   rm -rf ErsatzTV/bin/Debug/net10.0/wwwroot
   cp -R ErsatzTV/wwwroot ErsatzTV/bin/Debug/net10.0/wwwroot
   

   The rm matters: if the destination dir already exists (any prior run), cp -R src dst copies into it (dst/wwwroot/...) and the server silently keeps serving the previous run's stale assets. scripts/e2e-local.sh does the rm+copy for you. If you rebuild the SPA (npm run build) while the dotnet process from step 4 is already running, re-copy and then restart the process — see the "why" note above; it will not pick up new files live.

3. Use a fresh scratch config folder — never reuse one across test runs:

   CONFIG_DIR=$(mktemp -d)
   

4. Run the app, pointed at the scratch folder:

   cd ErsatzTV/bin/Debug/net10.0
   ETV_CONFIG_FOLDER="$CONFIG_DIR" dotnet ErsatzTV.dll
   

   Wait for the log line Done migrating search index (emitted by RebuildSearchIndexHandler in ErsatzTV.Application/Search/Commands/, with a ... in {Duration} suffix) — that's the last long-running startup step; before that, requests may 404/error. The UI and the entire /api/* surface are served on the same port, 8409 by default (override with ETV_UI_PORT).

   On a reused config dir that line never appears. RebuildSearchIndexHandler logs one of two mutually-exclusive lines immediately before SystemStartup.SearchIndexIsReady(), and which one depends on whether the index had anything to migrate:

   Config dir	Line logged
   fresh	Done migrating search index in {Duration}
   reused (index already current)	Search index is already version {Version}

   So when waiting on readiness, match either line — the handler's if/else is exhaustive, so the pair covers every path to ready. Watching only for Done migrating search index against a reused dir waits out the full timeout and then kills a perfectly healthy server; that cost one session two failed launches before the cause was found (ersatztv#533).

5. Seed data as needed via the API. The /api surface is fail-closed since #197 (writes always, and reads because Api:RequireKeyForReads defaults true), so curl seeding must send the machine key the server generates at startup: -H "X-Api-Key: $(cat "$CONFIG_DIR/api.key")". (Mutations from a session also need -H 'X-CSRF: 1', but for scripted seeding the machine key is simpler and CSRF-immune.) Examples:

   • Create a channel: POST /api/v1/channels — check the current ErsatzTV/wwwroot/openapi/v1.json (or CreateChannelRequest.cs) for the exact required field list before assuming these are complete, but as of this writing it requires (among plain fields) these enums: PlayoutSource: "Generated", PlayoutMode: "Continuous", SongVideoMode: "Default", TranscodeMode: "OnDemand", IdleBehavior: "StopOnDisconnect".
   • Create a playout for an existing channel: POST /api/v1/playouts with {"channelId": <id>, "scheduleKind": "Block"} (or "Classic"/"Scripted"/"Sequential" per ChannelPlayoutSource/schedule-kind enums — check v1.json for the current set).

5b. Browser flow (since #295 session auth): the first /app load hits a boot gate. On a fresh config it shows Setup — claim the local admin (pick any username/password); that issues the session cookie and drops you into the app. On a subsequent run of the same config it shows Login instead. (To skip the browser claim in automation, set Auth:LocalAdmin:Password before launch — the env seed provisions the admin and disables the browser setup-claim.) The machine key still works for curl seeding regardless. Driving this with Playwright: fill the Setup/Login form before asserting any authed screen.

6. Tear down: kill the dotnet ErsatzTV.dll process by the PID you captured at launch, and confirm the port is freed (lsof -i :8409 should return nothing) before starting another run — a stray process holding the port will make the next run's health check hang or fail confusingly.

   ⚠️ Kill by PID. Never pkill -f a pattern.

   This machine is shared by parallel sessions, and several of them run this same binary at once. pkill -f "dotnet ErsatzTV.dll" reaps every one of them, not just yours.

   kill "$PID"                            # ✅ the PID e2e-local.sh printed
   pkill -f "dotnet ErsatzTV.dll"         # ❌ NEVER — reaps other sessions' servers
   

   Filtering by port does not make a pattern kill safe — pkill -f matches the command line, not the port, so it hits every instance whatever port each one chose. Nor are the ports actually separated: the CI functional-e2e step exports ETV_UI_PORT=8409, the same port local runs use. (8410 is the ersatztv-test container on jazz — a different thing; the ui-E2E step from #445 does use it.) Scope by PID, not by pattern and not by port.

   If the port is already busy, that process is not yours — diagnose, don't reap. Report it and move to another port; scripts/e2e-local.sh pre-flights this for you and prints the foreign PID:

   lsof -ti :8409                         # who is holding it
   ETV_UI_PORT=8420 scripts/e2e-local.sh  # your run, out of the way
   

   Launching the DLL by hand? Set both ports. Program.cs binds a second listener on ETV_STREAMING_PORT, which defaults to 8409 regardless of ETV_UI_PORT — so ETV_UI_PORT=8420 dotnet ErsatzTV.dll still binds 8409 and dies against a foreign holder:

   ETV_UI_PORT=8420 ETV_STREAMING_PORT=8420 dotnet ErsatzTV.dll
   

   scripts/e2e-local.sh defaults ETV_STREAMING_PORT to whatever port you gave it, so through the script ETV_UI_PORT=8420 alone is sufficient.

   Why this is a hard rule rather than a preference: killing another session's harness mid-run does not fail loudly. It truncates that run's output into plausible-looking-but-wrong data — the failure class that survives review. A near-miss of exactly this shape is recorded in docs/decisions/workflow-process.md → testing.e2e-cleanup-scope-by-pid (ersatztv#586).

   Writing a brief for a delegated agent? Put this constraint in it. The #586 incident was a delegation gap, not agent error: the brief specified a fresh config dir but said nothing about process cleanup, so the agent invented a reasonable-looking pattern kill. An omitted rule is not an unenforced rule — it is a rule replaced by whatever plausible default the agent reaches for.

Playwright MCP screenshots

If you're driving the browser via the Playwright MCP server for visual verification, screenshots land in the MCP server process's own cwd (this repo's root, not wherever you ran the dotnet process from) — expect stray *.png files at the repo root after a session; this is tolerated, not a bug to fix, but don't check them in.

Script: scripts/e2e-local.sh

A copy of this script is included in this doc's directory; it is intended to land at scripts/e2e-local.sh in the repo. It automates steps 2–4 above (build is assumed already done — run dotnet build / npm run build yourself first, since rebuilding on every invocation is slow and this script is meant to be re-run often during a debugging session).

Usage:

scripts/e2e-local.sh [CONFIG_DIR]

• CONFIG_DIR defaults to a fresh mktemp -d if omitted. Prefer a fresh dir per run for state-bleed reasons (above); the readiness probe itself handles a reused dir since #533.
• ⚠️ Never run two instances concurrently, even on different ports. Each run rm -rfs and re-copies the same ErsatzTV/bin/<config>/net10.0/wwwroot, so the second run pulls the static files out from under a still-starting first instance. The symptom — a readiness timeout, or /app 404ing on the instance that did start — looks nothing like a shared-directory race. Run sequentially; the CI job's curl and UI-E2E steps do.
• ETV_PIDFILE (optional): the server PID is written here the instant it forks, before the readiness wait. scripts/e2e-ui.sh sets it so it can still reap the server if it is killed mid-boot, before it has parsed PID= from stdout. Unset means unchanged behaviour.
• Copies ErsatzTV/wwwroot → ErsatzTV/bin/<config>/net10.0/wwwroot.
• Launches dotnet ErsatzTV.dll in the background with ETV_CONFIG_FOLDER set.
• Waits (up to 120s) for readiness — either Done migrating search index (fresh config) or Search index is already version (reused config); see the step-4 table above.
• Pre-flights both bound ports (ETV_UI_PORT and ETV_STREAMING_PORT): if something is already listening, it fails immediately naming the offending PID, rather than letting Kestrel fail its bind a moment later and surface as process N exited before becoming ready plus a log tail — a framing that reads like a broken build. It reports, never reaps — on this shared machine a foreign listener is most likely another session's harness mid-run. Re-run with ETV_UI_PORT=<other>.
• Defaults ETV_STREAMING_PORT to ETV_UI_PORT so that re-run actually works: the app binds a second listener that otherwise defaults to 8409 no matter what the UI port is. An explicit ETV_STREAMING_PORT still wins, and CI (which uses 8409) is unaffected.
• Prints the PID and port, then exits leaving the server running — the caller owns that PID and is responsible for killing it when done (kill "$PID", never a pkill -f pattern — see the teardown warning in step 6).
• Does not trap-and-kill on exit, by design — unlike scripts/e2e-ui.sh, which owns its instance's whole lifecycle and kills the server from an EXIT INT TERM trap. The two scripts sit on opposite sides of that contract on purpose: this one is the launcher and hands a running instance to its caller (an EXIT trap here would kill the server the instant it returned, breaking every caller including e2e-ui.sh and the CI functional-e2e step); e2e-ui.sh is a lifecycle owner and traps. If you write a new harness that boots and then finishes on its own, follow e2e-ui.sh: capture the PID and trap.
• ETV_BUILD_CONFIG selects which build output to launch (Debug default for local dev; the CI functional-e2e job sets Release). It must match the dotnet build --configuration you ran first — the script only copies wwwroot + launches; it does not build.

Functional-E2E harness: scripts/e2e-functional.sh

The ad-hoc curl scenarios sessions run against a live instance (redirect sweeps, auth flows, the scan status contract, If-Match/412) are codified into a single harness so they run identically by hand and in CI (the advisory functional-e2e job — docs/ci-cd.md). It does not boot the app; pair it with scripts/e2e-local.sh above (or point it at any running instance):

CFG=$(mktemp -d)
eval "$(scripts/e2e-local.sh "$CFG" | sed -n 's/^\(PID\|PORT\)=/\1=/p')"   # launch, capture PID/PORT
scripts/e2e-functional.sh "http://localhost:${PORT}" "$CFG"                 # assert; exit 1 on any failure
kill "$PID"

CONFIG_DIR (arg 2) is required — the harness reads the instance's machine key from $CONFIG_DIR/api.key and sends X-Api-Key on every /api call (the surface is fail-closed). It runs each assertion even after a failure and prints a pass/fail summary, exiting non-zero if any failed.

What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browser); the lock-contention section (added by ersatztv#363, extended by #444) is the exception — it seeds DB rows + media.

• Legacy→SPA redirects: a sweep of representative LegacyUiRedirects routes 302→/app/*, plus the /api + /artwork never-redirect exemption (asserted as "did not 302 to /app", since those 4xx from their own handlers — /api/* 404s, /artwork/* 400s).
• Library-scan status contract: create an empty local library → scan 202, unknown-library scan 404, scan-status 200.
• Optimistic concurrency: create a collection + a rerun-collections targeting it → GET emits an ETag → PUT with a stale If-Match 412, current 200, malformed 400.
• Lock contention (409) — the three IEntityLocker 409s, made deterministic by firing the racing request only once the lock is provably held (never a sleep-and-hope). All seed rows the API can't create straight into the running instance's DB via python3's stdlib sqlite3 (whose busy-timeout retry serializes behind the app's writer):
  • library-scan "already scanning": seed ~60 tiny ffmpeg clips into the built-in Shows library (Id=2) so the scanner subprocess runs a few seconds → POST .../scan 202 (local scans always ForceScan, so no deep needed) → poll GET /libraries/scan-status until library 2 shows active (that window is a strict subset of the scan lock's held window — ScannerProxyService.StartScan fires after LockLibrary, EndScan before UnlockLibrary) → a second POST .../scan is a 409, deterministic bar a tiny residual TOCTOU gap the multi-second scan covers. Self-skips (advisory) if ffmpeg is absent.
  • external-collections "already scanning": seed a Jellyfin media-source row pointing at a non-routable address (so the background sync hangs and the per-family lock stays held). The lock is taken synchronously before the 202, so the 202 proves it held → collections-scan-status shows the jellyfin family → a second POST .../scan-collections is 409; unknown source 404. Needs neither ffmpeg nor the scanner.
  • playout-build "build in progress" (#215/#444, the hardest): a build is enqueued onto the single-consumer WorkerService channel and the trigger returns before BuildPlayoutHandler dequeues and LockPlayouts (released in its finally), so an accepted trigger does not prove the lock is held — poll GET /playouts/{id} until isLocked:true, then fire. Seed a few short ffmpeg episodes → a Collection → a Classic Flood schedule → a Classic playout, and crank PlayoutDaysToBuild (config playout.days_to_build) so a single build is wide enough to observe: 5 days ≈ 43k playout items ≈ ~1s locally, wider on slower CI (sized by measurement — item count is window / item-duration, so the flood over a handful of short clips scales with the day count; going too wide is counter-productive — a 777k-item build saturates the single worker with its post-build gap/overlap jobs). While provably locked: PUT /playouts/{id} → 409, POST .../playout/reset → 409, and the GET /playouts list projection shows isLocked:true; after the build the lock clears (isLocked:false) and the same PUT now 200s (proving the 409 is lock-specific). Restores the config default afterwards. Self-skips (advisory) without ffmpeg, or if the build is never observed locked (never asserts a race it can't prove it won).
• Auth/CSRF/security-stamp (mirrors the #295 manual set; runs last — setup-claim is one-shot and logout revokes the session): fresh auth/config setupRequired:true → read-gate 401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation 403-without-X-CSRF → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the same cookie is 401 (rotated security stamp).

The playout-build lock 409 + isLocked projection (#215) landed in #444 — see the lock-contention bullet above. The genuinely UI-interactive Playwright flows landed in #445 — see the next section. When you extend the harness, add the observed contract here and keep the assertions deterministic (probe the real instance first — the initial cut caught that /artwork/* 400s where a guess said 404; the #363 cut measured the real scan window before sizing the seed).

UI-E2E harness: scripts/e2e-ui.sh + web/e2e/*.spec.ts

The handful of contracts that cannot be expressed as curl calls, driven by headless Playwright (never claude-in-chrome, per project convention — ersatztv#445). Unlike e2e-functional.sh, this script owns the whole lifecycle: it boots a fresh instance, runs the specs, and always kills the server (trap on EXIT/INT/TERM), exiting with Playwright's status:

scripts/e2e-ui.sh                       # fresh config dir, all specs
scripts/e2e-ui.sh "$(mktemp -d)" --headed --debug   # extra args pass through to `playwright test`

A fresh config dir is required, not merely preferred — the first spec asserts the one-shot Setup (first-run admin claim) gate, which only exists while the server has no local admin. Pointing it at a reused dir fails that spec by design.

What it covers, and the rule for extending it. Only assert here what curl structurally cannot — the curl harness above already covers the auth HTTP contracts, so re-asserting them through a browser buys nothing but flake surface. What qualifies:

Contract	Why curl can't reach it
Setup card's confirm-password gate	Pure React state; makes no request, so there is no HTTP contract
AuthGate states (Setup vs Login vs app)	Assertion is about what renders, not a status code
Session cookie on the SPA's own /api XHRs	curl proves the cookie works for curl, not that the app sends it
Sign-out via the UserMenu	A DOM interaction chain, not a single endpoint
A client-side field bound that gates the Save button (ffmpeg-pacing.spec.ts, #735)	The rejected draft never leaves the browser — the screen renders the message and disables Save, so there is no request for curl to observe. The server's own 422 for the same input IS curl-reachable and belongs there, not here

Determinism rules (#445 asked for deterministic flows, so these are deliberate):

• retries: 0, in CI too — a retry would let a genuinely flaky flow merge looking green.
• serial + workers: 1: server state is shared and partly one-shot (the setup-claim).
• Each test still gets its own browser context (own cookie jar) — that is what gives the login specs a genuinely signed-out browser without a logout dance. Anything needing a session held across steps must stay inside one test.
• Address form fields by their unique placeholders, not labels: the shared <Input> wraps its <input> in a <label>, so a field's accessible name absorbs its error text when invalid ("Confirm password Passwords do not match."). A settings-row field is the other way round — Row's label is a plain <div>, not bound to the control, so such an input has no accessible name at all until the screen passes <Input ariaLabel>. Pass it when you add a settings field you intend to reach by role (FFmpegProfilesScreen's pacing fields do); do not fall back to positional selectors.
• A fresh config is not empty — DbInitializer seeds one default channel ("ErsatzTV", number 1), so the Channels empty state is unreachable there; assert the seeded row instead.

The browser is baked into the CI toolchain image, not installed per run (docker/ci/Dockerfile: chromium-headless-shell, 267M vs 656M for full chromium). Two consequences:

• web/package.json's @playwright/test pin is EXACT (no caret) because Playwright ties a browser revision to the package version. Bumping it requires rebuilding the CI image — scripts/e2e-ui.sh guards this by launching a browser up front and failing with that instruction. See docs/ci-cd.md → "CI toolchain image".
• A headed run inside the CI image would fail (headless shell only). Locally, run cd web && npx playwright install chromium once and headed works from your own cache.

Vitest deliberately excludes e2e/** (web/vite.config.ts) — its default **/*.spec.* glob would otherwise try to run these under jsdom.

Seeding a local TV library for E2E

Channels/playouts are API-seedable (step 5 above), but a local media library is not — there is no /api/* endpoint to add a local library folder. To exercise media-browse / search / detail screens you need real scanned items. Recipe (used to verify the #220 episode-nav PR):

1. Generate tiny media files on disk — one show (one season, ~3 episodes), plus a second show whose title contains the first as a substring (good substring-search sanity data), plus a movie if you need a non-episode kind. Keep TV and movies under separate roots so each library scans cleanly (a Shows library pointed at a folder that also contains movies will try to parse the movies as shows). Each file is a 2-second testsrc clip:

   MEDIA=/tmp/etv-media    # any scratch path
   mkdir -p "$MEDIA/tv/Show Alpha/Season 01" \
            "$MEDIA/tv/Show Alpha Returns/Season 01" \
            "$MEDIA/movies/Test Movie (2020)"
   for n in 01 02 03; do
     ffmpeg -y -f lavfi -i testsrc=duration=2:size=320x240:rate=10 -c:v libx264 -pix_fmt yuv420p \
       "$MEDIA/tv/Show Alpha/Season 01/Show Alpha - s01e$n.mkv"
   done
   ffmpeg -y -f lavfi -i testsrc=duration=2:size=320x240:rate=10 -c:v libx264 -pix_fmt yuv420p \
     "$MEDIA/tv/Show Alpha Returns/Season 01/Show Alpha Returns - s01e01.mkv"
   ffmpeg -y -f lavfi -i testsrc=duration=2:size=320x240:rate=10 -c:v libx264 -pix_fmt yuv420p \
     "$MEDIA/movies/Test Movie (2020)/Test Movie (2020).mkv"
   

2. Attach the folders to the built-in local libraries via SQLite. A fresh config DB already has the seven default local libraries (Library rows for a single LocalMediaSource): Movies is Id=1, Shows is Id=2. LibraryPath is just (Path TEXT, LibraryId INT) — insert one row per root, pointing each at the matching library:

   DB="$CONFIG_DIR/ersatztv.sqlite3"    # CONFIG_DIR from the run above; server may be running
   sqlite3 "$DB" "INSERT INTO LibraryPath (Path, LibraryId) VALUES ('$MEDIA/tv', 2);"     # Shows
   sqlite3 "$DB" "INSERT INTO LibraryPath (Path, LibraryId) VALUES ('$MEDIA/movies', 1);" # Movies
   

3. Trigger a scan and wait for items to appear. The scan endpoint takes an empty body:

   curl -s -X POST http://localhost:8409/api/v1/libraries/2/scan -H 'Content-Type: application/json' -d '{}'
   curl -s -X POST http://localhost:8409/api/v1/libraries/1/scan -H 'Content-Type: application/json' -d '{}'
   # poll until episodes show up (scanner runs as a background subprocess):
   curl -s "http://localhost:8409/api/v1/library/browse?mediaType=Episode&pageSize=50"
   

   The scan runs even though LibraryPath was inserted after startup — the scan handler re-reads the library from the DB. Browse (/api/v1/library/browse) reads straight from the DB, so items appear there within a few seconds.

Gotchas

• Do NOT delete the search-index/ folder to "reset" search. On startup the app recreates the index empty (Search index failed to initialize; will delete and recreate → Migrating search index to version N) and that migration does not re-index from the DB — only a scan writes documents into the Lucene index. The scanner subprocess writes the index while running; a restart never rebuilds it from existing DB rows. If you wipe search-index/, a rescan of unchanged files won't repopulate it (the scanner skips unchanged items), so search stays empty. The clean recovery is a fresh CONFIG_DIR: launch → insert LibraryPath → scan once → leave the index alone.
• Search query relevance is field-scoped, not free-text. The /api/v1/search default field does not match bare title words: Alpha and Show return nothing for a "Show Alpha" title, while title:Alpha, Show*, or *Alpha* all match. The SPA search box forwards the query verbatim, so when driving search-result screens in E2E use a field/wildcard query (e.g. title:Alpha) to get deterministic hits. (This is pre-existing ErsatzTV search behavior, independent of any SPA change.)