# Local live-E2E recipe Purpose: how to stand up a real, running instance of this fork locally (dotnet host + built SPA) for manual or Playwright-MCP-driven end-to-end verification — no live Docker/prod dependency. **Update this doc (and `scripts/e2e-local.sh`) in the same PR that changes any convention below.** This is for interactive/agent-driven verification, not CI (`docs/ci-cd.md` covers the CI pipeline, which never runs the app itself). ## When live-E2E is required (not optional) A live-E2E pass through this recipe is a **required** step — not a nicety — for any PR that changes an **API write-path handler**: a `POST`/`PUT`/`DELETE` `/api/*` command that mutates state and then reloads it through the read path. Reason: this class has a **correlated blind spot** that unit and characterization tests share. A green fixed-point test passed while a write-path returned a 500 in production because the handler returned a *lazy* LanguageExt `Map` the test never enumerated (#229, PR — see `docs/decisions.md` and `api-conventions.md` §7); the reload-through-read-path mechanics can throw only when the result is actually materialised, which the SPA does and the test did not. Live driving the real screen (or `curl`ing the real endpoint) is the only net that reliably catches it. Concretely, for a write-path PR: stand up the instance (Steps below), exercise the changed create/update/delete flow against the real endpoint or its SPA screen, and confirm the mutation **round-trips through a subsequent read** (list/detail/browse) — not just that the write returned 2xx. Pure-SPA/read-only or docs PRs don't need it. State in the PR/close comment that live-E2E ran (or, for a non-write-path change, that it wasn't required) — the same auditable-exemption rule the review skip rubric uses (kickoff workflow lore). ## Why the steps are in this order - **`ErsatzTV/Startup.cs`'s `/app` SPA middleware resolves its static-file root once, at startup** (`SpaStaticFileRoot()`, checked with `Directory.Exists` and swapped to a `NullFileProvider` if missing — see `Startup.cs` around the `app.MapWhen(... "/app" ...)` block). If `wwwroot/app` doesn't exist yet when the process starts, **the SPA will 404 forever until you restart the process** — copying the built files in after the fact does nothing for an already-running instance. - The dotnet host and the SPA share **one port** (default 8409, `ETV_UI_PORT` / `SystemEnvironment. UiPort` in `ErsatzTV.Core/SystemEnvironment.cs`) — there's no separate dev server/proxy in this workflow; you're testing the actual production static-hosting path. - A fresh config folder per run avoids state bleed (leftover channels/schedules/DB) between test sessions corrupting your assertions. ## Steps 1. **Build the prerequisites** (once, or after any source change): ```bash dotnet build ErsatzTV.sln cd web && npm run build && cd .. # → ErsatzTV/wwwroot/app (vite.config.ts outDir) ``` 2. **Copy `wwwroot` into the build output** (the `dotnet build` output directory does not automatically pick up `web/`'s build artifacts placed directly into the source `wwwroot`): ```bash rm -rf ErsatzTV/bin/Debug/net10.0/wwwroot cp -R ErsatzTV/wwwroot ErsatzTV/bin/Debug/net10.0/wwwroot ``` The `rm` matters: if the destination dir already exists (any prior run), `cp -R src dst` copies **into** it (`dst/wwwroot/...`) and the server silently keeps serving the previous run's stale assets. `scripts/e2e-local.sh` does the rm+copy for you. If you rebuild the SPA (`npm run build`) while the dotnet process from step 4 is already running, **re-copy and then restart the process** — see the "why" note above; it will not pick up new files live. 3. **Use a fresh scratch config folder** — never reuse one across test runs: ```bash CONFIG_DIR=$(mktemp -d) ``` 4. **Run the app**, pointed at the scratch folder: ```bash cd ErsatzTV/bin/Debug/net10.0 ETV_CONFIG_FOLDER="$CONFIG_DIR" dotnet ErsatzTV.dll ``` Wait for the log line **`Done migrating search index`** (emitted by `RebuildSearchIndexHandler` in `ErsatzTV.Application/Search/Commands/`, with a `... in {Duration}` suffix) — that's the last long-running startup step; before that, requests may 404/error. The UI and the entire `/api/*` surface are served on the **same port**, 8409 by default (override with `ETV_UI_PORT`). **On a reused config dir that line never appears.** `RebuildSearchIndexHandler` logs one of two mutually-exclusive lines immediately before `SystemStartup.SearchIndexIsReady()`, and which one depends on whether the index had anything to migrate: | Config dir | Line logged | | --- | --- | | fresh | `Done migrating search index in {Duration}` | | reused (index already current) | `Search index is already version {Version}` | So when waiting on readiness, match **either** line — the handler's `if`/`else` is exhaustive, so the pair covers every path to ready. Watching only for `Done migrating search index` against a reused dir waits out the full timeout and then kills a perfectly healthy server; that cost one session two failed launches before the cause was found (ersatztv#533). 5. **Seed data** as needed via the API. **The `/api` surface is fail-closed since #197** (writes always, and reads because `Api:RequireKeyForReads` defaults true), so curl seeding must send the machine key the server generates at startup: `-H "X-Api-Key: $(cat "$CONFIG_DIR/api.key")"`. (Mutations from a *session* also need `-H 'X-CSRF: 1'`, but for scripted seeding the machine key is simpler and CSRF-immune.) Examples: - Create a channel: `POST /api/v1/channels` — check the current `ErsatzTV/wwwroot/openapi/v1.json` (or `CreateChannelRequest.cs`) for the exact required field list before assuming these are complete, but as of this writing it requires (among plain fields) these enums: `PlayoutSource: "Generated"`, `PlayoutMode: "Continuous"`, `SongVideoMode: "Default"`, `TranscodeMode: "OnDemand"`, `IdleBehavior: "StopOnDisconnect"`. - Create a playout for an existing channel: `POST /api/v1/playouts` with `{"channelId": , "scheduleKind": "Block"}` (or `"Classic"`/`"Scripted"`/`"Sequential"` per `ChannelPlayoutSource`/schedule-kind enums — check `v1.json` for the current set). 5b. **Browser flow (since #295 session auth)**: the first `/app` load hits a **boot gate**. On a fresh config it shows **Setup** — claim the local admin (pick any username/password); that issues the session cookie and drops you into the app. On a subsequent run of the *same* config it shows **Login** instead. (To skip the browser claim in automation, set `Auth:LocalAdmin:Password` before launch — the env seed provisions the admin and disables the browser setup-claim.) The machine key still works for curl seeding regardless. Driving this with Playwright: fill the Setup/Login form before asserting any authed screen. 6. **Tear down**: kill the `dotnet ErsatzTV.dll` process **by the PID you captured at launch**, and confirm the port is freed (`lsof -i :8409` should return nothing) before starting another run — a stray process holding the port will make the next run's health check hang or fail confusingly. > ### ⚠️ Kill by PID. Never `pkill -f` a pattern. > > This machine is shared by parallel sessions, and **several of them run this same binary at > once**. `pkill -f "dotnet ErsatzTV.dll"` reaps every one of them, not just yours. > > ```bash > kill "$PID" # ✅ the PID e2e-local.sh printed > pkill -f "dotnet ErsatzTV.dll" # ❌ NEVER — reaps other sessions' servers > ``` > > **Filtering by port does not make a pattern kill safe** — `pkill -f` matches the *command line*, > not the port, so it hits every instance whatever port each one chose. Nor are the ports actually > separated: the CI `functional-e2e` step exports `ETV_UI_PORT=8409`, the **same** port local runs > use. (8410 is the `ersatztv-test` *container* on jazz — a different thing; the ui-E2E step from > #445 does use it.) Scope by PID, not by pattern and not by port. > > **If the port is already busy, that process is not yours — diagnose, don't reap.** Report it and > move to another port; `scripts/e2e-local.sh` pre-flights this for you and prints the foreign PID: > ```bash > lsof -ti :8409 # who is holding it > ETV_UI_PORT=8420 scripts/e2e-local.sh # your run, out of the way > ``` > > **Launching the DLL by hand? Set both ports.** `Program.cs` binds a second listener on > `ETV_STREAMING_PORT`, which defaults to **8409 regardless of `ETV_UI_PORT`** — so > `ETV_UI_PORT=8420 dotnet ErsatzTV.dll` still binds 8409 and dies against a foreign holder: > ```bash > ETV_UI_PORT=8420 ETV_STREAMING_PORT=8420 dotnet ErsatzTV.dll > ``` > `scripts/e2e-local.sh` defaults `ETV_STREAMING_PORT` to whatever port you gave it, so through the > script `ETV_UI_PORT=8420` alone is sufficient. > > Why this is a hard rule rather than a preference: killing another session's harness mid-run does > **not** fail loudly. It truncates that run's output into plausible-looking-but-wrong data — the > failure class that survives review. A near-miss of exactly this shape is recorded in > `docs/decisions/workflow-process.md` → `testing.e2e-cleanup-scope-by-pid` (ersatztv#586). > > **Writing a brief for a delegated agent? Put this constraint in it.** The #586 incident was a > delegation gap, not agent error: the brief specified a fresh config dir but said nothing about > process cleanup, so the agent invented a reasonable-looking pattern kill. An omitted rule is not > an unenforced rule — it is a rule replaced by whatever plausible default the agent reaches for. ## Playwright MCP screenshots If you're driving the browser via the Playwright MCP server for visual verification, screenshots land in **the MCP server process's own cwd** (this repo's root, not wherever you ran the dotnet process from) — expect stray `*.png` files at the repo root after a session; this is tolerated, not a bug to fix, but don't check them in. ## Script: `scripts/e2e-local.sh` A copy of this script is included in this doc's directory; it is intended to land at `scripts/e2e-local.sh` in the repo. It automates steps 2–4 above (build is assumed already done — run `dotnet build` / `npm run build` yourself first, since rebuilding on every invocation is slow and this script is meant to be re-run often during a debugging session). Usage: ```bash scripts/e2e-local.sh [CONFIG_DIR] ``` - `CONFIG_DIR` defaults to a fresh `mktemp -d` if omitted. Prefer a fresh dir per run for **state-bleed** reasons (above); the readiness probe itself handles a reused dir since #533. - ⚠️ **Never run two instances concurrently, even on different ports.** Each run `rm -rf`s and re-copies the *same* `ErsatzTV/bin//net10.0/wwwroot`, so the second run pulls the static files out from under a still-starting first instance. The symptom — a readiness timeout, or `/app` 404ing on the instance that *did* start — looks nothing like a shared-directory race. Run sequentially; the CI job's curl and UI-E2E steps do. - **`ETV_PIDFILE`** (optional): the server PID is written here the instant it forks, *before* the readiness wait. `scripts/e2e-ui.sh` sets it so it can still reap the server if it is killed mid-boot, before it has parsed `PID=` from stdout. Unset means unchanged behaviour. - Copies `ErsatzTV/wwwroot` → `ErsatzTV/bin//net10.0/wwwroot`. - Launches `dotnet ErsatzTV.dll` in the background with `ETV_CONFIG_FOLDER` set. - Waits (up to 120s) for readiness — either `Done migrating search index` (fresh config) or `Search index is already version` (reused config); see the step-4 table above. - **Pre-flights both bound ports** (`ETV_UI_PORT` and `ETV_STREAMING_PORT`): if something is already listening, it fails immediately naming the offending PID, rather than letting Kestrel fail its bind a moment later and surface as `process N exited before becoming ready` plus a log tail — a framing that reads like a broken build. It **reports, never reaps** — on this shared machine a foreign listener is most likely another session's harness mid-run. Re-run with `ETV_UI_PORT=`. - **Defaults `ETV_STREAMING_PORT` to `ETV_UI_PORT`** so that re-run actually works: the app binds a second listener that otherwise defaults to 8409 no matter what the UI port is. An explicit `ETV_STREAMING_PORT` still wins, and CI (which uses 8409) is unaffected. - Prints the PID and port, then **exits leaving the server running** — the caller owns that PID and is responsible for killing it when done (`kill "$PID"`, never a `pkill -f` pattern — see the teardown warning in step 6). - **Does not trap-and-kill on exit, by design** — unlike `scripts/e2e-ui.sh`, which owns its instance's whole lifecycle and kills the server from an `EXIT INT TERM` trap. The two scripts sit on opposite sides of that contract on purpose: this one is the *launcher* and hands a running instance to its caller (an `EXIT` trap here would kill the server the instant it returned, breaking every caller including `e2e-ui.sh` and the CI `functional-e2e` step); `e2e-ui.sh` is a *lifecycle owner* and traps. If you write a new harness that boots and then finishes on its own, follow `e2e-ui.sh`: capture the PID and trap. - **`ETV_BUILD_CONFIG`** selects which build output to launch (`Debug` default for local dev; the CI `functional-e2e` job sets `Release`). It must match the `dotnet build --configuration` you ran first — the script only copies `wwwroot` + launches; it does not build. ## Functional-E2E harness: `scripts/e2e-functional.sh` The ad-hoc curl scenarios sessions run against a live instance (redirect sweeps, auth flows, the scan status contract, `If-Match`/412) are codified into a single harness so they run identically by hand and in CI (the advisory `functional-e2e` job — `docs/ci-cd.md`). It does **not** boot the app; pair it with `scripts/e2e-local.sh` above (or point it at any running instance): ```bash CFG=$(mktemp -d) eval "$(scripts/e2e-local.sh "$CFG" | sed -n 's/^\(PID\|PORT\)=/\1=/p')" # launch, capture PID/PORT scripts/e2e-functional.sh "http://localhost:${PORT}" "$CFG" # assert; exit 1 on any failure kill "$PID" ``` `CONFIG_DIR` (arg 2) is **required** — the harness reads the instance's machine key from `$CONFIG_DIR/api.key` and sends `X-Api-Key` on every `/api` call (the surface is fail-closed). It runs each assertion even after a failure and prints a pass/fail summary, exiting non-zero if any failed. What it covers. Most assertions are curl-only (no seeded media / ffmpeg / browser); the **lock-contention** section (added by ersatztv#363, extended by #444) is the exception — it seeds DB rows + media. - **Legacy→SPA redirects**: a sweep of representative `LegacyUiRedirects` routes 302→`/app/*`, plus the `/api` + `/artwork` never-redirect exemption (asserted as "did not 302 to `/app`", since those 4xx from their own handlers — `/api/*` 404s, `/artwork/*` 400s). - **Library-scan status contract**: create an empty local library → scan `202`, unknown-library scan `404`, `scan-status` `200`. - **Optimistic concurrency**: create a collection + a `rerun-collections` targeting it → `GET` emits an `ETag` → `PUT` with a stale `If-Match` `412`, current `200`, malformed `400`. - **Lock contention (409)** — the three `IEntityLocker` 409s, made **deterministic** by firing the racing request only once the lock is *provably* held (never a sleep-and-hope). All seed rows the API can't create straight into the running instance's DB via python3's stdlib `sqlite3` (whose busy-timeout retry serializes behind the app's writer): - **library-scan "already scanning"**: seed ~60 tiny ffmpeg clips into the built-in Shows library (Id=2) so the scanner subprocess runs a few seconds → `POST .../scan` `202` (local scans always ForceScan, so no `deep` needed) → poll `GET /libraries/scan-status` until library 2 shows active (that window is a strict subset of the scan lock's held window — `ScannerProxyService.StartScan` fires *after* `LockLibrary`, `EndScan` *before* `UnlockLibrary`) → a second `POST .../scan` is a `409`, deterministic bar a tiny residual TOCTOU gap the multi-second scan covers. Self-skips (advisory) if ffmpeg is absent. - **external-collections "already scanning"**: seed a Jellyfin media-source row pointing at a non-routable address (so the background sync hangs and the per-family lock stays held). The lock is taken synchronously *before* the `202`, so the `202` proves it held → `collections-scan-status` shows the `jellyfin` family → a second `POST .../scan-collections` is `409`; unknown source `404`. Needs neither ffmpeg nor the scanner. - **playout-build "build in progress"** (#215/#444, the hardest): a build is enqueued onto the single-consumer `WorkerService` channel and the trigger returns *before* `BuildPlayoutHandler` dequeues and `LockPlayout`s (released in its `finally`), so an accepted trigger does **not** prove the lock is held — poll `GET /playouts/{id}` until `isLocked:true`, then fire. Seed a few short ffmpeg episodes → a Collection → a **Classic Flood** schedule → a Classic playout, and crank `PlayoutDaysToBuild` (config `playout.days_to_build`) so a single build is wide enough to observe: 5 days ≈ 43k playout items ≈ ~1s locally, wider on slower CI (sized by measurement — item count is `window / item-duration`, so the flood over a handful of short clips scales with the day count; going *too* wide is counter-productive — a 777k-item build saturates the single worker with its post-build gap/overlap jobs). While provably locked: `PUT /playouts/{id}` → 409, `POST .../playout/reset` → 409, and the `GET /playouts` list projection shows `isLocked:true`; after the build the lock clears (`isLocked:false`) and the same `PUT` now `200`s (proving the 409 is lock-specific). Restores the config default afterwards. Self-skips (advisory) without ffmpeg, or if the build is never observed locked (never asserts a race it can't prove it won). - **Auth/CSRF/security-stamp** (mirrors the #295 manual set; runs **last** — setup-claim is one-shot and logout revokes the session): fresh `auth/config` `setupRequired:true` → read-gate 401-without-key / 200-with-key → setup-claim 200 → re-claim 409 → session mutation 403-without-`X-CSRF` → login 401-then-200 → logout 403-without-CSRF / 204-with → post-logout the same cookie is 401 (rotated security stamp). The playout-build lock 409 + `isLocked` projection (#215) landed in #444 — see the lock-contention bullet above. The genuinely **UI-interactive Playwright** flows landed in #445 — see the next section. When you extend the harness, add the observed contract here and keep the assertions deterministic (probe the real instance first — the initial cut caught that `/artwork/*` 400s where a guess said 404; the #363 cut measured the real scan window before sizing the seed). ## UI-E2E harness: `scripts/e2e-ui.sh` + `web/e2e/*.spec.ts` The handful of contracts that **cannot** be expressed as curl calls, driven by **headless Playwright** (never claude-in-chrome, per project convention — ersatztv#445). Unlike `e2e-functional.sh`, this script owns the whole lifecycle: it boots a fresh instance, runs the specs, and always kills the server (trap on `EXIT`/`INT`/`TERM`), exiting with Playwright's status: ```bash scripts/e2e-ui.sh # fresh config dir, all specs scripts/e2e-ui.sh "$(mktemp -d)" --headed --debug # extra args pass through to `playwright test` ``` **A fresh config dir is required, not merely preferred** — the first spec asserts the one-shot Setup (first-run admin claim) gate, which only exists while the server has no local admin. Pointing it at a reused dir fails that spec by design. **What it covers, and the rule for extending it.** Only assert here what curl *structurally cannot* — the curl harness above already covers the auth HTTP contracts, so re-asserting them through a browser buys nothing but flake surface. The four things that qualify: | Contract | Why curl can't reach it | | --- | --- | | Setup card's confirm-password gate | Pure React state; makes no request, so there is no HTTP contract | | `AuthGate` states (Setup vs Login vs app) | Assertion is about what *renders*, not a status code | | Session cookie on the SPA's own `/api` XHRs | curl proves the cookie works *for curl*, not that the app sends it | | Sign-out via the `UserMenu` | A DOM interaction chain, not a single endpoint | **Determinism rules** (#445 asked for deterministic flows, so these are deliberate): - `retries: 0`, in CI too — a retry would let a genuinely flaky flow merge looking green. - `serial` + `workers: 1`: server state is shared and partly one-shot (the setup-claim). - Each `test` still gets its own browser context (own cookie jar) — that is what gives the login specs a genuinely signed-out browser without a logout dance. **Anything needing a session held across steps must stay inside one `test`.** - Address form fields by their unique **placeholders**, not labels: the shared `` wraps its `` in a `