Files

T

timothy d8c0b3e752 feat(445,533): headless Playwright UI-E2E flows + fix e2e-local readiness probe [decisions-edit]

Adds the last deferred #299/#363 follow-up: the flows that CANNOT be expressed
as curl calls. Scope rule (the durable part) — assert only what the curl
harness structurally cannot reach:

  1. client-side form validation (the Setup confirm-password gate is pure React
     state and makes no request, so there is no HTTP contract to assert)
  2. AuthGate's RENDERED states (Setup vs Login vs app)
  3. the session cookie authenticating the SPA's OWN /api XHRs — curl proves the
     cookie works for curl, not that the app sends it
  4. sign-out through the UserMenu back to the login gate

New: web/e2e/boot-gate.spec.ts, web/playwright.config.ts, scripts/e2e-ui.sh
(owns the whole lifecycle: fresh config dir -> boot -> specs -> always kill).

Runs as a second step of the EXISTING advisory `functional-e2e` job rather than
a new job: the dominant cost there is `npm ci` + the Release build, both already
done, so this adds ~5s instead of duplicating a heavy job. It boots its own
fresh instance on port 8410 because the first spec asserts the one-shot Setup
gate that the curl step has already claimed on its config dir.

Determinism (the issue asked for it explicitly): `serial`, `workers: 1`,
`retries: 0` even in CI — a retry would let a flaky flow merge looking green.
Measured 5 consecutive clean runs, ~2s each.

Pins all five `container:` jobs to the toolchain image built by the preceding
commit, which bakes `chromium-headless-shell`.

Non-obvious coupling fixed: vitest's default include glob would have collected
web/e2e/*.spec.ts and run it under jsdom. Excluded `e2e/**` by spreading
`configDefaults.exclude` rather than narrowing `include` to `src/**`, because
web/scripts/ holds a real vitest test an src-only include would silently stop
running.

`RebuildSearchIndexHandler` logs one of two mutually-exclusive lines just before
`SystemStartup.SearchIndexIsReady()`:

  fresh config  -> "Done migrating search index in {Duration}"
  reused config -> "Search index is already version {Version}"

The probe watched only the first, so a reused dir waited out the full 120s
timeout and then killed a perfectly healthy server. Widened to a `grep -Eq`
alternation; the handler's if/else is exhaustive, so the pair covers every path
to readiness.

Verified with a negative control: on a reused dir the server is ready in 2s via
the "already version" line, and the OLD probe string is genuinely ABSENT from
that run's log — so the old code would have hung, i.e. the fix is load-bearing
rather than incidentally passing.

The "prefer a fresh config dir" guidance stays: that guards state bleed, which
is a separate concern from the probe hanging.

- `wait "$PID"` in the cleanup trap was a NO-OP: the server is a grandchild
  (launched in e2e-local.sh's subshell, which then exits), so `wait` fails
  instantly and was swallowed by `|| true` — cleanup did not actually ensure the
  port was released, exactly what its comment claimed. Replaced with a bounded
  `kill -0` poll, then SIGKILL.
- Added a port pre-flight check: previously an occupied port surfaced as a 120s
  readiness timeout that reads like a broken build. Now fails in 0s naming the
  PIDs, and warns against blanket-killing `dotnet ErsatzTV.dll` (that reaps
  other sessions' servers).

- UI-E2E: 5x clean (3 specs, ~2s); back-to-back runs pass with no manual cleanup
- curl harness unaffected by the boot-script change: 45/45 PASS
- web: 983 tests / 105 files green; typecheck + lint clean
- vitest collection verified: excludes web/e2e, still collects web/scripts
- Dockerfile sequence + browser launch validated verbatim in a container on the
  real amd64 base before committing; chromium launches as root with NO sandbox
  opt-out needed
- decisions validator green; catalog regenerated
- docs/decisions.md TOC repaired: it had drifted to 69 of 97 records and held a
  dangling anchor to the #72 record that #415 superseded into archive/.
  Regenerated with a generator validated against the 68 existing anchors (0
  mismatches) -> 97/97, no dangling, no duplicates.

Docs: docs/e2e-local.md (new "UI-E2E harness" section), docs/ci-cd.md (toolchain
image + UI-E2E step), docs/testing.md, docs/README.md, docs/decisions.md
(new `ci.ui-e2e-harness` record; `ci.functional-e2e-harness` amended — its Rule
said "curl-only", now accurate).

Refs #445 #533

2026-07-25 14:03:59 +02:00

7.7 KiB

Raw Blame History

Testing map

Purpose: authoritative map of what each test project/suite covers and how to run it. Read this before adding tests, not just docs/contributing.md §8 (which now just points here).

Test projects

Project	Covers	Notes
`ErsatzTV.Tests`	API controllers + MediatR handlers	In-memory SQLite fixture: a shared `SqliteConnection("Data Source=:memory:;Foreign Keys=False")` kept open + `EnsureCreatedAsync()` (not full migration replay) + `PRAGMA foreign_keys=OFF`, then seed; a tiny `IDbContextFactory` wraps `new TvContext(...)`. 828 tests currently.
`ErsatzTV.Core.Tests`	Domain logic, scheduling, IPTV/XMLTV generation	References `ErsatzTV.Application` directly — there is no separate `Application.Tests` project. 543 tests + 1 skipped under `TZ=UTC` (the Block playout golden additionally skips under a non-UTC `TZ`; see Golden-file nets).
`ErsatzTV.Scanner.Tests`	Library scanning: scan handlers, folder scanners, NFO readers	Handler tests substitute the folder scanners + `ILibraryRepository` and assert the resulting repository writes (e.g. `ScanLocalLibraryHandlerTests` pins which `LastScan` levels a scan records — ersatztv#264). Fakes/`Testably` back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains `Core/FFmpeg/TranscodingTests` — `[Explicit]` + `[Combinatorial]`, so it never runs in CI or a plain `dotnet test` (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline.
`ErsatzTV.Architecture.Tests`	Layering rules via NetArchTest.eNhancedEdition	Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See `docs/contributing.md` §1.
`ErsatzTV.FFmpeg.Tests`	FFmpeg command construction	Build a pipeline, assert the exact rendered arg string (`PipelineBuilderBaseTests.cs`).
`web/` (vitest)	React SPA unit tests	983 tests across 105 files; run alongside typecheck + build (see below). Collects `src/*` and* `web/scripts/`, but deliberately excludes `web/e2e/` (the Playwright specs — vitest's default `*/.spec.*` glob would otherwise run them under jsdom).
`web/e2e/` (Playwright)	UI-interactive E2E flows against a live instance	Not a unit suite and not part of `npm test` — needs a running server, so it runs via `scripts/e2e-ui.sh` (boots its own fresh instance) and in CI as a step of the `functional-e2e` job. Headless Chromium, `serial`, `retries: 0`. Scope rule: assert only what the curl harness structurally cannot. See `docs/e2e-local.md` → "UI-E2E harness".

Golden-file nets

Three golden-file suites guard the highest-value, most-subtle output:

M3U: ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs (ersatztv#11) — env var ETV_UPDATE_GOLDENS
XMLTV: ChannelGuideGoldenTests (ersatztv#28) — env var ETV_UPDATE_GOLDENS
Playout build: ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs (ersatztv#163) — env var ETV_UPDATE_PLAYOUT_GOLDENS (deliberately separate from ETV_UPDATE_GOLDENS so regenerating one net can't silently rewrite the other). Snapshots the PlayoutItems each builder produces over a pinned build window. Covers the Classic (PlaybackOrder.Chronological), Block, and Sequential (YAML) builders. The Scripted end-to-end pipeline is excluded from this net — ScriptedPlayoutBuilder runs a user-authored external process that drives the engine over HTTP, which the in-memory harness can't pin; that integration harness is tracked in ersatztv#563. The scheduling behavior those scripts drive, though, lives in the in-process SchedulingEngine (the HTTP controller is a 1:1 pass-through) and IS directly testable — SchedulingEngineTests news it up with substitutes, and ContentEnumeratorBuilderTests (ersatztv#395) is the direct regression net over the enumerator-construction helper the Scripted and Sequential/YAML engines now share (decision: testing.scripted-playout-golden-deferred). The Sequential case (Sequential_yaml, ersatztv#381) builds from a committed YAML fixture (Goldens/Fixtures/sequential-schedule.yml) instead of a ProgramSchedule; it is TZ-independent (the count/all/duration handlers do UTC-only arithmetic — it passes, not skips, under a non-UTC TZ) so needs no Assume guard. A third case, Classic_clock_padded (ersatztv#77), locks clock-boundary padding: a FillerMode.Pad + PadToNearestMinute=15 PostRoll preset snaps content to :15, and the test both goldens the output and asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs (guide programmes stop on the padded boundary). The build reads no wall clock — time enters only via the caller-supplied start — so a pinned start is fully deterministic. Snapshots the raw PlayoutItem.Start/Finish (UTC), not the *Offset properties (those call .ToLocalTime() and would make the golden machine-TZ dependent). The Block case is TZ-sensitive by construction (BlockPlayoutBuilder maps template times via TimeZoneInfo.Local), so it is guarded with Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero): it runs under TZ=UTC (CI) and reports inconclusive (a graceful skip, not a failure) under any other TZ. A real TZ seam for the block builder is ersatztv#380's scope.

All three locate their golden files via [CallerFilePath]. A missing golden is a hard fail, not a skip. Regenerate via ETV_UPDATE_GOLDENS=1 dotnet test ... (M3U/XMLTV) or ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ... (playout build).

Never set ETV_UPDATE_GOLDENS / ETV_UPDATE_PLAYOUT_GOLDENS in CI or from an agent. A golden diff during normal test runs means the code changed the output — regenerating to make the diff go away hides the change instead of surfacing it. Only a human who has confirmed the change is intentional should regenerate.

Timezone independence

The suite is timezone-independent (ersatztv#24). When constructing test PlayoutItems, always set a real Start (e.g. startState.CurrentTime.UtcDateTime) — never rely on the default DateTime.MinValue, which underflows DateTimeOffset.MinValue once a non-UTC local offset is applied (StartOffset calls ToLocalTime()). CI runs in UTC; local runs may not.

Running tests

Full .NET gate:

dotnet build ErsatzTV.sln
TZ=UTC dotnet test

CI adds --blame-hang-timeout 2m to catch hangs.

Fast subsets:

# single project
dotnet test ErsatzTV.Tests

# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests

Web (web/):

npm test           # vitest (excludes web/e2e — those need a live server)
npm run typecheck  # tsc -b --pretty false
npm run lint       # eslint .
npm run build      # tsc -b && vite build

UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):

scripts/e2e-ui.sh              # from the repo root, NOT web/

Per-PR verification gate

Before opening a PR: build the solution, run ErsatzTV.Tests + ErsatzTV.Core.Tests (plus ErsatzTV.Scanner.Tests, ErsatzTV.Architecture.Tests and ErsatzTV.FFmpeg.Tests if touched), and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.

7.7 KiB Raw Blame History