timothy
d8c0b3e752
Adds the last deferred #299/#363 follow-up: the flows that CANNOT be expressed
as curl calls. Scope rule (the durable part) — assert only what the curl
harness structurally cannot reach:
1. client-side form validation (the Setup confirm-password gate is pure React
state and makes no request, so there is no HTTP contract to assert)
2. AuthGate's RENDERED states (Setup vs Login vs app)
3. the session cookie authenticating the SPA's OWN /api XHRs — curl proves the
cookie works for curl, not that the app sends it
4. sign-out through the UserMenu back to the login gate
New: web/e2e/boot-gate.spec.ts, web/playwright.config.ts, scripts/e2e-ui.sh
(owns the whole lifecycle: fresh config dir -> boot -> specs -> always kill).
Runs as a second step of the EXISTING advisory `functional-e2e` job rather than
a new job: the dominant cost there is `npm ci` + the Release build, both already
done, so this adds ~5s instead of duplicating a heavy job. It boots its own
fresh instance on port 8410 because the first spec asserts the one-shot Setup
gate that the curl step has already claimed on its config dir.
Determinism (the issue asked for it explicitly): `serial`, `workers: 1`,
`retries: 0` even in CI — a retry would let a flaky flow merge looking green.
Measured 5 consecutive clean runs, ~2s each.
Pins all five `container:` jobs to the toolchain image built by the preceding
commit, which bakes `chromium-headless-shell`.
Non-obvious coupling fixed: vitest's default include glob would have collected
web/e2e/*.spec.ts and run it under jsdom. Excluded `e2e/**` by spreading
`configDefaults.exclude` rather than narrowing `include` to `src/**`, because
web/scripts/ holds a real vitest test an src-only include would silently stop
running.
`RebuildSearchIndexHandler` logs one of two mutually-exclusive lines just before
`SystemStartup.SearchIndexIsReady()`:
fresh config -> "Done migrating search index in {Duration}"
reused config -> "Search index is already version {Version}"
The probe watched only the first, so a reused dir waited out the full 120s
timeout and then killed a perfectly healthy server. Widened to a `grep -Eq`
alternation; the handler's if/else is exhaustive, so the pair covers every path
to readiness.
Verified with a negative control: on a reused dir the server is ready in 2s via
the "already version" line, and the OLD probe string is genuinely ABSENT from
that run's log — so the old code would have hung, i.e. the fix is load-bearing
rather than incidentally passing.
The "prefer a fresh config dir" guidance stays: that guards state bleed, which
is a separate concern from the probe hanging.
- `wait "$PID"` in the cleanup trap was a NO-OP: the server is a grandchild
(launched in e2e-local.sh's subshell, which then exits), so `wait` fails
instantly and was swallowed by `|| true` — cleanup did not actually ensure the
port was released, exactly what its comment claimed. Replaced with a bounded
`kill -0` poll, then SIGKILL.
- Added a port pre-flight check: previously an occupied port surfaced as a 120s
readiness timeout that reads like a broken build. Now fails in 0s naming the
PIDs, and warns against blanket-killing `dotnet ErsatzTV.dll` (that reaps
other sessions' servers).
- UI-E2E: 5x clean (3 specs, ~2s); back-to-back runs pass with no manual cleanup
- curl harness unaffected by the boot-script change: 45/45 PASS
- web: 983 tests / 105 files green; typecheck + lint clean
- vitest collection verified: excludes web/e2e, still collects web/scripts
- Dockerfile sequence + browser launch validated verbatim in a container on the
real amd64 base before committing; chromium launches as root with NO sandbox
opt-out needed
- decisions validator green; catalog regenerated
- docs/decisions.md TOC repaired: it had drifted to 69 of 97 records and held a
dangling anchor to the #72 record that #415 superseded into archive/.
Regenerated with a generator validated against the 68 existing anchors (0
mismatches) -> 97/97, no dangling, no duplicates.
Docs: docs/e2e-local.md (new "UI-E2E harness" section), docs/ci-cd.md (toolchain
image + UI-E2E step), docs/testing.md, docs/README.md, docs/decisions.md
(new `ci.ui-e2e-harness` record; `ci.functional-e2e-harness` amended — its Rule
said "curl-only", now accurate).
Refs #445 #533
117 lines
7.7 KiB
Markdown
117 lines
7.7 KiB
Markdown
# Testing map
|
|
|
|
Purpose: authoritative map of what each test project/suite covers and how to run it. Read this
|
|
before adding tests, not just `docs/contributing.md` §8 (which now just points here).
|
|
|
|
## Test projects
|
|
|
|
| Project | Covers | Notes |
|
|
|---|---|---|
|
|
| `ErsatzTV.Tests` | API controllers + MediatR handlers | In-memory SQLite fixture: a shared `SqliteConnection("Data Source=:memory:;Foreign Keys=False")` kept open + `EnsureCreatedAsync()` (**not** full migration replay) + `PRAGMA foreign_keys=OFF`, then seed; a tiny `IDbContextFactory` wraps `new TvContext(...)`. 828 tests currently. |
|
|
| `ErsatzTV.Core.Tests` | Domain logic, scheduling, IPTV/XMLTV generation | References `ErsatzTV.Application` directly — there is no separate `Application.Tests` project. 543 tests + 1 skipped under `TZ=UTC` (the Block playout golden additionally skips under a non-UTC `TZ`; see Golden-file nets). |
|
|
| `ErsatzTV.Scanner.Tests` | Library scanning: scan handlers, folder scanners, NFO readers | Handler tests substitute the folder scanners + `ILibraryRepository` and assert the resulting repository writes (e.g. `ScanLocalLibraryHandlerTests` pins which `LastScan` levels a scan records — ersatztv#264). Fakes/`Testably` back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains `Core/FFmpeg/TranscodingTests` — `[Explicit]` + `[Combinatorial]`, so it never runs in CI or a plain `dotnet test` (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline. |
|
|
| `ErsatzTV.Architecture.Tests` | Layering rules via NetArchTest.eNhancedEdition | Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See `docs/contributing.md` §1. |
|
|
| `ErsatzTV.FFmpeg.Tests` | FFmpeg command construction | Build a pipeline, assert the exact rendered arg string (`PipelineBuilderBaseTests.cs`). |
|
|
| `web/` (vitest) | React SPA unit tests | 983 tests across 105 files; run alongside typecheck + build (see below). Collects `src/**` *and* `web/scripts/**`, but deliberately **excludes** `web/e2e/**` (the Playwright specs — vitest's default `**/*.spec.*` glob would otherwise run them under jsdom). |
|
|
| `web/e2e/` (Playwright) | UI-interactive E2E flows against a **live** instance | Not a unit suite and **not** part of `npm test` — needs a running server, so it runs via `scripts/e2e-ui.sh` (boots its own fresh instance) and in CI as a step of the `functional-e2e` job. Headless Chromium, `serial`, `retries: 0`. Scope rule: assert only what the curl harness structurally cannot. See `docs/e2e-local.md` → "UI-E2E harness". |
|
|
|
|
## Golden-file nets
|
|
|
|
Three golden-file suites guard the highest-value, most-subtle output:
|
|
|
|
- **M3U**: `ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs` (ersatztv#11) — env var `ETV_UPDATE_GOLDENS`
|
|
- **XMLTV**: `ChannelGuideGoldenTests` (ersatztv#28) — env var `ETV_UPDATE_GOLDENS`
|
|
- **Playout build**: `ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs` (ersatztv#163)
|
|
— env var **`ETV_UPDATE_PLAYOUT_GOLDENS`** (deliberately separate from `ETV_UPDATE_GOLDENS` so
|
|
regenerating one net can't silently rewrite the other). Snapshots the `PlayoutItem`s each builder
|
|
produces over a pinned build window. Covers the **Classic** (`PlaybackOrder.Chronological`),
|
|
**Block**, and **Sequential (YAML)** builders. The **Scripted** *end-to-end pipeline* is excluded from this
|
|
net — `ScriptedPlayoutBuilder` runs a user-authored external process that drives the engine over HTTP, which
|
|
the in-memory harness can't pin; that integration harness is tracked in ersatztv#563. The scheduling
|
|
*behavior* those scripts drive, though, lives in the in-process `SchedulingEngine` (the HTTP controller is a
|
|
1:1 pass-through) and IS directly testable — `SchedulingEngineTests` news it up with substitutes, and
|
|
`ContentEnumeratorBuilderTests` (ersatztv#395) is the direct regression net over the enumerator-construction
|
|
helper the Scripted and Sequential/YAML engines now share (decision:
|
|
`testing.scripted-playout-golden-deferred`). The **Sequential** case
|
|
(`Sequential_yaml`, ersatztv#381)
|
|
builds from a committed YAML fixture (`Goldens/Fixtures/sequential-schedule.yml`) instead of a
|
|
`ProgramSchedule`; it is TZ-independent (the `count`/`all`/`duration` handlers do UTC-only arithmetic —
|
|
it passes, not skips, under a non-UTC `TZ`) so needs no `Assume` guard. A third case,
|
|
`Classic_clock_padded` (ersatztv#77), locks clock-boundary padding: a `FillerMode.Pad` +
|
|
`PadToNearestMinute=15` PostRoll preset snaps content to `:15`, and the test both goldens the output and
|
|
asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is
|
|
`ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs` (guide programmes stop on the padded
|
|
boundary). The build reads
|
|
no wall clock — time enters only via the caller-supplied `start` — so a pinned `start` is fully
|
|
deterministic. Snapshots the raw `PlayoutItem.Start`/`Finish` (UTC), **not** the `*Offset` properties
|
|
(those call `.ToLocalTime()` and would make the golden machine-TZ dependent). The **Block** case is
|
|
TZ-sensitive by construction (`BlockPlayoutBuilder` maps template times via `TimeZoneInfo.Local`), so
|
|
it is guarded with `Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero)`: it runs under `TZ=UTC`
|
|
(CI) and reports **inconclusive** (a graceful skip, not a failure) under any other TZ. A real TZ seam
|
|
for the block builder is ersatztv#380's scope.
|
|
|
|
All three locate their golden files via `[CallerFilePath]`. A missing golden is a hard fail, not a
|
|
skip. Regenerate via `ETV_UPDATE_GOLDENS=1 dotnet test ...` (M3U/XMLTV) or
|
|
`ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ...` (playout build).
|
|
|
|
**Never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS` in CI or from an agent.** A golden
|
|
diff during normal test runs means the code changed the output — regenerating to make the diff go
|
|
away hides the change instead of surfacing it. Only a human who has confirmed the change is
|
|
intentional should regenerate.
|
|
|
|
## Timezone independence
|
|
|
|
The suite is timezone-independent (ersatztv#24). When constructing test `PlayoutItem`s, always
|
|
set a real `Start` (e.g. `startState.CurrentTime.UtcDateTime`) — never rely on the default
|
|
`DateTime.MinValue`, which underflows `DateTimeOffset.MinValue` once a non-UTC local offset is
|
|
applied (`StartOffset` calls `ToLocalTime()`). CI runs in UTC; local runs may not.
|
|
|
|
## Running tests
|
|
|
|
Full .NET gate:
|
|
|
|
```bash
|
|
dotnet build ErsatzTV.sln
|
|
TZ=UTC dotnet test
|
|
```
|
|
|
|
CI adds `--blame-hang-timeout 2m` to catch hangs.
|
|
|
|
Fast subsets:
|
|
|
|
```bash
|
|
# single project
|
|
dotnet test ErsatzTV.Tests
|
|
|
|
# filtered
|
|
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests
|
|
```
|
|
|
|
Web (`web/`):
|
|
|
|
```bash
|
|
npm test # vitest (excludes web/e2e — those need a live server)
|
|
npm run typecheck # tsc -b --pretty false
|
|
npm run lint # eslint .
|
|
npm run build # tsc -b && vite build
|
|
```
|
|
|
|
UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):
|
|
|
|
```bash
|
|
scripts/e2e-ui.sh # from the repo root, NOT web/
|
|
```
|
|
|
|
## Per-PR verification gate
|
|
|
|
Before opening a PR: build the solution, run `ErsatzTV.Tests` + `ErsatzTV.Core.Tests` (plus
|
|
`ErsatzTV.Scanner.Tests`, `ErsatzTV.Architecture.Tests` and `ErsatzTV.FFmpeg.Tests` if touched),
|
|
and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or
|
|
an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.
|
|
|
|
## See also
|
|
|
|
- `docs/ci-cd.md` — CI pipeline (test → migrations → build), versioning, dependency management.
|
|
- `docs/contributing.md` §1 — layering rules enforced by `ErsatzTV.Architecture.Tests`.
|
|
- `docs/contributing.md` §8 — short pointer back to this doc.
|