Files
ersatztv/docs/testing.md
T
timothy d8c29fd11e fix(819): mutation-proof the fix that closed the blind spot
Round 2's cold review found the previous commit had reproduced, one layer up,
the exact defect it was fixing: `expectedTrackedSources` — the function that
closes the dot-directory / case-divergence hole — had no clause proof. Measured:
replacing its body with `return []` left all 1249 tests green while the hole it
exists to catch went undetected, and deleting the `absentFromDisk` subtraction
(the whole point of the plugin's second field) was green too.

Three proofs added, each MEASURED red against the mutation it names:
- a direct fixture test pinning all four terms of `expectedTrackedSources`
  (`return []` and dropping the subtraction now each redden it);
- an anti-vacuity floor INSIDE the shortfall test, so an empty expected set
  cannot pass the comparison it feeds;
- a table pinning `isInScopeSourcePath`. That one closes a hole the review
  measured separately: the walk and the index side applied the SAME scope
  filters, so scope cancelled out of the equality between them — excluding
  `src/components/` dropped 13 real files with the whole suite green. The two
  copies of that predicate are now ONE named function, and the table is what
  reddens a change to it, since no assertion comparing the two sides can.

Also corrected, all found by the same review:
- The comment justifying the `absentFromDisk` subtraction cited
  `scripts/tests/tracked_files.py` as stating the rule it follows. That module
  states the OPPOSITE and enforces it with `assert absolute.is_file()`. It is
  now recorded as a deliberate DEVIATION with its reason (that population is a
  handful of hooks where a missing file is remarkable; this one is ~260 SPA
  sources a developer edits continuously), rather than as an instance of it.
- `absentFromDisk` is named for what it measures — no READABLE file — not for
  its common cause. `existsSync` follows links, so a tracked BROKEN SYMLINK is a
  committed, permanent state landing in that set, which "deleted in the working
  tree" licensed subtracting without saying so.
- The Dockerfile comment claimed the `needs: [test, …]` edge alone guarantees no
  image ships from an untested tree. `Test SPA` is also gated on `docs_only` and
  the #420 revalidate skip, and `build` is not gated on `revalidate`; the
  property survives via #420's byte-identical-tree argument, which the comment
  now names instead of implying the wrong mechanism.
- `docs/testing.md`'s web row was falsified by this PR: `web/vite-plugins/**` is
  a third collection root, and `npm test` now hard-requires a git checkout.
- Two prose slips: guard-inventory counted "the first of those" onto the wrong
  file, and both `docs/spa-conventions.md` and the Dockerfile said "guards"
  plural where exactly one derives a population.

Rebased onto 4b3f6f6c9 (#872), which rewrote `docs/guard-inventory.md`.

refs #819
2026-08-29 05:12:23 +02:00

125 lines
9.0 KiB
Markdown

# Testing map
Purpose: authoritative map of what each test project/suite covers and how to run it. Read this
before adding tests, not just `docs/contributing.md` §8 (which now just points here).
## Test projects
> **Font dependency (ersatztv#732).** `Infrastructure/Graphics/TextElementBackgroundBoxTests` is the
> only suite that rasterises text, so it needs at least one system font to lay anything out. The CI
> image installs none explicitly — fonts arrive via `playwright install --with-deps` in
> `docker/ci/Dockerfile` (351 present in the pinned image, measured 2026-08-26). This is a real
> dependency, declared here so a future slimming of that install produces a known cause rather than a
> mystery red. `Baseline_Renders_A_Non_Empty_Bitmap` exists to make that failure loud: without it a
> fontless host would render a 0x0 bitmap and every relative geometry assertion would pass vacuously.
| Project | Covers | Notes |
|---|---|---|
| `ErsatzTV.Tests` | API controllers + MediatR handlers, plus the SkiaSharp text-overlay rasteriser (`Infrastructure/Graphics/`) | In-memory SQLite fixture: a shared `SqliteConnection("Data Source=:memory:;Foreign Keys=False")` kept open + `EnsureCreatedAsync()` (**not** full migration replay) + `PRAGMA foreign_keys=OFF`, then seed; a tiny `IDbContextFactory` wraps `new TvContext(...)`. ~1,870 tests (approximate on purpose — an exact count goes stale on every PR that adds one; the previous hardcoded 828 was off by over a thousand). |
| `ErsatzTV.Core.Tests` | Domain logic, scheduling, IPTV/XMLTV generation | References `ErsatzTV.Application` directly — there is no separate `Application.Tests` project. ~650 tests + 1 skipped under `TZ=UTC` (the Block playout golden additionally skips under a non-UTC `TZ`; see Golden-file nets). |
| `ErsatzTV.Scanner.Tests` | Library scanning: scan handlers, folder scanners, NFO readers | Handler tests substitute the folder scanners + `ILibraryRepository` and assert the resulting repository writes (e.g. `ScanLocalLibraryHandlerTests` pins which `LastScan` levels a scan records — ersatztv#264). Fakes/`Testably` back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains `Core/FFmpeg/TranscodingTests` — `[Explicit]` + `[Combinatorial]`, so it never runs in CI or a plain `dotnet test` (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline. |
| `ErsatzTV.Architecture.Tests` | Layering rules via NetArchTest.eNhancedEdition | Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See `docs/contributing.md` §1. |
| `ErsatzTV.FFmpeg.Tests` | FFmpeg command construction | Build a pipeline, assert the exact rendered arg string (`PipelineBuilderBaseTests.cs`). |
| `web/` (vitest) | React SPA unit tests | Run alongside typecheck + build (see below). Collects `src/**`, `web/scripts/**` *and* `web/vite-plugins/**`, but deliberately **excludes** `web/e2e/**` (the Playwright specs — vitest's default `**/*.spec.*` glob would otherwise run them under jsdom). **Requires a git checkout and a `git` binary** since ersatztv#819: `web/src/api/pageSizeCallSites.guard.test.ts` derives its file population from `git ls-files` (via `web/vite-plugins/trackedSourceFiles.ts`) rather than a directory walk, and refuses rather than falling back, so the suite cannot run against an exported tree with no index. `docker/Dockerfile` no longer runs it for exactly that reason. |
| `web/e2e/` (Playwright) | UI-interactive E2E flows against a **live** instance | Not a unit suite and **not** part of `npm test` — needs a running server, so it runs via `scripts/e2e-ui.sh` (boots its own fresh instance) and in CI as a step of the `functional-e2e` job. Headless Chromium, `serial`, `retries: 0`. Scope rule: assert only what the curl harness structurally cannot. See `docs/e2e-local.md` → "UI-E2E harness". |
## Golden-file nets
Three golden-file suites guard the highest-value, most-subtle output:
- **M3U**: `ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs` (ersatztv#11) — env var `ETV_UPDATE_GOLDENS`
- **XMLTV**: `ChannelGuideGoldenTests` (ersatztv#28) — env var `ETV_UPDATE_GOLDENS`
- **Playout build**: `ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs` (ersatztv#163)
— env var **`ETV_UPDATE_PLAYOUT_GOLDENS`** (deliberately separate from `ETV_UPDATE_GOLDENS` so
regenerating one net can't silently rewrite the other). Snapshots the `PlayoutItem`s each builder
produces over a pinned build window. Covers the **Classic** (`PlaybackOrder.Chronological`),
**Block**, and **Sequential (YAML)** builders. The **Scripted** *end-to-end pipeline* is excluded from this
net — `ScriptedPlayoutBuilder` runs a user-authored external process that drives the engine over HTTP, which
the in-memory harness can't pin; that integration harness is tracked in ersatztv#563. The scheduling
*behavior* those scripts drive, though, lives in the in-process `SchedulingEngine` (the HTTP controller is a
1:1 pass-through) and IS directly testable — `SchedulingEngineTests` news it up with substitutes, and
`ContentEnumeratorBuilderTests` (ersatztv#395) is the direct regression net over the enumerator-construction
helper the Scripted and Sequential/YAML engines now share (decision:
`testing.scripted-playout-golden-deferred`). The **Sequential** case
(`Sequential_yaml`, ersatztv#381)
builds from a committed YAML fixture (`Goldens/Fixtures/sequential-schedule.yml`) instead of a
`ProgramSchedule`; it is TZ-independent (the `count`/`all`/`duration` handlers do UTC-only arithmetic —
it passes, not skips, under a non-UTC `TZ`) so needs no `Assume` guard. A third case,
`Classic_clock_padded` (ersatztv#77), locks clock-boundary padding: a `FillerMode.Pad` +
`PadToNearestMinute=15` PostRoll preset snaps content to `:15`, and the test both goldens the output and
asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is
`ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs` (guide programmes stop on the padded
boundary). The build reads
no wall clock — time enters only via the caller-supplied `start` — so a pinned `start` is fully
deterministic. Snapshots the raw `PlayoutItem.Start`/`Finish` (UTC), **not** the `*Offset` properties
(those call `.ToLocalTime()` and would make the golden machine-TZ dependent). The **Block** case is
TZ-sensitive by construction (`BlockPlayoutBuilder` maps template times via `TimeZoneInfo.Local`), so
it is guarded with `Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero)`: it runs under `TZ=UTC`
(CI) and reports **inconclusive** (a graceful skip, not a failure) under any other TZ. A real TZ seam
for the block builder is ersatztv#380's scope.
All three locate their golden files via `[CallerFilePath]`. A missing golden is a hard fail, not a
skip. Regenerate via `ETV_UPDATE_GOLDENS=1 dotnet test ...` (M3U/XMLTV) or
`ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ...` (playout build).
**Never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS` in CI or from an agent.** A golden
diff during normal test runs means the code changed the output — regenerating to make the diff go
away hides the change instead of surfacing it. Only a human who has confirmed the change is
intentional should regenerate.
## Timezone independence
The suite is timezone-independent (ersatztv#24). When constructing test `PlayoutItem`s, always
set a real `Start` (e.g. `startState.CurrentTime.UtcDateTime`) — never rely on the default
`DateTime.MinValue`, which underflows `DateTimeOffset.MinValue` once a non-UTC local offset is
applied (`StartOffset` calls `ToLocalTime()`). CI runs in UTC; local runs may not.
## Running tests
Full .NET gate:
```bash
dotnet build ErsatzTV.sln
TZ=UTC dotnet test
```
CI adds `--blame-hang-timeout 2m` to catch hangs.
Fast subsets:
```bash
# single project
dotnet test ErsatzTV.Tests
# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests
```
Web (`web/`):
```bash
npm test # vitest (excludes web/e2e — those need a live server)
npm run typecheck # tsc -b --pretty false
npm run lint # eslint .
npm run build # tsc -b && vite build
```
UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):
```bash
scripts/e2e-ui.sh # from the repo root, NOT web/
```
## Per-PR verification gate
Before opening a PR: build the solution, run `ErsatzTV.Tests` + `ErsatzTV.Core.Tests` (plus
`ErsatzTV.Scanner.Tests`, `ErsatzTV.Architecture.Tests` and `ErsatzTV.FFmpeg.Tests` if touched),
and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or
an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.
## See also
- `docs/ci-cd.md` — CI pipeline (test → migrations → build), versioning, dependency management.
- `docs/contributing.md` §1 — layering rules enforced by `ErsatzTV.Architecture.Tests`.
- `docs/contributing.md` §8 — short pointer back to this doc.