ersatztv/docs/testing.md

# Testing map

Purpose: authoritative map of what each test project/suite covers and how to run it. Read this
before adding tests, not just `docs/contributing.md` §8 (which now just points here).

## Test projects

> **Font dependency (ersatztv#732).** `Infrastructure/Graphics/TextElementBackgroundBoxTests` is the
> only suite that rasterises text, so it needs at least one system font to lay anything out. The CI
> image installs none explicitly — fonts arrive via `playwright install --with-deps` in
> `docker/ci/Dockerfile` (351 present in the pinned image, measured 2026-08-26). This is a real
> dependency, declared here so a future slimming of that install produces a known cause rather than a
> mystery red. `Baseline_Renders_A_Non_Empty_Bitmap` exists to make that failure loud: without it a
> fontless host would render a 0x0 bitmap and every relative geometry assertion would pass vacuously.

| Project | Covers | Notes |
|---|---|---|
| `ErsatzTV.Tests` | API controllers + MediatR handlers, plus the SkiaSharp text-overlay rasteriser (`Infrastructure/Graphics/`) | In-memory SQLite fixture: a shared `SqliteConnection("Data Source=:memory:;Foreign Keys=False")` kept open + `EnsureCreatedAsync()` (**not** full migration replay) + `PRAGMA foreign_keys=OFF`, then seed; a tiny `IDbContextFactory` wraps `new TvContext(...)`. ~1,870 tests (approximate on purpose — an exact count goes stale on every PR that adds one; the previous hardcoded 828 was off by over a thousand). |
| `ErsatzTV.Core.Tests` | Domain logic, scheduling, IPTV/XMLTV generation | References `ErsatzTV.Application` directly — there is no separate `Application.Tests` project. ~650 tests + 1 skipped under `TZ=UTC` (the Block playout golden additionally skips under a non-UTC `TZ`; see Golden-file nets). |
| `ErsatzTV.Scanner.Tests` | Library scanning: scan handlers, folder scanners, NFO readers | Handler tests substitute the folder scanners + `ILibraryRepository` and assert the resulting repository writes (e.g. `ScanLocalLibraryHandlerTests` pins which `LastScan` levels a scan records — ersatztv#264). Fakes/`Testably` back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains `Core/FFmpeg/TranscodingTests` — `[Explicit]` + `[Combinatorial]`, so it never runs in CI or a plain `dotnet test` (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline. |
| `ErsatzTV.Architecture.Tests` | Layering rules via NetArchTest.eNhancedEdition | Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See `docs/contributing.md` §1. |
| `ErsatzTV.FFmpeg.Tests` | FFmpeg command construction | Build a pipeline, assert the exact rendered arg string (`PipelineBuilderBaseTests.cs`). |
| `web/` (vitest) | React SPA unit tests | Run alongside typecheck + build (see below). Collects `src/**`, `web/scripts/**` *and* `web/vite-plugins/**`, but deliberately **excludes** `web/e2e/**` (the Playwright specs — vitest's default `**/*.spec.*` glob would otherwise run them under jsdom). Some files have git prerequisites since ersatztv#819, and the set is not fixed — ersatztv#883 added one. `web/src/api/pageSizeCallSites.guard.test.ts` and `web/src/api/completeAnnotations.guard.test.ts` need a git **checkout** AND, through it, the **binary**: they derive their file population from `git ls-files` via `web/vite-plugins/trackedSourceFiles.ts` rather than a directory walk, and refuse rather than falling back. `web/vite-plugins/trackedSourceFiles.realgit.test.ts` needs the **binary** but no checkout — it builds its own temp repository to prove that derivation by executing it. So it is not checkout-versus-binary: supplying a `.git` alone would not let any of them run. **The suite therefore runs only where git is present, and `docker/Dockerfile` is not such a place** — its web-build stage builds the SPA and does not test it (ersatztv#887). Enumerating the git-dependent files as Docker `--exclude`s was tried and REVERSED: that list is a population nothing derives, it went stale the first time a guard was added, and the resulting red is unreachable on a PR — `Build & push image (amd64)` is `if: github.event_name != 'pull_request'` — so it landed on `main` and on the release tag instead. The image is gated on `docker-build.yml`'s `test` job running the whole suite on a real checkout, held by `scripts/tests/test_image_build_delegates_the_spa_suite.py`. |
| `web/e2e/` (Playwright) | UI-interactive E2E flows against a **live** instance | Not a unit suite and **not** part of `npm test` — needs a running server, so it runs via `scripts/e2e-ui.sh` (boots its own fresh instance) and in CI as a step of the `functional-e2e` job. Headless Chromium, `serial`, `retries: 0`. Scope rule: assert only what the curl harness structurally cannot. See `docs/e2e-local.md` → "UI-E2E harness". |

## Golden-file nets

Three golden-file suites guard the highest-value, most-subtle output:

- **M3U**: `ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs` (ersatztv#11) — env var `ETV_UPDATE_GOLDENS`
- **XMLTV**: `ChannelGuideGoldenTests` (ersatztv#28) — env var `ETV_UPDATE_GOLDENS`
- **Playout build**: `ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs` (ersatztv#163)
  — env var **`ETV_UPDATE_PLAYOUT_GOLDENS`** (deliberately separate from `ETV_UPDATE_GOLDENS` so
  regenerating one net can't silently rewrite the other). Snapshots the `PlayoutItem`s each builder
  produces over a pinned build window. Covers the **Classic** (`PlaybackOrder.Chronological`),
  **Block**, and **Sequential (YAML)** builders. The **Scripted** *end-to-end pipeline* is excluded from this
  net — `ScriptedPlayoutBuilder` runs a user-authored external process that drives the engine over HTTP, which
  the in-memory harness can't pin. Scripted is instead covered in-process at two levels, described under
  "Scripted playout coverage" below (decision: `testing.scripted-engine-in-process-net`, ersatztv#563).
  The **Sequential** case (`Sequential_yaml`, ersatztv#381) builds from a committed YAML fixture
  (`Goldens/Fixtures/sequential-schedule.yml`) instead of a
  `ProgramSchedule`; it is TZ-independent (the `count`/`all`/`duration` handlers do UTC-only arithmetic —
  it passes, not skips, under a non-UTC `TZ`) so needs no `Assume` guard. A third case,
  `Classic_clock_padded` (ersatztv#77), locks clock-boundary padding: a `FillerMode.Pad` +
  `PadToNearestMinute=15` PostRoll preset snaps content to `:15`, and the test both goldens the output and
  asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is
  `ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs` (guide programmes stop on the padded
  boundary). The build reads
  no wall clock — time enters only via the caller-supplied `start` — so a pinned `start` is fully
  deterministic. Snapshots the raw `PlayoutItem.Start`/`Finish` (UTC), **not** the `*Offset` properties
  (those call `.ToLocalTime()` and would make the golden machine-TZ dependent). The **Block** case is
  TZ-sensitive by construction (`BlockPlayoutBuilder` maps template times via `TimeZoneInfo.Local`), so
  it is guarded with `Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero)`: it runs under `TZ=UTC`
  (CI) and reports **inconclusive** (a graceful skip, not a failure) under any other TZ. A real TZ seam
  for the block builder is ersatztv#380's scope.

All three locate their golden files via `[CallerFilePath]`. A missing golden is a hard fail, not a
skip. Regenerate via `ETV_UPDATE_GOLDENS=1 dotnet test ...` (M3U/XMLTV) or
`ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ...` (playout build).

**Never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS` in CI or from an agent.** A golden
diff during normal test runs means the code changed the output — regenerating to make the diff go
away hides the change instead of surfacing it. Only a human who has confirmed the change is
intentional should regenerate.

## Scripted playout coverage

Scripted playout is characterized in-process at two levels, and the transport is deliberately not covered
at all (ersatztv#563, decision `testing.scripted-engine-in-process-net`).

| Where | Covers |
| --- | --- |
| `ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs` | The engine build API a script drives: `AddCollection`, `AddCount`, `AddAll`, `AddDuration` (stop-before-end and trim), `PadUntilExact`, EPG guide-group locking, per-item `PlayoutHistory`, the `IsDone` no-progress halt and its reset, and the `GetAnchor`/`RestoreOrReset` round-trip a Continue build resumes from. Substituted repositories, no database. |
| `ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs` | A committed script fixture (`Controllers/Fixtures/scripted-build.json`), bound with the production body-binder configuration (`ApiJsonSettings`) and replayed through the real `ScriptedScheduleController` + `ScriptedPlayoutBuilderService.MockSession` + `SchedulingEngine`, with a pinned 13-item snapshot in the same line format the playout goldens use (both trimming actions target an instant *between* two content boundaries, so each one's `trim` argument reaches the engine's trim branch rather than sitting inert), plus the four adapter mappings: 404 on an unknown build id, 400 on an unparseable playback order, a silent fall back to `FillerKind.None` on an unparseable filler kind, and the engine's no-progress `InvalidOperationException` translated to a 400. |
| `ErsatzTV.Core.Tests/Scheduling/ContentEnumeratorBuilderTests.cs` | The enumerator-construction helper the Scripted and Sequential/YAML engines share (ersatztv#395). |
| `ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs` | What the replay's standalone binder does and does not share with the one MVC runs, so the paragraph below stays a measurement rather than a claim. |
| `ErsatzTV.Tests/Filters/ScriptedScheduleAuthorizationTests.cs` | The `ApiAuthorizationFilter` decision for every `ScriptedScheduleController` action, driven through the real filter over the endpoint metadata MVC builds from the production `Startup`: no credential is a 401 (with `Api:RequireKeyForReads` opted off, on the mutating actions only), the running build's own token is accepted (no `X-CSRF`), another build's token, an ended build's token, a wrong or empty token and the token on an unmarked endpoint are all 401, and the machine key still works (decision `api.scripted-build-token`, ersatztv#913). |
| `ErsatzTV.Core.Tests/Scheduling/ScriptedScheduling/ScriptedBuildTokenTests.cs` | Minting and validating the build-scoped token in `ScriptedPlayoutBuilderService`, and `ScriptedPlayoutBuilder.CreateCommand` putting it in the program's environment and not on its command line. |

The expected snapshot is a string constant in the test rather than a fourth golden file: the golden
harness lives in `ErsatzTV.Core.Tests`, which cannot reference a controller, and duplicating it would
create a second action-to-engine mapping — the thing the single-mapping design exists to avoid.

**Not covered, and not scheduled to be** (measured 2026-09-05): the `Cli.Wrap` launch of the user's own
program — exit code, the `PlayoutScriptedScheduleTimeoutSeconds` timeout, stdout capture; Kestrel and the
`Startup` middleware, including the host/`Settings.UiPort` check that 404s a foreign request; that
`ScriptedPlayoutBuilder.Build` launches through `CreateCommand`, the one place the build token is put in
the program's environment; and MVC model binding as a *wrapper* — the input formatter, model validation
(a non-nullable reference type picks up an implicit required check there) and the `[ApiController]`
automatic 400 either produces before the action runs, since the replay hands each action an
already-bound object. Hosting the real `Startup` would drag in the whole DI graph, a
hand-rolled minimal host would test a transport the product does not have, and the script is
user-authored by definition — so any committed script is a stand-in either way. The
`ApiAuthorizationFilter` *decision* is the exception: it is a pure function of the request and the
endpoint metadata, so `ScriptedScheduleAuthorizationTests` covers it without a host — the blind spot the
decision record named, which was a real 401 on every callback (ersatztv#913).

The *serializer* inside that binding wrapper is covered rather than scoped out. The replay deserializes
fixture bodies through `ErsatzTV.Serialization.ApiJsonSettings`, the same function `Startup` hands to
`AddNewtonsoftJson`, and two tests witness that choice through the replay's own bind helper. The
fixture's own bodies parse the same way under every plausible replacement, so each test carries a body
built to separate one of them:

| Replacement the test separates the production binder from | The body that separates them |
| --- | --- |
| `System.Text.Json` with web defaults | A body omitting the C# `required` member `collection`. Newtonsoft has no notion of `required` and deserializes it to a default; System.Text.Json rejects the body outright. `Production_Body_Binder_Ignores_Required_Members` asserts both halves. |
| a bare `new JsonSerializerSettings()` — still Newtonsoft, but without the production configuration | An explicit `"order": null`. `NullValueHandling.Ignore` keeps `ContentCollection.Order` at its declared `"shuffle"`; Newtonsoft's own `Include` default writes the null through, and `AddCollection`'s `Enum.TryParse` then returns a 400. `Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null` asserts the production side end to end. |

Both are statements about the serializer and stop there — what MVC validation does with such a body
belongs to the wrapper named above. What each test does to a suite when the binder is actually swapped
was measured while building it and is recorded in ersatztv#563, not here: under
`testing.mutation-claims-are-executed` such a sentence is a `CLAIMS` entry in
`scripts/tests/mutation_manifest.py` that executes every run or it is not written, and that harness runs
pytest, so an NUnit proof cannot be declared in it.

What is shared with production is the *configuration*, not the settings **object**. MVC applies it to
settings it has already configured; `ApiJsonSettings.Create()`, which every test outside the pipeline
uses, applies it to a bare one. Measured 2026-09-05, the standalone object therefore keeps Newtonsoft's
`MaxDepth` of 64 instead of MVC's stricter 32 and lacks MVC's `ProblemDetailsConverter` and
`ValidationProblemDetailsConverter`; `MissingMemberHandling`, `TypeNameHandling` and `DateParseHandling`
match. Neither gap can reach a scripted request body — the DTOs nest two levels and are never a
`ProblemDetails` — which is what makes `Create()` usable in a test at all, and no test may generalize
from it to "production" beyond that. `ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs` pins the
whole delta in both directions, so it fails rather than rots if either object moves.

`ApiJsonSettings` exists so that binder is *defined* once; it is not a drift detector, and the difference
is worth stating because it bounds what the suite can promise. Nothing here observes
`Startup.ConfigureServices`, and a byte-equal hand-copy of `Apply` is behaviourally indistinguishable
from calling it — so the extraction removes the duplicate rather than detecting drift in one. A mirror
that has *lost* something is what the suites separate: the table above on the read path, and
`OpenApiSerializerContractTests` on the write path, whose four cases assert the camelCase keys
`CustomContractResolver` produces and nothing else in the configuration supplies. Drift confined to
`ReferenceLoopHandling` or the `StringEnumConverter` is separated by neither.

## Timezone independence

The suite is timezone-independent (ersatztv#24). When constructing test `PlayoutItem`s, always
set a real `Start` (e.g. `startState.CurrentTime.UtcDateTime`) — never rely on the default
`DateTime.MinValue`, which underflows `DateTimeOffset.MinValue` once a non-UTC local offset is
applied (`StartOffset` calls `ToLocalTime()`). CI runs in UTC; local runs may not.

**Which `SchedulingEngine` calls are safe to put in a fixture** (audited 2026-09-05). `AddCount`,
`AddAll`, `AddDuration` and `PadUntilExact` preserve the instant — items are always written as
`_state.CurrentTime.UtcDateTime`, and `PadUntilExact`'s `ToLocalTime()` changes the offset the state
carries, not the moment. `WaitUntil(TimeOnly)` and `PadUntil(string)` are **not** safe: both read the
LOCAL day and time-of-day off `CurrentTime` and rebuild a target from them. `PadToNext` is safe only
until something localizes `CurrentTime` — it reads `.Year`/`.Month`/`.Day`/`.Hour`/`.Minute` in whatever
offset that value happens to carry, so a fixture that calls it after `PadUntilExact`, `WaitUntilExact` or
a Continue anchor becomes TZ-sensitive by ordering rather than by call. The scripted fixtures therefore
use Chronological order and only the instant-preserving instructions; they pass, rather than skip, under
`TZ=UTC`, `America/New_York`, `Australia/Lord_Howe` and `Asia/Kathmandu`.

## Running tests

Full .NET gate:

```bash
dotnet build ErsatzTV.sln
TZ=UTC dotnet test
```

CI adds `--blame-hang-timeout 2m` to catch hangs.

Fast subsets:

```bash
# single project
dotnet test ErsatzTV.Tests

# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests
```

Web (`web/`):

```bash
npm test -- --run  # vitest, single pass — use this for the SPA guards (see below)
npm test           # vitest WATCH mode. `pageSizeCallSites.guard.test.ts` reads the git index ONCE
                   # per dev-server lifetime while the glob refreshes, so it drifts BOTH ways: a
                   # file created mid-session reddens it misleadingly, and a file that was already
                   # untracked when the watcher started stays invisible to it after `git add` — a
                   # green that is not authoritative. Restarting the watcher swaps the first
                   # problem for the second; confirm with `npm test -- --run`.
npm run typecheck  # tsc -b --pretty false
npm run lint       # eslint .
npm run build      # tsc -b && vite build
```

UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):

```bash
scripts/e2e-ui.sh              # from the repo root, NOT web/
```

## Provider-parity fixtures (opt-in MySQL)

Most of `ErsatzTV.Tests` runs on the in-memory SQLite harness described above. A few fixtures in
`ErsatzTV.Tests/Integration/` instead drive a **real, migrated** database, and run the SAME body against
**both** providers because the behaviour they pin is provider-specific:

| Fixture | What is provider-specific about it |
| --- | --- |
| `LibraryFolderDedupeMigrationTests` | the #491 dedupe DML — two MySQL-only collation defects (case-insensitive grouping, then `PAD SPACE`) were unreachable from SQLite |
| `SchedulingCollectionColumnNullTests` | what a NULL column materializes as through a value converter (ersatztv#823) |
| `SearchFieldValuesProviderTests` | the search-field-values query shape, which differs per provider |

The MySQL half needs a live server, supplied as `ETV_TEST_MYSQL_CONNECTION`. **Without it these
fixtures `Assert.Ignore` — a visible skip, never a silent pass**, so an ordinary local run needs no
MySQL. Setting `ETV_REQUIRE_MYSQL_TESTS=1` turns that skip into a hard failure, for a runner that is
supposed to have one.

```bash
ETV_TEST_MYSQL_CONNECTION='Server=<host>;Port=3306;Uid=root;Pwd=<pw>;DefaultCommandTimeout=300;' \
  dotnet test ErsatzTV.Tests --filter FullyQualifiedName~SchedulingCollectionColumnNullTests
```

Each test uses a database name it generates per run, so isolation does not depend on a wipe
succeeding, and drops it in teardown. **CI does not currently run any of these MySQL halves** — the
`migrations` job spins a `mysql:8.4` service but only applies migrations to a fresh EMPTY database,
so it executes no data rows; re-arming these fixtures there is tracked by ersatztv#627.

## Per-PR verification gate

Before opening a PR: build the solution, run `ErsatzTV.Tests` + `ErsatzTV.Core.Tests` (plus
`ErsatzTV.Scanner.Tests`, `ErsatzTV.Architecture.Tests` and `ErsatzTV.FFmpeg.Tests` if touched),
and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or
an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.

## See also

- `docs/ci-cd.md` — CI pipeline (test → migrations → build), versioning, dependency management.
- `docs/contributing.md` §1 — layering rules enforced by `ErsatzTV.Architecture.Tests`.
- `docs/contributing.md` §8 — short pointer back to this doc.