ersatztv/docs/testing.md

Testing map

Purpose: authoritative map of what each test project/suite covers and how to run it. Read this before adding tests, not just docs/contributing.md §8 (which now just points here).

Test projects

Font dependency (ersatztv#732). Infrastructure/Graphics/TextElementBackgroundBoxTests is the only suite that rasterises text, so it needs at least one system font to lay anything out. The CI image installs none explicitly — fonts arrive via playwright install --with-deps in docker/ci/Dockerfile (351 present in the pinned image, measured 2026-08-26). This is a real dependency, declared here so a future slimming of that install produces a known cause rather than a mystery red. Baseline_Renders_A_Non_Empty_Bitmap exists to make that failure loud: without it a fontless host would render a 0x0 bitmap and every relative geometry assertion would pass vacuously.

Project	Covers	Notes
ErsatzTV.Tests	API controllers + MediatR handlers, plus the SkiaSharp text-overlay rasteriser (Infrastructure/Graphics/)	In-memory SQLite fixture: a shared SqliteConnection("Data Source=:memory:;Foreign Keys=False") kept open + EnsureCreatedAsync() (not full migration replay) + PRAGMA foreign_keys=OFF, then seed; a tiny IDbContextFactory wraps new TvContext(...). ~1,870 tests (approximate on purpose — an exact count goes stale on every PR that adds one; the previous hardcoded 828 was off by over a thousand).
ErsatzTV.Core.Tests	Domain logic, scheduling, IPTV/XMLTV generation	References ErsatzTV.Application directly — there is no separate Application.Tests project. ~650 tests + 1 skipped under TZ=UTC (the Block playout golden additionally skips under a non-UTC TZ; see Golden-file nets).
ErsatzTV.Scanner.Tests	Library scanning: scan handlers, folder scanners, NFO readers	Handler tests substitute the folder scanners + ILibraryRepository and assert the resulting repository writes (e.g. ScanLocalLibraryHandlerTests pins which LastScan levels a scan records — ersatztv#264). Fakes/Testably back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains Core/FFmpeg/TranscodingTests — [Explicit] + [Combinatorial], so it never runs in CI or a plain dotnet test (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline.
ErsatzTV.Architecture.Tests	Layering rules via NetArchTest.eNhancedEdition	Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See docs/contributing.md §1.
ErsatzTV.FFmpeg.Tests	FFmpeg command construction	Build a pipeline, assert the exact rendered arg string (PipelineBuilderBaseTests.cs).
web/ (vitest)	React SPA unit tests	Run alongside typecheck + build (see below). Collects src/**, web/scripts/** and web/vite-plugins/**, but deliberately excludes web/e2e/** (the Playwright specs — vitest's default **/*.spec.* glob would otherwise run them under jsdom). Some files have git prerequisites since ersatztv#819, and the set is not fixed — ersatztv#883 added one. web/src/api/pageSizeCallSites.guard.test.ts and web/src/api/completeAnnotations.guard.test.ts need a git checkout AND, through it, the binary: they derive their file population from git ls-files via web/vite-plugins/trackedSourceFiles.ts rather than a directory walk, and refuse rather than falling back. web/vite-plugins/trackedSourceFiles.realgit.test.ts needs the binary but no checkout — it builds its own temp repository to prove that derivation by executing it. So it is not checkout-versus-binary: supplying a .git alone would not let any of them run. The suite therefore runs only where git is present, and docker/Dockerfile is not such a place — its web-build stage builds the SPA and does not test it (ersatztv#887). Enumerating the git-dependent files as Docker --excludes was tried and REVERSED: that list is a population nothing derives, it went stale the first time a guard was added, and the resulting red is unreachable on a PR — Build & push image (amd64) is if: github.event_name != 'pull_request' — so it landed on main and on the release tag instead. The image is gated on docker-build.yml's test job running the whole suite on a real checkout, held by scripts/tests/test_image_build_delegates_the_spa_suite.py.
web/e2e/ (Playwright)	UI-interactive E2E flows against a live instance	Not a unit suite and not part of npm test — needs a running server, so it runs via scripts/e2e-ui.sh (boots its own fresh instance) and in CI as a step of the functional-e2e job. Headless Chromium, serial, retries: 0. Scope rule: assert only what the curl harness structurally cannot. See docs/e2e-local.md → "UI-E2E harness".

Golden-file nets

Three golden-file suites guard the highest-value, most-subtle output:

• M3U: ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs (ersatztv#11) — env var ETV_UPDATE_GOLDENS
• XMLTV: ChannelGuideGoldenTests (ersatztv#28) — env var ETV_UPDATE_GOLDENS
• Playout build: ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs (ersatztv#163) — env var ETV_UPDATE_PLAYOUT_GOLDENS (deliberately separate from ETV_UPDATE_GOLDENS so regenerating one net can't silently rewrite the other). Snapshots the PlayoutItems each builder produces over a pinned build window. Covers the Classic (PlaybackOrder.Chronological), Block, and Sequential (YAML) builders. The Scripted end-to-end pipeline is excluded from this net — ScriptedPlayoutBuilder runs a user-authored external process that drives the engine over HTTP, which the in-memory harness can't pin. Scripted is instead covered in-process at two levels, described under "Scripted playout coverage" below (decision: testing.scripted-engine-in-process-net, ersatztv#563). The Sequential case (Sequential_yaml, ersatztv#381) builds from a committed YAML fixture (Goldens/Fixtures/sequential-schedule.yml) instead of a ProgramSchedule; it is TZ-independent (the count/all/duration handlers do UTC-only arithmetic — it passes, not skips, under a non-UTC TZ) so needs no Assume guard. A third case, Classic_clock_padded (ersatztv#77), locks clock-boundary padding: a FillerMode.Pad + PadToNearestMinute=15 PostRoll preset snaps content to :15, and the test both goldens the output and asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs (guide programmes stop on the padded boundary). The build reads no wall clock — time enters only via the caller-supplied start — so a pinned start is fully deterministic. Snapshots the raw PlayoutItem.Start/Finish (UTC), not the *Offset properties (those call .ToLocalTime() and would make the golden machine-TZ dependent). The Block case is TZ-sensitive by construction (BlockPlayoutBuilder maps template times via TimeZoneInfo.Local), so it is guarded with Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero): it runs under TZ=UTC (CI) and reports inconclusive (a graceful skip, not a failure) under any other TZ. A real TZ seam for the block builder is ersatztv#380's scope.

All three locate their golden files via [CallerFilePath]. A missing golden is a hard fail, not a skip. Regenerate via ETV_UPDATE_GOLDENS=1 dotnet test ... (M3U/XMLTV) or ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ... (playout build).

Never set ETV_UPDATE_GOLDENS / ETV_UPDATE_PLAYOUT_GOLDENS in CI or from an agent. A golden diff during normal test runs means the code changed the output — regenerating to make the diff go away hides the change instead of surfacing it. Only a human who has confirmed the change is intentional should regenerate.

Scripted playout coverage

Scripted playout is characterized in-process at two levels, and the transport is deliberately not covered at all (ersatztv#563, decision testing.scripted-engine-in-process-net).

Where	Covers
ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs	The engine build API a script drives: AddCollection, AddCount, AddAll, AddDuration (stop-before-end and trim), PadUntilExact, EPG guide-group locking, per-item PlayoutHistory, the IsDone no-progress halt and its reset, and the GetAnchor/RestoreOrReset round-trip a Continue build resumes from. Substituted repositories, no database.
ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs	A committed script fixture (Controllers/Fixtures/scripted-build.json), bound with the production body-binder configuration (ApiJsonSettings) and replayed through the real ScriptedScheduleController + ScriptedPlayoutBuilderService.MockSession + SchedulingEngine, with a pinned 13-item snapshot in the same line format the playout goldens use, plus the four adapter mappings: 404 on an unknown build id, 400 on an unparseable playback order, a silent fall back to FillerKind.None on an unparseable filler kind, and the engine's no-progress InvalidOperationException translated to a 400.
ErsatzTV.Core.Tests/Scheduling/ContentEnumeratorBuilderTests.cs	The enumerator-construction helper the Scripted and Sequential/YAML engines share (ersatztv#395).

The expected snapshot is a string constant in the test rather than a fourth golden file: the golden harness lives in ErsatzTV.Core.Tests, which cannot reference a controller, and duplicating it would create a second action-to-engine mapping — the thing the single-mapping design exists to avoid.

Not covered, and not scheduled to be (measured 2026-09-05): the Cli.Wrap launch of the user's own program — exit code, the PlayoutScriptedScheduleTimeoutSeconds timeout, stdout capture; Kestrel and the Startup middleware, including the host/Settings.UiPort check that 404s a foreign request; ApiAuthorizationFilter, which fail-closes every mutating verb for an endpoint without [SkipApiAuthorization]; and MVC model binding as a wrapper — the input formatter and the [ApiController] automatic 400 a failed bind produces before the action runs, since the replay hands each action an already-bound object. Hosting the real Startup would drag in the whole DI graph, a hand-rolled minimal host would test a transport the product does not have, and the script is user-authored by definition — so any committed script is a stand-in either way. The decision record names the concrete blind spot this leaves.

The serializer inside that binding wrapper is covered rather than scoped out, and the distinction is load-bearing: production binds /api/* bodies with Newtonsoft (Startup → AddNewtonsoftJson), which has no notion of the C# required keyword, so a body omitting a required member binds to a default and reaches the action where System.Text.Json would reject it outright. The replay therefore deserializes fixture bodies through ErsatzTV.Serialization.ApiJsonSettings — the same function Startup calls, not a mirror of it — and Production_Body_Binder_Ignores_Required_Members pins both halves of that divergence, so a swap to a lookalike serializer reddens instead of passing on bodies the two happen to agree about. OpenApiSerializerContractTests reads the same settings from the same place.

Timezone independence

The suite is timezone-independent (ersatztv#24). When constructing test PlayoutItems, always set a real Start (e.g. startState.CurrentTime.UtcDateTime) — never rely on the default DateTime.MinValue, which underflows DateTimeOffset.MinValue once a non-UTC local offset is applied (StartOffset calls ToLocalTime()). CI runs in UTC; local runs may not.

Which SchedulingEngine calls are safe to put in a fixture (audited 2026-09-05). AddCount, AddAll, AddDuration and PadUntilExact preserve the instant — items are always written as _state.CurrentTime.UtcDateTime, and PadUntilExact's ToLocalTime() changes the offset the state carries, not the moment. WaitUntil(TimeOnly) and PadUntil(string) are not safe: both read the LOCAL day and time-of-day off CurrentTime and rebuild a target from them. PadToNext is safe only until something localizes CurrentTime — it reads .Year/.Month/.Day/.Hour/.Minute in whatever offset that value happens to carry, so a fixture that calls it after PadUntilExact, WaitUntilExact or a Continue anchor becomes TZ-sensitive by ordering rather than by call. The scripted fixtures therefore use Chronological order and only the instant-preserving instructions; they pass, rather than skip, under TZ=UTC, America/New_York, Australia/Lord_Howe and Asia/Kathmandu.

Running tests

Full .NET gate:

dotnet build ErsatzTV.sln
TZ=UTC dotnet test

CI adds --blame-hang-timeout 2m to catch hangs.

Fast subsets:

# single project
dotnet test ErsatzTV.Tests

# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests

Web (web/):

npm test -- --run  # vitest, single pass — use this for the SPA guards (see below)
npm test           # vitest WATCH mode. `pageSizeCallSites.guard.test.ts` reads the git index ONCE
                   # per dev-server lifetime while the glob refreshes, so it drifts BOTH ways: a
                   # file created mid-session reddens it misleadingly, and a file that was already
                   # untracked when the watcher started stays invisible to it after `git add` — a
                   # green that is not authoritative. Restarting the watcher swaps the first
                   # problem for the second; confirm with `npm test -- --run`.
npm run typecheck  # tsc -b --pretty false
npm run lint       # eslint .
npm run build      # tsc -b && vite build

UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):

scripts/e2e-ui.sh              # from the repo root, NOT web/

Provider-parity fixtures (opt-in MySQL)

Most of ErsatzTV.Tests runs on the in-memory SQLite harness described above. A few fixtures in ErsatzTV.Tests/Integration/ instead drive a real, migrated database, and run the SAME body against both providers because the behaviour they pin is provider-specific:

Fixture	What is provider-specific about it
LibraryFolderDedupeMigrationTests	the #491 dedupe DML — two MySQL-only collation defects (case-insensitive grouping, then PAD SPACE) were unreachable from SQLite
SchedulingCollectionColumnNullTests	what a NULL column materializes as through a value converter (ersatztv#823)
SearchFieldValuesProviderTests	the search-field-values query shape, which differs per provider

The MySQL half needs a live server, supplied as ETV_TEST_MYSQL_CONNECTION. Without it these fixtures Assert.Ignore — a visible skip, never a silent pass, so an ordinary local run needs no MySQL. Setting ETV_REQUIRE_MYSQL_TESTS=1 turns that skip into a hard failure, for a runner that is supposed to have one.

ETV_TEST_MYSQL_CONNECTION='Server=<host>;Port=3306;Uid=root;Pwd=<pw>;DefaultCommandTimeout=300;' \
  dotnet test ErsatzTV.Tests --filter FullyQualifiedName~SchedulingCollectionColumnNullTests

Each test uses a database name it generates per run, so isolation does not depend on a wipe succeeding, and drops it in teardown. CI does not currently run any of these MySQL halves — the migrations job spins a mysql:8.4 service but only applies migrations to a fresh EMPTY database, so it executes no data rows; re-arming these fixtures there is tracked by ersatztv#627.

Per-PR verification gate

Before opening a PR: build the solution, run ErsatzTV.Tests + ErsatzTV.Core.Tests (plus ErsatzTV.Scanner.Tests, ErsatzTV.Architecture.Tests and ErsatzTV.FFmpeg.Tests if touched), and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.

See also

• docs/ci-cd.md — CI pipeline (test → migrations → build), versioning, dependency management.
• docs/contributing.md §1 — layering rules enforced by ErsatzTV.Architecture.Tests.
• docs/contributing.md §8 — short pointer back to this doc.