ersatztv/docs/testing.md

Testing map

Purpose: authoritative map of what each test project/suite covers and how to run it. Read this before adding tests, not just docs/contributing.md §8 (which now just points here).

Test projects

Font dependency (ersatztv#732). Infrastructure/Graphics/TextElementBackgroundBoxTests is the only suite that rasterises text, so it needs at least one system font to lay anything out. The CI image installs none explicitly — fonts arrive via playwright install --with-deps in docker/ci/Dockerfile (351 present in the pinned image, measured 2026-08-26). This is a real dependency, declared here so a future slimming of that install produces a known cause rather than a mystery red. Baseline_Renders_A_Non_Empty_Bitmap exists to make that failure loud: without it a fontless host would render a 0x0 bitmap and every relative geometry assertion would pass vacuously.

Project	Covers	Notes
ErsatzTV.Tests	API controllers + MediatR handlers, plus the SkiaSharp text-overlay rasteriser (Infrastructure/Graphics/)	In-memory SQLite fixture: a shared SqliteConnection("Data Source=:memory:;Foreign Keys=False") kept open + EnsureCreatedAsync() (not full migration replay) + PRAGMA foreign_keys=OFF, then seed; a tiny IDbContextFactory wraps new TvContext(...). ~1,870 tests (approximate on purpose — an exact count goes stale on every PR that adds one; the previous hardcoded 828 was off by over a thousand).
ErsatzTV.Core.Tests	Domain logic, scheduling, IPTV/XMLTV generation	References ErsatzTV.Application directly — there is no separate Application.Tests project. ~650 tests + 1 skipped under TZ=UTC (the Block playout golden additionally skips under a non-UTC TZ; see Golden-file nets).
ErsatzTV.Scanner.Tests	Library scanning: scan handlers, folder scanners, NFO readers	Handler tests substitute the folder scanners + ILibraryRepository and assert the resulting repository writes (e.g. ScanLocalLibraryHandlerTests pins which LastScan levels a scan records — ersatztv#264). Fakes/Testably back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains Core/FFmpeg/TranscodingTests — [Explicit] + [Combinatorial], so it never runs in CI or a plain dotnet test (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline.
ErsatzTV.Architecture.Tests	Layering rules via NetArchTest.eNhancedEdition	Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See docs/contributing.md §1.
ErsatzTV.FFmpeg.Tests	FFmpeg command construction	Build a pipeline, assert the exact rendered arg string (PipelineBuilderBaseTests.cs).
web/ (vitest)	React SPA unit tests	Run alongside typecheck + build (see below). Collects src/**, web/scripts/** and web/vite-plugins/**, but deliberately excludes web/e2e/** (the Playwright specs — vitest's default **/*.spec.* glob would otherwise run them under jsdom). Some files have git prerequisites since ersatztv#819, and the set is not fixed — ersatztv#883 added one. web/src/api/pageSizeCallSites.guard.test.ts and web/src/api/completeAnnotations.guard.test.ts need a git checkout AND, through it, the binary: they derive their file population from git ls-files via web/vite-plugins/trackedSourceFiles.ts rather than a directory walk, and refuse rather than falling back. web/vite-plugins/trackedSourceFiles.realgit.test.ts needs the binary but no checkout — it builds its own temp repository to prove that derivation by executing it. So it is not checkout-versus-binary: supplying a .git alone would not let any of them run. The suite therefore runs only where git is present, and docker/Dockerfile is not such a place — its web-build stage builds the SPA and does not test it (ersatztv#887). Enumerating the git-dependent files as Docker --excludes was tried and REVERSED: that list is a population nothing derives, it went stale the first time a guard was added, and the resulting red is unreachable on a PR — Build & push image (amd64) is if: github.event_name != 'pull_request' — so it landed on main and on the release tag instead. The image is gated on docker-build.yml's test job running the whole suite on a real checkout, held by scripts/tests/test_image_build_delegates_the_spa_suite.py.
web/e2e/ (Playwright)	UI-interactive E2E flows against a live instance	Not a unit suite and not part of npm test — needs a running server, so it runs via scripts/e2e-ui.sh (boots its own fresh instance) and in CI as a step of the functional-e2e job. Headless Chromium, serial, retries: 0. Scope rule: assert only what the curl harness structurally cannot. See docs/e2e-local.md → "UI-E2E harness".

Golden-file nets

Three golden-file suites guard the highest-value, most-subtle output:

• M3U: ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs (ersatztv#11) — env var ETV_UPDATE_GOLDENS
• XMLTV: ChannelGuideGoldenTests (ersatztv#28) — env var ETV_UPDATE_GOLDENS
• Playout build: ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs (ersatztv#163) — env var ETV_UPDATE_PLAYOUT_GOLDENS (deliberately separate from ETV_UPDATE_GOLDENS so regenerating one net can't silently rewrite the other). Snapshots the PlayoutItems each builder produces over a pinned build window. Covers the Classic (PlaybackOrder.Chronological), Block, and Sequential (YAML) builders. The Scripted end-to-end pipeline is excluded from this net — ScriptedPlayoutBuilder runs a user-authored external process that drives the engine over HTTP, which the in-memory harness can't pin. Scripted is instead covered in-process at two levels, described under "Scripted playout coverage" below (decision: testing.scripted-engine-in-process-net, ersatztv#563). The Sequential case (Sequential_yaml, ersatztv#381) builds from a committed YAML fixture (Goldens/Fixtures/sequential-schedule.yml) instead of a ProgramSchedule; it is TZ-independent (the count/all/duration handlers do UTC-only arithmetic — it passes, not skips, under a non-UTC TZ) so needs no Assume guard. A third case, Classic_clock_padded (ersatztv#77), locks clock-boundary padding: a FillerMode.Pad + PadToNearestMinute=15 PostRoll preset snaps content to :15, and the test both goldens the output and asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs (guide programmes stop on the padded boundary). The build reads no wall clock — time enters only via the caller-supplied start — so a pinned start is fully deterministic. Snapshots the raw PlayoutItem.Start/Finish (UTC), not the *Offset properties (those call .ToLocalTime() and would make the golden machine-TZ dependent). The Block case is TZ-sensitive by construction (BlockPlayoutBuilder maps template times via TimeZoneInfo.Local), so it is guarded with Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero): it runs under TZ=UTC (CI) and reports inconclusive (a graceful skip, not a failure) under any other TZ. A real TZ seam for the block builder is ersatztv#380's scope.

All three locate their golden files via [CallerFilePath]. A missing golden is a hard fail, not a skip. Regenerate via ETV_UPDATE_GOLDENS=1 dotnet test ... (M3U/XMLTV) or ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ... (playout build).

Never set ETV_UPDATE_GOLDENS / ETV_UPDATE_PLAYOUT_GOLDENS in CI or from an agent. A golden diff during normal test runs means the code changed the output — regenerating to make the diff go away hides the change instead of surfacing it. Only a human who has confirmed the change is intentional should regenerate.

Scripted playout coverage

Scripted playout is characterized in-process at two levels, and the transport is deliberately not covered at all (ersatztv#563, decision testing.scripted-engine-in-process-net).

Where	Covers
ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs	The engine build API a script drives: AddCollection, AddCount, AddAll, AddDuration (stop-before-end and trim), PadUntilExact, EPG guide-group locking, per-item PlayoutHistory, the IsDone no-progress halt and its reset, and the GetAnchor/RestoreOrReset round-trip a Continue build resumes from. Substituted repositories, no database.
ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs	A committed script fixture (Controllers/Fixtures/scripted-build.json), bound with the production body-binder configuration (ApiJsonSettings) and replayed through the real ScriptedScheduleController + ScriptedPlayoutBuilderService.MockSession + SchedulingEngine, with a pinned 13-item snapshot in the same line format the playout goldens use (both trimming actions target an instant between two content boundaries, so each one's trim argument reaches the engine's trim branch rather than sitting inert), plus the four adapter mappings: 404 on an unknown build id, 400 on an unparseable playback order, a silent fall back to FillerKind.None on an unparseable filler kind, and the engine's no-progress InvalidOperationException translated to a 400.
ErsatzTV.Core.Tests/Scheduling/ContentEnumeratorBuilderTests.cs	The enumerator-construction helper the Scripted and Sequential/YAML engines share (ersatztv#395).
ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs	What the replay's standalone binder does and does not share with the one MVC runs, so the paragraph below stays a measurement rather than a claim.
ErsatzTV.Tests/Filters/ScriptedScheduleAuthorizationTests.cs	The ApiAuthorizationFilter decision for every ScriptedScheduleController action, driven through the real filter over the endpoint metadata MVC builds from the production Startup: no credential is a 401 (with Api:RequireKeyForReads opted off, on the mutating actions only), the running build's own token is accepted (no X-CSRF), another build's token, an ended build's token, a wrong or empty token and the token on an unmarked endpoint are all 401, and the machine key still works (decision api.scripted-build-token, ersatztv#913).
ErsatzTV.Core.Tests/Scheduling/ScriptedScheduling/ScriptedBuildTokenTests.cs	Minting and validating the build-scoped token in ScriptedPlayoutBuilderService, and ScriptedPlayoutBuilder.CreateCommand putting it in the program's environment and not on its command line.

The expected snapshot is a string constant in the test rather than a fourth golden file: the golden harness lives in ErsatzTV.Core.Tests, which cannot reference a controller, and duplicating it would create a second action-to-engine mapping — the thing the single-mapping design exists to avoid.

Not covered, and not scheduled to be (measured 2026-09-05): the Cli.Wrap launch of the user's own program — exit code, the PlayoutScriptedScheduleTimeoutSeconds timeout, stdout capture; Kestrel and the Startup middleware, including the host/Settings.UiPort check that 404s a foreign request; that ScriptedPlayoutBuilder.Build launches through CreateCommand, the one place the build token is put in the program's environment; and MVC model binding as a wrapper — the input formatter, model validation (a non-nullable reference type picks up an implicit required check there) and the [ApiController] automatic 400 either produces before the action runs, since the replay hands each action an already-bound object. Hosting the real Startup would drag in the whole DI graph, a hand-rolled minimal host would test a transport the product does not have, and the script is user-authored by definition — so any committed script is a stand-in either way. The ApiAuthorizationFilter decision is the exception: it is a pure function of the request and the endpoint metadata, so ScriptedScheduleAuthorizationTests covers it without a host — the blind spot the decision record named, which was a real 401 on every callback (ersatztv#913).

The serializer inside that binding wrapper is covered rather than scoped out. The replay deserializes fixture bodies through ErsatzTV.Serialization.ApiJsonSettings, the same function Startup hands to AddNewtonsoftJson, and two tests witness that choice through the replay's own bind helper. The fixture's own bodies parse the same way under every plausible replacement, so each test carries a body built to separate one of them:

Replacement the test separates the production binder from	The body that separates them
System.Text.Json with web defaults	A body omitting the C# required member collection. Newtonsoft has no notion of required and deserializes it to a default; System.Text.Json rejects the body outright. Production_Body_Binder_Ignores_Required_Members asserts both halves.
a bare new JsonSerializerSettings() — still Newtonsoft, but without the production configuration	An explicit "order": null. NullValueHandling.Ignore keeps ContentCollection.Order at its declared "shuffle"; Newtonsoft's own Include default writes the null through, and AddCollection's Enum.TryParse then returns a 400. Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null asserts the production side end to end.

Both are statements about the serializer and stop there — what MVC validation does with such a body belongs to the wrapper named above. What each test does to a suite when the binder is actually swapped was measured while building it and is recorded in ersatztv#563, not here: under testing.mutation-claims-are-executed such a sentence is a CLAIMS entry in scripts/tests/mutation_manifest.py that executes every run or it is not written, and that harness runs pytest, so an NUnit proof cannot be declared in it.

What is shared with production is the configuration, not the settings object. MVC applies it to settings it has already configured; ApiJsonSettings.Create(), which every test outside the pipeline uses, applies it to a bare one. Measured 2026-09-05, the standalone object therefore keeps Newtonsoft's MaxDepth of 64 instead of MVC's stricter 32 and lacks MVC's ProblemDetailsConverter and ValidationProblemDetailsConverter; MissingMemberHandling, TypeNameHandling and DateParseHandling match. Neither gap can reach a scripted request body — the DTOs nest two levels and are never a ProblemDetails — which is what makes Create() usable in a test at all, and no test may generalize from it to "production" beyond that. ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs pins the whole delta in both directions, so it fails rather than rots if either object moves.

ApiJsonSettings exists so that binder is defined once; it is not a drift detector, and the difference is worth stating because it bounds what the suite can promise. Nothing here observes Startup.ConfigureServices, and a byte-equal hand-copy of Apply is behaviourally indistinguishable from calling it — so the extraction removes the duplicate rather than detecting drift in one. A mirror that has lost something is what the suites separate: the table above on the read path, and OpenApiSerializerContractTests on the write path, whose four cases assert the camelCase keys CustomContractResolver produces and nothing else in the configuration supplies. Drift confined to ReferenceLoopHandling or the StringEnumConverter is separated by neither.

Timezone independence

The suite is timezone-independent (ersatztv#24). When constructing test PlayoutItems, always set a real Start (e.g. startState.CurrentTime.UtcDateTime) — never rely on the default DateTime.MinValue, which underflows DateTimeOffset.MinValue once a non-UTC local offset is applied (StartOffset calls ToLocalTime()). CI runs in UTC; local runs may not.

Which SchedulingEngine calls are safe to put in a fixture (audited 2026-09-05). AddCount, AddAll, AddDuration and PadUntilExact preserve the instant — items are always written as _state.CurrentTime.UtcDateTime, and PadUntilExact's ToLocalTime() changes the offset the state carries, not the moment. WaitUntil(TimeOnly) and PadUntil(string) are not safe: both read the LOCAL day and time-of-day off CurrentTime and rebuild a target from them. PadToNext is safe only until something localizes CurrentTime — it reads .Year/.Month/.Day/.Hour/.Minute in whatever offset that value happens to carry, so a fixture that calls it after PadUntilExact, WaitUntilExact or a Continue anchor becomes TZ-sensitive by ordering rather than by call. The scripted fixtures therefore use Chronological order and only the instant-preserving instructions; they pass, rather than skip, under TZ=UTC, America/New_York, Australia/Lord_Howe and Asia/Kathmandu.

Running tests

Full .NET gate:

dotnet build ErsatzTV.sln
TZ=UTC dotnet test

CI adds --blame-hang-timeout 2m to catch hangs.

Fast subsets:

# single project
dotnet test ErsatzTV.Tests

# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests

Web (web/):

npm test -- --run  # vitest, single pass — use this for the SPA guards (see below)
npm test           # vitest WATCH mode. `pageSizeCallSites.guard.test.ts` reads the git index ONCE
                   # per dev-server lifetime while the glob refreshes, so it drifts BOTH ways: a
                   # file created mid-session reddens it misleadingly, and a file that was already
                   # untracked when the watcher started stays invisible to it after `git add` — a
                   # green that is not authoritative. Restarting the watcher swaps the first
                   # problem for the second; confirm with `npm test -- --run`.
npm run typecheck  # tsc -b --pretty false
npm run lint       # eslint .
npm run build      # tsc -b && vite build

UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):

scripts/e2e-ui.sh              # from the repo root, NOT web/

Provider-parity fixtures (opt-in MySQL)

Most of ErsatzTV.Tests runs on the in-memory SQLite harness described above. A few fixtures in ErsatzTV.Tests/Integration/ instead drive a real, migrated database, and run the SAME body against both providers because the behaviour they pin is provider-specific:

Fixture	What is provider-specific about it
LibraryFolderDedupeMigrationTests	the #491 dedupe DML — two MySQL-only collation defects (case-insensitive grouping, then PAD SPACE) were unreachable from SQLite
SchedulingCollectionColumnNullTests	what a NULL column materializes as through a value converter (ersatztv#823)
SearchFieldValuesProviderTests	the search-field-values query shape, which differs per provider

The MySQL half needs a live server, supplied as ETV_TEST_MYSQL_CONNECTION. Without it these fixtures Assert.Ignore — a visible skip, never a silent pass, so an ordinary local run needs no MySQL. Setting ETV_REQUIRE_MYSQL_TESTS=1 turns that skip into a hard failure, for a runner that is supposed to have one.

ETV_TEST_MYSQL_CONNECTION='Server=<host>;Port=3306;Uid=root;Pwd=<pw>;DefaultCommandTimeout=300;' \
  dotnet test ErsatzTV.Tests --filter FullyQualifiedName~SchedulingCollectionColumnNullTests

Each test uses a database name it generates per run, so isolation does not depend on a wipe succeeding, and drops it in teardown. CI does not currently run any of these MySQL halves — the migrations job spins a mysql:8.4 service but only applies migrations to a fresh EMPTY database, so it executes no data rows; re-arming these fixtures there is tracked by ersatztv#627.

Per-PR verification gate

Before opening a PR: build the solution, run ErsatzTV.Tests + ErsatzTV.Core.Tests (plus ErsatzTV.Scanner.Tests, ErsatzTV.Architecture.Tests and ErsatzTV.FFmpeg.Tests if touched), and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.

See also

• docs/ci-cd.md — CI pipeline (test → migrations → build), versioning, dependency management.
• docs/contributing.md §1 — layering rules enforced by ErsatzTV.Architecture.Tests.
• docs/contributing.md §8 — short pointer back to this doc.