Files
ersatztv/docs/testing.md
T
timothy bef3f54798 fix(732): close the review findings on the background box and the default attach
Two independent cold reviews (Codex GPT-5.6 and a worktree-isolated Opus agent) converged on the
same three serious problems; both returned BLOCKED.

Boot safety. The already-seeded branch of SeedOnNowNext used to touch the filesystem not at all,
so a template the app cannot read - one edited as root via `docker exec` while the app runs under
PUID/PGID, or a read-only /config - was harmless. Reading and rewriting it there put unguarded IO
ahead of DatabaseIsReady() in a BackgroundService whose default behavior is StopHost, i.e. a
restart loop triggered by a file permission an operator can set by hand. The upgrade is now
best-effort (broad catch, cancellation still propagates) and CreateDirectory moved back below the
already-seeded check.

Stranding. The backfill wrote its marker even when the built-in GraphicsElement row was merely
undiscovered - the row is normally created by RefreshGraphicsElements, which runs long after
startup. An install upgrading from before #74 seeds the template and backfills on the same boot,
so every pre-existing channel would have been left unattached forever, with no second chance. The
seeder now creates the row itself, so the marker is only ever written against a resolved element.
The test that asserted the old behavior is replaced by one that pins the upgrade case.

Overclaimed records. The never-re-attach guarantee holds from the marker onwards, not across the
upgrade boundary: an operator who cleared the overlay before this landed left no state that
survives, and the one-time backfill re-adds it. That is inherent to "on by default" rather than a
defect, but the record asserted otherwise. Also recorded: the upgrade is a one-way door, because
the loader rejects unknown YAML keys, so a rollback drops the overlay on every channel at once.

Smaller, all with tests: the wrap budget now subtracts the same integer inset the bitmap grows by
(a fractional padding overflowed width_percent by a pixel); an inset that cannot fit is clamped
instead of squeezing the text to 1px and letting the box grow unbounded; FitTextBlock carries halo
and line height across, which only became reachable once a box's insets could push an element into
the Scale path; the built-in lookup filters on Kind so a same-named element in another template
folder cannot win; CreateChannelHandler adds to the collection rather than assigning over it.

width_percent is documented as a budget rather than a guarantee - wrapping cannot break below one
glyph, so it overflows with or without a box. The doc also no longer claims an unparseable colour
is never substituted (true for the box, false for text_color) or that an unset background_color
means no box (a border alone draws one).

The mutation claim is now reproducible: all 19 clause mutations are listed in
docs/graphics-elements.md with the test each one reddens. Two candidate mutations produced NO test
summary because they failed to compile under warnings-as-error, which reads exactly like a pass in
a scrolled log - that is called out there too.

The new pixel tests are the first in the repo to rasterise text, so the CI image's font dependency
is now declared in docs/testing.md rather than incidental (351 fonts present in the pinned image,
measured; they arrive via playwright install --with-deps).

Decisions-Edit: yes
Refs: #732
2026-08-26 20:37:06 +02:00

8.6 KiB

Testing map

Purpose: authoritative map of what each test project/suite covers and how to run it. Read this before adding tests, not just docs/contributing.md §8 (which now just points here).

Test projects

Font dependency (ersatztv#732). Infrastructure/Graphics/TextElementBackgroundBoxTests is the only suite that rasterises text, so it needs at least one system font to lay anything out. The CI image installs none explicitly — fonts arrive via playwright install --with-deps in docker/ci/Dockerfile (351 present in the pinned image, measured 2026-08-26). This is a real dependency, declared here so a future slimming of that install produces a known cause rather than a mystery red. Baseline_Renders_A_Non_Empty_Bitmap exists to make that failure loud: without it a fontless host would render a 0x0 bitmap and every relative geometry assertion would pass vacuously.

Project Covers Notes
ErsatzTV.Tests API controllers + MediatR handlers, plus the SkiaSharp text-overlay rasteriser (Infrastructure/Graphics/) In-memory SQLite fixture: a shared SqliteConnection("Data Source=:memory:;Foreign Keys=False") kept open + EnsureCreatedAsync() (not full migration replay) + PRAGMA foreign_keys=OFF, then seed; a tiny IDbContextFactory wraps new TvContext(...). ~1,870 tests (approximate on purpose — an exact count goes stale on every PR that adds one; the previous hardcoded 828 was off by over a thousand).
ErsatzTV.Core.Tests Domain logic, scheduling, IPTV/XMLTV generation References ErsatzTV.Application directly — there is no separate Application.Tests project. ~650 tests + 1 skipped under TZ=UTC (the Block playout golden additionally skips under a non-UTC TZ; see Golden-file nets).
ErsatzTV.Scanner.Tests Library scanning: scan handlers, folder scanners, NFO readers Handler tests substitute the folder scanners + ILibraryRepository and assert the resulting repository writes (e.g. ScanLocalLibraryHandlerTests pins which LastScan levels a scan records — ersatztv#264). Fakes/Testably back the file-system-facing scanners. ~1,485 tests (approximate on purpose — an exact count goes stale on every PR that adds one). Additionally contains Core/FFmpeg/TranscodingTests — [Explicit] + [Combinatorial], so it never runs in CI or a plain dotnet test (it needs real ffmpeg/hardware) and contributes 0 to that count; run it by name when touching the transcoding pipeline.
ErsatzTV.Architecture.Tests Layering rules via NetArchTest.eNhancedEdition Core↛Infra/App/EF; FFmpeg↛all; App↛concrete providers. 5 tests. See docs/contributing.md §1.
ErsatzTV.FFmpeg.Tests FFmpeg command construction Build a pipeline, assert the exact rendered arg string (PipelineBuilderBaseTests.cs).
web/ (vitest) React SPA unit tests 995 tests across 105 files; run alongside typecheck + build (see below). Collects src/** and web/scripts/**, but deliberately excludes web/e2e/** (the Playwright specs — vitest's default **/*.spec.* glob would otherwise run them under jsdom).
web/e2e/ (Playwright) UI-interactive E2E flows against a live instance Not a unit suite and not part of npm test — needs a running server, so it runs via scripts/e2e-ui.sh (boots its own fresh instance) and in CI as a step of the functional-e2e job. Headless Chromium, serial, retries: 0. Scope rule: assert only what the curl harness structurally cannot. See docs/e2e-local.md → "UI-E2E harness".

Golden-file nets

Three golden-file suites guard the highest-value, most-subtle output:

  • M3U: ErsatzTV.Core.Tests/Iptv/ChannelPlaylistGoldenTests.cs (ersatztv#11) — env var ETV_UPDATE_GOLDENS
  • XMLTV: ChannelGuideGoldenTests (ersatztv#28) — env var ETV_UPDATE_GOLDENS
  • Playout build: ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs (ersatztv#163) — env var ETV_UPDATE_PLAYOUT_GOLDENS (deliberately separate from ETV_UPDATE_GOLDENS so regenerating one net can't silently rewrite the other). Snapshots the PlayoutItems each builder produces over a pinned build window. Covers the Classic (PlaybackOrder.Chronological), Block, and Sequential (YAML) builders. The Scripted end-to-end pipeline is excluded from this net — ScriptedPlayoutBuilder runs a user-authored external process that drives the engine over HTTP, which the in-memory harness can't pin; that integration harness is tracked in ersatztv#563. The scheduling behavior those scripts drive, though, lives in the in-process SchedulingEngine (the HTTP controller is a 1:1 pass-through) and IS directly testable — SchedulingEngineTests news it up with substitutes, and ContentEnumeratorBuilderTests (ersatztv#395) is the direct regression net over the enumerator-construction helper the Scripted and Sequential/YAML engines now share (decision: testing.scripted-playout-golden-deferred). The Sequential case (Sequential_yaml, ersatztv#381) builds from a committed YAML fixture (Goldens/Fixtures/sequential-schedule.yml) instead of a ProgramSchedule; it is TZ-independent (the count/all/duration handlers do UTC-only arithmetic — it passes, not skips, under a non-UTC TZ) so needs no Assume guard. A third case, Classic_clock_padded (ersatztv#77), locks clock-boundary padding: a FillerMode.Pad + PadToNearestMinute=15 PostRoll preset snaps content to :15, and the test both goldens the output and asserts every content item after the first starts on a quarter-hour. Its EPG counterpart is ErsatzTV.Core.Tests/Channels/ChannelGuideProjectorClockPadTests.cs (guide programmes stop on the padded boundary). The build reads no wall clock — time enters only via the caller-supplied start — so a pinned start is fully deterministic. Snapshots the raw PlayoutItem.Start/Finish (UTC), not the *Offset properties (those call .ToLocalTime() and would make the golden machine-TZ dependent). The Block case is TZ-sensitive by construction (BlockPlayoutBuilder maps template times via TimeZoneInfo.Local), so it is guarded with Assume.That(TimeZoneInfo.Local.BaseUtcOffset == Zero): it runs under TZ=UTC (CI) and reports inconclusive (a graceful skip, not a failure) under any other TZ. A real TZ seam for the block builder is ersatztv#380's scope.

All three locate their golden files via [CallerFilePath]. A missing golden is a hard fail, not a skip. Regenerate via ETV_UPDATE_GOLDENS=1 dotnet test ... (M3U/XMLTV) or ETV_UPDATE_PLAYOUT_GOLDENS=1 dotnet test ... (playout build).

Never set ETV_UPDATE_GOLDENS / ETV_UPDATE_PLAYOUT_GOLDENS in CI or from an agent. A golden diff during normal test runs means the code changed the output — regenerating to make the diff go away hides the change instead of surfacing it. Only a human who has confirmed the change is intentional should regenerate.

Timezone independence

The suite is timezone-independent (ersatztv#24). When constructing test PlayoutItems, always set a real Start (e.g. startState.CurrentTime.UtcDateTime) — never rely on the default DateTime.MinValue, which underflows DateTimeOffset.MinValue once a non-UTC local offset is applied (StartOffset calls ToLocalTime()). CI runs in UTC; local runs may not.

Running tests

Full .NET gate:

dotnet build ErsatzTV.sln
TZ=UTC dotnet test

CI adds --blame-hang-timeout 2m to catch hangs.

Fast subsets:

# single project
dotnet test ErsatzTV.Tests

# filtered
dotnet test ErsatzTV.Core.Tests --filter FullyQualifiedName~ChannelPlaylistGoldenTests

Web (web/):

npm test           # vitest (excludes web/e2e — those need a live server)
npm run typecheck  # tsc -b --pretty false
npm run lint       # eslint .
npm run build      # tsc -b && vite build

UI-E2E (needs a built solution + built SPA; boots and tears down its own instance):

scripts/e2e-ui.sh              # from the repo root, NOT web/

Per-PR verification gate

Before opening a PR: build the solution, run ErsatzTV.Tests + ErsatzTV.Core.Tests (plus ErsatzTV.Scanner.Tests, ErsatzTV.Architecture.Tests and ErsatzTV.FFmpeg.Tests if touched), and run the web test/lint/typecheck/build steps above. All must be green. A golden-file diff or an architecture-test failure is a hard stop — fix the code, don't regenerate/relax the test.

See also

  • docs/ci-cd.md — CI pipeline (test → migrations → build), versioning, dependency management.
  • docs/contributing.md §1 — layering rules enforced by ErsatzTV.Architecture.Tests.
  • docs/contributing.md §8 — short pointer back to this doc.