timothy/ersatztv
1
0
Fork 0
Code Issues 58 Pull Requests 8 Actions Packages Projects Releases 9 Wiki Activity

Scripted playout: integration-test harness (deferred from #381 golden net) #563

New Issue
Closed
opened 2026-07-22 19:12:07 +02:00 by timothy · 3 comments
timothy commented 2026-07-22 19:12:07 +02:00
Owner
Copy Link
Copy Source

Follow-up to #381, which added the Sequential (YAML) golden and deferred Scripted from the in-memory PlayoutBuildGoldenTests net.

Why scripted can't be an in-memory golden

ScriptedPlayoutBuilder (ErsatzTV.Core/Scheduling/ScriptedScheduling/ScriptedPlayoutBuilder.cs) does not build items in-process: it shells out via Cli.Wrap(scriptFile).ExecuteBufferedAsync(...) to an external script process (the JS interpreter driving ErsatzTV/Resources/Scripts/*.js), which talks back over http://localhost:{Settings.UiPort} to SchedulingEngine (scriptedPlayoutBuilderService.StartSession(engine)). Golden characterization (shared in-memory SQLite, no I/O, pinned clock) fundamentally does not fit an external-process + HTTP-callback architecture.

The deterministic core the scripts drive — SchedulingEngine — is in-process and unit-testable (there is a seed ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs, currently minimal), but the scripted builder itself is process orchestration + HTTP transport, which is integration-test territory.

Scope

Stand up an integration-test harness that:

  • boots the SchedulingEngine HTTP endpoint on an ephemeral port,
  • runs a committed deterministic script through the real Cli.Wrap path (or a documented in-process stand-in), and
  • asserts the resulting PlayoutItems over a pinned window (golden-style snapshot once the harness is deterministic).

Alternatively, expand SchedulingEngineTests to characterize the engine's build API directly (the part scripts call) and explicitly scope the Cli.Wrap/HTTP shell out as out-of-test.

Done-when

  • Either a scripted integration harness with a deterministic snapshot assertion, or expanded SchedulingEngineTests covering the engine build API with the process/HTTP shell-out explicitly scoped out (documented reasoning).
  • Tests TZ-independent or guarded per the #24 rule.
  • docs/testing.md updated.
  • Adversarial review passed.

Decision context recorded in docs/decisions.md (testing.scripted-playout-golden-deferred) and docs/testing.md.

Follow-up to #381, which added the **Sequential (YAML)** golden and deferred **Scripted** from the in-memory `PlayoutBuildGoldenTests` net. ## Why scripted can't be an in-memory golden `ScriptedPlayoutBuilder` (`ErsatzTV.Core/Scheduling/ScriptedScheduling/ScriptedPlayoutBuilder.cs`) does not build items in-process: it shells out via `Cli.Wrap(scriptFile).ExecuteBufferedAsync(...)` to an **external script process** (the JS interpreter driving `ErsatzTV/Resources/Scripts/*.js`), which talks back over `http://localhost:{Settings.UiPort}` to `SchedulingEngine` (`scriptedPlayoutBuilderService.StartSession(engine)`). Golden characterization (shared in-memory SQLite, no I/O, pinned clock) fundamentally does not fit an external-process + HTTP-callback architecture. The deterministic *core* the scripts drive — `SchedulingEngine` — is in-process and unit-testable (there is a seed `ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs`, currently minimal), but the scripted **builder** itself is process orchestration + HTTP transport, which is integration-test territory. ## Scope Stand up an integration-test harness that: - boots the `SchedulingEngine` HTTP endpoint on an ephemeral port, - runs a committed deterministic script through the real `Cli.Wrap` path (or a documented in-process stand-in), and - asserts the resulting `PlayoutItem`s over a pinned window (golden-style snapshot once the harness is deterministic). Alternatively, expand `SchedulingEngineTests` to characterize the engine's build API directly (the part scripts call) and explicitly scope the `Cli.Wrap`/HTTP shell out as out-of-test. ## Done-when - [x] Either a scripted integration harness with a deterministic snapshot assertion, or expanded `SchedulingEngineTests` covering the engine build API with the process/HTTP shell-out explicitly scoped out (documented reasoning). - [x] Tests TZ-independent or guarded per the #24 rule. - [x] `docs/testing.md` updated. - [x] Adversarial review passed. Decision context recorded in `docs/decisions.md` (`testing.scripted-playout-golden-deferred`) and `docs/testing.md`.
timothy added the enhancementci-cdpriority: low labels 2026-07-22 19:12:07 +02:00
timothy commented 2026-07-22 19:29:36 +02:00
Author
Owner
Copy Link
Copy Source

🔗 Session bundle — E2E / test-harness infra: #445, #533, #563

Test-harness infrastructure: #445 (UI-interactive Playwright headless flows for functional-E2E), #533 (e2e-local.sh readiness probe hangs on a reused config dir), #563 (scripted-playout integration harness deferred from #381). Sibling to the just-landed #381 golden work; all live in the E2E/test tooling.

When you claim any member, check this cluster for co-workable siblings and sweep the disjoint set in one session (per docs/handoffs/chicorytv-issue-queue.md → bundles).

🔗 **Session bundle — E2E / test-harness infra:** #445, #533, #563 Test-harness infrastructure: #445 (UI-interactive Playwright headless flows for functional-E2E), #533 (e2e-local.sh readiness probe hangs on a reused config dir), #563 (scripted-playout integration harness deferred from #381). Sibling to the just-landed #381 golden work; all live in the E2E/test tooling. When you claim any member, check this cluster for co-workable siblings and sweep the disjoint set in one session (per docs/handoffs/chicorytv-issue-queue.md → bundles).
timothy added the in-progress label 2026-09-05 04:56:56 +02:00
timothy commented 2026-09-05 04:57:08 +02:00
Author
Owner
Copy Link
Copy Source

Claiming #563 (process.parallel-session-claim).

  • branch: 563-scripted-playout-integration-harness
  • worktree: /Users/timothy/orca/workspaces/ersatztv/wt-563
  • base: origin/main @ 4a56d54283ef9be1f1b8ca50f8553b09d65720e8

Four-way check before the label: no open PR body references #563 (5 open PRs, all Renovate), git ls-remote --heads origin '*563*' is empty, the only pre-existing comment is the 2026-07-22 bundle note (#445 and #533 both closed), and origin/main was re-fetched.

Claiming #563 (process.parallel-session-claim). - branch: `563-scripted-playout-integration-harness` - worktree: `/Users/timothy/orca/workspaces/ersatztv/wt-563` - base: `origin/main` @ `4a56d54283ef9be1f1b8ca50f8553b09d65720e8` Four-way check before the label: no open PR body references #563 (5 open PRs, all Renovate), `git ls-remote --heads origin '*563*'` is empty, the only pre-existing comment is the 2026-07-22 bundle note (#445 and #533 both closed), and `origin/main` was re-fetched.
timothy commented 2026-09-05 10:20:45 +02:00
Author
Owner
Copy Link
Copy Source

Closing record

Outcome: PR http://192.168.1.95:3000/timothy/ersatztv/pulls/915 (563-scripted-playout-integration-harness @ 4454f1adfcdc5a4d94a848b3896a1ff3f739c73c). Takes the second arm from this issue: SchedulingEngineTests expanded from 1 to 21 tests covering the engine build API, plus ScriptedScheduleControllerTests replaying a committed script fixture through the real ScriptedScheduleController as the documented in-process stand-in for the Cli.Wrap/HTTP shell-out, which remains explicitly out of scope.

Root cause: N/A (test-harness build-out, not a bug fix). The original gap: ScriptedPlayoutBuilder shells out via Cli.Wrap to an external script process talking to SchedulingEngine over HTTP loopback, which the shared in-memory-SQLite golden harness from #381 doesn't fit.

Decisions/conventions changed: testing.scripted-engine-in-process-net (new, active) supersedes testing.scripted-playout-golden-deferred (git mv'd to docs/decisions/archive/testing/, frontmatter retargeted, body byte-identical). docs/decisions.md and the generated catalog (docs/decisions/README.md) updated accordingly.

Reusable knowledge: The replay fixture must bind through the production Newtonsoft configuration (ErsatzTV/Serialization/ApiJsonSettings.cs, shared with Startup and OpenApiSerializerContractTests) — a System.Text.Json stand-in silently diverges on a body omitting a required member. And a trimming instruction (AddDuration/PadUntilExact) that lands exactly on a content boundary silences its own Trim flag mutant — a fixture must aim off-boundary or the controller's request.Trim argument is witnessed by nothing. Both are now stated in the decision record's rule:.

Verification:

  • dotnet build ErsatzTV.sln -c Debug: 0 Warning(s), 0 Error(s).
  • dotnet test on all five relevant projects: ErsatzTV.Tests 2111 passed/6 skipped/0 failed; ErsatzTV.Core.Tests 716/1/0; ErsatzTV.Infrastructure.Tests 114/0/0; ErsatzTV.FFmpeg.Tests 83/0/0; ErsatzTV.Scanner.Tests 1504/0/0; ErsatzTV.Mcp.Tests 61/0/0; ErsatzTV.Architecture.Tests 7/0/0.
  • Mutation proof re-executed at finish time: ScriptedScheduleController.PadUntilExact(..., request.Trim, ...) -> false reddens Committed_Script_Fixture_Produces_The_Pinned_Snapshot (Failed: 1, Passed: 8, Total: 9); mutant reverted via git checkout --, tree verified clean.
  • TZ-independence: SchedulingEngineTests/ScriptedScheduleControllerTests verified passing (not skipped) under TZ=UTC, America/New_York, Australia/Lord_Howe, Asia/Kathmandu (per #24 rule).
  • PYTHONPATH=. python3 scripts/decisions_validate.py: OK, exit 0. build_decisions_catalog.py: no diff. check-doc-narrative.py --diff origin/main: 0 warnings.
  • BOM-check + dotnet format whitespace --verify-no-changes on every touched .cs file: clean.
  • CI armed on the PR head sha; awaiting Gitea run completion.

Deferred: The Cli.Wrap/HTTP shell-out itself (external script process launch, real Kestrel endpoint, ScriptedPlayoutBuilderService.StartSession over loopback) stays untested — the first arm of this issue, not taken. A pre-existing 401 blind spot on ScriptedScheduleController (no request-key credential supplied anywhere in the scripted pipeline) surfaced during this work and is filed as ersatztv#913.

Docs updated: docs/testing.md (new "Scripted playout coverage" section + TZ-independence rationale), docs/decisions.md, docs/decisions/README.md (regenerated catalog), docs/decisions/records/testing/scripted-engine-in-process-net.md (new), docs/decisions/archive/testing/scripted-playout-golden-deferred.md (moved), docs/api-conventions.md (binder-settings note).

Done-when evidence (boxes left unticked for the orchestrator to check):

  • Either a scripted integration harness with a deterministic snapshot assertion, or expanded SchedulingEngineTests covering the engine build API with the process/HTTP shell-out explicitly scoped out (documented reasoning) — evidenced by ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs (1→21 tests) and ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs (9 tests replaying Fixtures/scripted-build.json through the real controller, pinned 13-item snapshot), plus docs/decisions/records/testing/scripted-engine-in-process-net.md's explicit scope-out of the Cli.Wrap/HTTP shell-out.
  • Tests TZ-independent or guarded per the #24 rule — evidenced by the TZ sweep above (UTC, America/New_York, Australia/Lord_Howe, Asia/Kathmandu, all passing not skipped) and the fixtures' Chronological-order/instant-preserving construction documented in docs/testing.md.
  • docs/testing.md updated — evidenced by the diff in PR #915 (new "Scripted playout coverage" section, Timezone independence section).
  • Adversarial review passed — evidenced by the review history in PR #915 (round 1: 1 blocking/0 should-fix/2 nit, all answered by fix commits; round 2: 0 blocking/0 should-fix/4 nit, clean, loop ended).
## Closing record **Outcome:** PR http://192.168.1.95:3000/timothy/ersatztv/pulls/915 (`563-scripted-playout-integration-harness` @ `4454f1adfcdc5a4d94a848b3896a1ff3f739c73c`). Takes the second arm from this issue: `SchedulingEngineTests` expanded from 1 to 21 tests covering the engine build API, plus `ScriptedScheduleControllerTests` replaying a committed script fixture through the real `ScriptedScheduleController` as the documented in-process stand-in for the `Cli.Wrap`/HTTP shell-out, which remains explicitly out of scope. **Root cause:** N/A (test-harness build-out, not a bug fix). The original gap: `ScriptedPlayoutBuilder` shells out via `Cli.Wrap` to an external script process talking to `SchedulingEngine` over HTTP loopback, which the shared in-memory-SQLite golden harness from #381 doesn't fit. **Decisions/conventions changed:** `testing.scripted-engine-in-process-net` (new, active) supersedes `testing.scripted-playout-golden-deferred` (git mv'd to `docs/decisions/archive/testing/`, frontmatter retargeted, body byte-identical). `docs/decisions.md` and the generated catalog (`docs/decisions/README.md`) updated accordingly. **Reusable knowledge:** The replay fixture must bind through the *production* Newtonsoft configuration (`ErsatzTV/Serialization/ApiJsonSettings.cs`, shared with `Startup` and `OpenApiSerializerContractTests`) — a `System.Text.Json` stand-in silently diverges on a body omitting a `required` member. And a trimming instruction (`AddDuration`/`PadUntilExact`) that lands exactly on a content boundary silences its own `Trim` flag mutant — a fixture must aim off-boundary or the controller's `request.Trim` argument is witnessed by nothing. Both are now stated in the decision record's `rule:`. **Verification:** - `dotnet build ErsatzTV.sln -c Debug`: 0 Warning(s), 0 Error(s). - `dotnet test` on all five relevant projects: `ErsatzTV.Tests` 2111 passed/6 skipped/0 failed; `ErsatzTV.Core.Tests` 716/1/0; `ErsatzTV.Infrastructure.Tests` 114/0/0; `ErsatzTV.FFmpeg.Tests` 83/0/0; `ErsatzTV.Scanner.Tests` 1504/0/0; `ErsatzTV.Mcp.Tests` 61/0/0; `ErsatzTV.Architecture.Tests` 7/0/0. - Mutation proof re-executed at finish time: `ScriptedScheduleController.PadUntilExact(..., request.Trim, ...) -> false` reddens `Committed_Script_Fixture_Produces_The_Pinned_Snapshot` (`Failed: 1, Passed: 8, Total: 9`); mutant reverted via `git checkout --`, tree verified clean. - TZ-independence: `SchedulingEngineTests`/`ScriptedScheduleControllerTests` verified passing (not skipped) under `TZ=UTC, America/New_York, Australia/Lord_Howe, Asia/Kathmandu` (per #24 rule). - `PYTHONPATH=. python3 scripts/decisions_validate.py`: OK, exit 0. `build_decisions_catalog.py`: no diff. `check-doc-narrative.py --diff origin/main`: 0 warnings. - BOM-check + `dotnet format whitespace --verify-no-changes` on every touched `.cs` file: clean. - CI armed on the PR head sha; awaiting Gitea run completion. **Deferred:** The `Cli.Wrap`/HTTP shell-out itself (external script process launch, real Kestrel endpoint, `ScriptedPlayoutBuilderService.StartSession` over loopback) stays untested — the first arm of this issue, not taken. A pre-existing 401 blind spot on `ScriptedScheduleController` (no request-key credential supplied anywhere in the scripted pipeline) surfaced during this work and is filed as ersatztv#913. **Docs updated:** `docs/testing.md` (new "Scripted playout coverage" section + TZ-independence rationale), `docs/decisions.md`, `docs/decisions/README.md` (regenerated catalog), `docs/decisions/records/testing/scripted-engine-in-process-net.md` (new), `docs/decisions/archive/testing/scripted-playout-golden-deferred.md` (moved), `docs/api-conventions.md` (binder-settings note). --- **Done-when evidence** (boxes left unticked for the orchestrator to check): - [ ] Either a scripted integration harness with a deterministic snapshot assertion, or expanded `SchedulingEngineTests` covering the engine build API with the process/HTTP shell-out explicitly scoped out (documented reasoning) — evidenced by `ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs` (1→21 tests) and `ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs` (9 tests replaying `Fixtures/scripted-build.json` through the real controller, pinned 13-item snapshot), plus `docs/decisions/records/testing/scripted-engine-in-process-net.md`'s explicit scope-out of the `Cli.Wrap`/HTTP shell-out. - [ ] Tests TZ-independent or guarded per the #24 rule — evidenced by the TZ sweep above (UTC, America/New_York, Australia/Lord_Howe, Asia/Kathmandu, all passing not skipped) and the fixtures' Chronological-order/instant-preserving construction documented in `docs/testing.md`. - [ ] `docs/testing.md` updated — evidenced by the diff in PR #915 (new "Scripted playout coverage" section, Timezone independence section). - [ ] Adversarial review passed — evidenced by the review history in PR #915 (round 1: 1 blocking/0 should-fix/2 nit, all answered by fix commits; round 2: 0 blocking/0 should-fix/4 nit, clean, loop ended).
timothy removed the in-progress label 2026-09-05 10:43:25 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels ci-cd enhancement priority: low
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#563