docs(563): state each binder test's mechanism instead of its mutant colour, per the amended CLAIMS rule
`testing.mutation-claims-are-executed` was amended on main while this branch was in review (#881, merged as #914): a sentence asserting that a specific mutation reddens — or does not redden — a named test is now either a `CLAIMS` entry in `scripts/tests/mutation_manifest.py` that executes every run, or it is not written. This branch carried six such sentences and none of them can be declared: `Claim.node_id` resolves a proof to `scripts/tests/<node id>` and `run_pytest` invokes pytest, so an NUnit proof has no representation in that harness at all. Durable prose now states the mechanism each test is built on — which serializer difference, which engine branch — which a reader re-checks by reading the code rather than by trusting a remembered outcome. The record says that in one paragraph, so the limit is stated rather than papered over. The outcomes themselves are here. Re-measured 2026-09-05 on this branch's tree (the commit before this one), each mutant applied to the working tree and restored from the index between runs, tree verified clean afterwards: positive control ScriptedScheduleControllerTests Passed: 9, Failed: 0 OpenApiSerializerContractTests Passed: 4, Failed: 0 Bind<T> -> System.Text.Json with JsonSerializerDefaults.Web Failed: 2, Passed: 7 — Production_Body_Binder_Ignores_Required_Members, Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null BodyBinderSettings = ApiJsonSettings.Create() -> new JsonSerializerSettings() Failed: 1, Passed: 8 — Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null OpenApiSerializerContractTests RuntimeSettings -> new JsonSerializerSettings() Failed: 4, Passed: 0 — all four cases, on PascalCase keys ScriptedScheduleController AddDuration(..., request.Trim, ...) -> false Failed: 1, Passed: 8 — Committed_Script_Fixture_Produces_The_Pinned_Snapshot ScriptedScheduleController PadUntilExact(..., request.Trim, ...) -> false Failed: 1, Passed: 8 — Committed_Script_Fixture_Produces_The_Pinned_Snapshot The last one is the round-two finding closed and re-witnessed: before the fixture's pad target moved off the content boundary, that mutant left all nine green. A squash merge writes its own message, so these figures also belong in the PR description. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QqCpYFsKgnAnx6jVwrKiV
This commit is contained in:
Claude Fable 5.1
@@ -25,11 +25,11 @@ namespace ErsatzTV.Tests.Controllers;
|
||||
public class OpenApiSerializerContractTests
|
||||
{
|
||||
// The configuration Startup.ConfigureServices -> AddNewtonsoftJson applies, from the same function
|
||||
// rather than a hand-copy of it. This fixture is the WRITE-side witness for the naming strategy: with
|
||||
// a plain JsonSerializerSettings here, all four cases go red on PascalCase keys. It says nothing about
|
||||
// NullValueHandling (every DTO member below is populated, so nothing is dropped either way) or the
|
||||
// StringEnumConverter (it compares key NAMES, not values). docs/testing.md -> "Scripted playout
|
||||
// coverage" tabulates which suite witnesses which half.
|
||||
// rather than a hand-copy of it. This fixture is the WRITE-side witness for the naming strategy: the
|
||||
// four cases below assert camelCase keys, which CustomContractResolver produces and a bare
|
||||
// JsonSerializerSettings does not. It says nothing about NullValueHandling (every DTO member below is
|
||||
// populated, so nothing is dropped either way) or the StringEnumConverter (it compares key NAMES, not
|
||||
// values). docs/testing.md -> "Scripted playout coverage" tabulates which suite witnesses which half.
|
||||
private static readonly JsonSerializerSettings RuntimeSettings = ApiJsonSettings.Create();
|
||||
|
||||
private static IEnumerable<TestCaseData> Cases()
|
||||
|
||||
@@ -68,7 +68,7 @@ public class ScriptedScheduleControllerTests
|
||||
private static readonly DateTimeOffset Start = new(2026, 1, 15, 6, 0, 0, TimeSpan.Zero);
|
||||
|
||||
// What Startup hands to AddNewtonsoftJson, from the same function rather than a hand-copy of it.
|
||||
// Replacing this with a plain `new()` reddens Production_Body_Binder_Keeps_Declared_Defaults_...
|
||||
// The two Production_Body_Binder_* tests below are what hold this to the production configuration.
|
||||
private static readonly JsonSerializerSettings BodyBinderSettings = ApiJsonSettings.Create();
|
||||
|
||||
// A body that omits the `required` member "collection"; Newtonsoft and System.Text.Json disagree.
|
||||
@@ -161,9 +161,9 @@ public class ScriptedScheduleControllerTests
|
||||
// The claim is about the SERIALIZER only — what MVC's validation layer then does with such a
|
||||
// body (non-nullable reference types pick up an implicit required check) is the uncovered
|
||||
// wrapper, not this. Binding through Bind itself is what gives this teeth: the fixture's own
|
||||
// bodies parse identically under either serializer, so only a body like this one witnesses the
|
||||
// swap. It witnesses THAT swap only — a Newtonsoft settings object that has merely lost the
|
||||
// production configuration still binds this body correctly, which is the sibling test's job.
|
||||
// bodies parse identically under either serializer, so only a body like this one separates them.
|
||||
// It separates THAT pair only — a Newtonsoft settings object that has merely lost the production
|
||||
// configuration reads this body exactly as production does, which is the sibling test's subject.
|
||||
using JsonDocument document = JsonDocument.Parse(BodyMissingRequiredMember);
|
||||
|
||||
ContentCollection bound = Bind<ContentCollection>(document.RootElement).ShouldNotBeNull();
|
||||
@@ -184,8 +184,8 @@ public class ScriptedScheduleControllerTests
|
||||
// production sets NullValueHandling.Ignore, so an explicit `"order": null` leaves ContentCollection
|
||||
// at its declared "shuffle" rather than overwriting it. Newtonsoft's own default is Include, which
|
||||
// writes the null through — and AddCollection's Enum.TryParse then rejects it as a 400. So this is
|
||||
// a behaviour difference a script would see, not a settings-shape assertion: it reddens if the bind
|
||||
// settings are replaced by a plain JsonSerializerSettings that merely looks like the production one.
|
||||
// a behaviour difference a script would see, not a settings-shape assertion — which is what makes
|
||||
// the "shuffle" and the OkResult below assertions about the production configuration itself.
|
||||
(ScriptedScheduleController controller, SchedulingEngine engine) = NewSession();
|
||||
|
||||
using JsonDocument document = JsonDocument.Parse(BodyWithExplicitNullOrder);
|
||||
|
||||
@@ -7,9 +7,9 @@ namespace ErsatzTV.Serialization;
|
||||
/// The Newtonsoft settings the MVC JSON formatters read and write `/api/*` bodies with. Defined once,
|
||||
/// here, so a test can deserialize a request body exactly the way the production body binder does
|
||||
/// instead of mirroring <see cref="Startup" />'s registration. This REMOVES the duplicate rather than
|
||||
/// detecting drift in one: no test observes <c>Startup.ConfigureServices</c>, so re-inlining its
|
||||
/// <c>AddNewtonsoftJson</c> lambda as a hand-copy of <see cref="Apply" /> would redden nothing. What
|
||||
/// the tests do catch is a caller that binds with something else — a body omitting a C#
|
||||
/// detecting drift in one: no test observes <c>Startup.ConfigureServices</c>, and a byte-equal hand-copy
|
||||
/// of <see cref="Apply" /> is behaviourally indistinguishable from calling it. What the tests do
|
||||
/// separate is a caller that binds with something else — a body omitting a C#
|
||||
/// <c>required</c> member deserializes to a default here where System.Text.Json throws, and
|
||||
/// <see cref="NullValueHandling" />.<c>Ignore</c> keeps a DTO's declared default over an explicit
|
||||
/// null where plain Newtonsoft writes the null through. Both are pinned by
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,7 +5,7 @@ status: active
|
||||
since: '2026-09-05'
|
||||
supersedes: testing.scripted-playout-golden-deferred@2026-07-22
|
||||
superseded-by: none
|
||||
rule: 'Scripted playout is characterized at two in-process levels and no further. (1) `SchedulingEngineTests` drives the engine build API directly with substituted repositories — the same setup order `ScriptedPlayoutBuilder` uses (`WithPlayoutId`/`WithMode`/`WithSeed`/`BuildBetween`/`WithReferenceData`/`RestoreOrReset`, and `WithReferenceData` MUST precede the last two or `PlayoutReferenceData.PlayoutHistory` NREs). (2) `ScriptedScheduleControllerTests` replays a committed script fixture (`ErsatzTV.Tests/Controllers/Fixtures/scripted-build.json`) through the REAL `ScriptedScheduleController` + `ScriptedPlayoutBuilderService.MockSession` + `SchedulingEngine` and pins the resulting `PlayoutItem`s as a snapshot in the golden line format — so exactly ONE action-to-engine mapping exists in the repo, the production one. Fixture bodies in that replay are deserialized with `ApiJsonSettings`, the configuration `Startup` applies to `AddNewtonsoftJson`, invoked from the same function rather than mirrored — and two tests WITNESS that choice instead of asserting it: `Production_Body_Binder_Ignores_Required_Members` reddens under a System.Text.Json binder, `Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null` reddens under a bare `JsonSerializerSettings`. What is shared is the CONFIGURATION, never MVC''s settings OBJECT: `ApiJsonSettings.Create()` applies it to a bare settings object, so MVC''s `MaxDepth` of 32 and its `ProblemDetails`/`ValidationProblemDetails` converters are absent (measured 2026-09-05, pinned by `ApiJsonSettingsTests`); neither is reachable from a scripted request body, and no test may generalize from `Create()` to production past that. What no test observes is the `Startup` registration itself, so the extraction REMOVES the duplicate rather than detecting its drift. The `Cli.Wrap` launch of the user-authored program (exit code, timeout, stdout capture), the Kestrel/middleware/`ApiAuthorizationFilter` transport it calls back over, and MVC model binding as a WRAPPER (the input formatter, model validation and the `[ApiController]` automatic 400 either produces before an action runs — the serializer inside that wrapper is covered, the wrapper around it is not) are NOT tested and are not scheduled to be; that residue is named below rather than argued away. Both fixtures must aim every trimming instruction at an instant BETWEEN two content boundaries: a pad or duration that happens to land on one never enters the engine''s trim branch, and that action''s `trim` argument then survives being replaced with `false`. The controller is a thin adapter, NOT a 1:1 pass-through: an unparseable playback order is a 400, an unparseable filler kind SILENTLY degrades to `FillerKind.None`, an unknown build id is a 404, and the engine no-progress `InvalidOperationException` is translated to a 400 — each pinned by a test, because a prose claim of equivalence decays.'
|
||||
rule: 'Scripted playout is characterized at two in-process levels and no further. (1) `SchedulingEngineTests` drives the engine build API directly with substituted repositories — the same setup order `ScriptedPlayoutBuilder` uses (`WithPlayoutId`/`WithMode`/`WithSeed`/`BuildBetween`/`WithReferenceData`/`RestoreOrReset`, and `WithReferenceData` MUST precede the last two or `PlayoutReferenceData.PlayoutHistory` NREs). (2) `ScriptedScheduleControllerTests` replays a committed script fixture (`ErsatzTV.Tests/Controllers/Fixtures/scripted-build.json`) through the REAL `ScriptedScheduleController` + `ScriptedPlayoutBuilderService.MockSession` + `SchedulingEngine` and pins the resulting `PlayoutItem`s as a snapshot in the golden line format — so exactly ONE action-to-engine mapping exists in the repo, the production one. Fixture bodies in that replay are deserialized with `ApiJsonSettings`, the configuration `Startup` applies to `AddNewtonsoftJson`, invoked from the same function rather than mirrored — and two tests WITNESS that choice instead of asserting it, each binding a body that separates the production binder from one plausible replacement: `Production_Body_Binder_Ignores_Required_Members` omits a C# `required` member, which Newtonsoft defaults and System.Text.Json rejects; `Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null` sends an explicit `"order": null`, which `NullValueHandling.Ignore` drops and a bare `JsonSerializerSettings` writes through into a 400. What is shared is the CONFIGURATION, never MVC''s settings OBJECT: `ApiJsonSettings.Create()` applies it to a bare settings object, so MVC''s `MaxDepth` of 32 and its `ProblemDetails`/`ValidationProblemDetails` converters are absent (measured 2026-09-05, pinned by `ApiJsonSettingsTests`); neither is reachable from a scripted request body, and no test may generalize from `Create()` to production past that. What no test observes is the `Startup` registration itself, so the extraction REMOVES the duplicate rather than detecting its drift. The `Cli.Wrap` launch of the user-authored program (exit code, timeout, stdout capture), the Kestrel/middleware/`ApiAuthorizationFilter` transport it calls back over, and MVC model binding as a WRAPPER (the input formatter, model validation and the `[ApiController]` automatic 400 either produces before an action runs — the serializer inside that wrapper is covered, the wrapper around it is not) are NOT tested and are not scheduled to be; that residue is named below rather than argued away. Both fixtures must aim every trimming instruction at an instant BETWEEN two content boundaries: a pad or duration that happens to land on one never enters the engine''s trim branch, so that action''s `trim` argument reaches nothing the snapshot can show. The controller is a thin adapter, NOT a 1:1 pass-through: an unparseable playback order is a 400, an unparseable filler kind SILENTLY degrades to `FillerKind.None`, an unknown build id is a 404, and the engine no-progress `InvalidOperationException` is translated to a 400 — each pinned by a test, because a prose claim of equivalence decays.'
|
||||
signals: 'why is there no scripted golden; scripted integration harness; scripted transport vs engine; SchedulingEngine is in-process testable; Cli.Wrap external process is transport only; shell-out scoped out; MockSession seam; in-process stand-in for a scripted script; engine build API characterization; committed script fixture replay; ScriptedScheduleController is an adapter not a pass-through; scripted filler kind silently falls back to None; scripted playout no-progress halt maps to 400; scripted build 401 blind spot; EnumeratorForContent · paths: `ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs`, `ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs`, `ErsatzTV.Tests/Controllers/Fixtures/scripted-build.json`, `ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs`, `ErsatzTV/Serialization/ApiJsonSettings.cs`, `ErsatzTV.Core/Scheduling/Engine/SchedulingEngine.cs`, `ErsatzTV/Controllers/Api/ScriptedScheduleController.cs`, `ErsatzTV.Core/Scheduling/ScriptedScheduling/ScriptedPlayoutBuilder.cs`, `ErsatzTV.Core.Tests/Scheduling/Goldens/PlayoutBuildGoldenTests.cs` · issues: #563, #381, #395, #163'
|
||||
mechanics: docs/testing.md → "Scripted playout coverage" and "Golden-file nets"
|
||||
---
|
||||
@@ -49,8 +49,17 @@ before an action runs, since every test hands an action an already-bound object;
|
||||
`ApiJsonSettings` itself, and two tests hold the two ways it can be replaced. `ApiJsonSettings` exists so
|
||||
that binder is *defined* once; it is not a drift **detector**, and the branch may not claim one, because
|
||||
nothing observes `Startup.ConfigureServices` and a byte-equal hand-copy of `Apply` is indistinguishable
|
||||
by construction. What the suites do catch is a mirror that has *lost* something. The mechanics doc
|
||||
tabulates which mutant reddens which suite, and what neither witnesses.
|
||||
by construction. What the suites do separate is a mirror that has *lost* something. The mechanics doc
|
||||
tabulates which suite separates which half of the configuration, and what neither reaches.
|
||||
|
||||
**Where the calibrating measurements are, and why not here.** `testing.mutation-claims-are-executed`
|
||||
admits a sentence asserting that a specific mutation reddens — or does not redden — a named test only as
|
||||
a `CLAIMS` entry in `scripts/tests/mutation_manifest.py`, executed every run; otherwise it is not
|
||||
written. That harness resolves a proof to `scripts/tests/<node id>` and runs pytest, so an NUnit proof
|
||||
cannot be declared in it and this coverage is entirely NUnit. The binder and `trim` mutants run while
|
||||
building these tests therefore stay in the commit messages that ran them, dated. Durable prose carries
|
||||
the mechanism each test is built on instead — which serializer difference, which engine branch — because
|
||||
that is re-checkable by reading the code rather than by trusting a remembered outcome.
|
||||
|
||||
`ApiAuthorizationFilter` on that list is not hypothetical — `ScriptedScheduleController` carries no
|
||||
`[SkipApiAuthorization]` while
|
||||
|
||||
+18
-13
@@ -71,7 +71,7 @@ at all (ersatztv#563, decision `testing.scripted-engine-in-process-net`).
|
||||
| Where | Covers |
|
||||
| --- | --- |
|
||||
| `ErsatzTV.Core.Tests/Scheduling/Engine/SchedulingEngineTests.cs` | The engine build API a script drives: `AddCollection`, `AddCount`, `AddAll`, `AddDuration` (stop-before-end and trim), `PadUntilExact`, EPG guide-group locking, per-item `PlayoutHistory`, the `IsDone` no-progress halt and its reset, and the `GetAnchor`/`RestoreOrReset` round-trip a Continue build resumes from. Substituted repositories, no database. |
|
||||
| `ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs` | A committed script fixture (`Controllers/Fixtures/scripted-build.json`), bound with the production body-binder configuration (`ApiJsonSettings`) and replayed through the real `ScriptedScheduleController` + `ScriptedPlayoutBuilderService.MockSession` + `SchedulingEngine`, with a pinned 13-item snapshot in the same line format the playout goldens use (both trimming actions target an instant *between* two content boundaries, so each one's `trim` argument changes the snapshot), plus the four adapter mappings: 404 on an unknown build id, 400 on an unparseable playback order, a silent fall back to `FillerKind.None` on an unparseable filler kind, and the engine's no-progress `InvalidOperationException` translated to a 400. |
|
||||
| `ErsatzTV.Tests/Controllers/ScriptedScheduleControllerTests.cs` | A committed script fixture (`Controllers/Fixtures/scripted-build.json`), bound with the production body-binder configuration (`ApiJsonSettings`) and replayed through the real `ScriptedScheduleController` + `ScriptedPlayoutBuilderService.MockSession` + `SchedulingEngine`, with a pinned 13-item snapshot in the same line format the playout goldens use (both trimming actions target an instant *between* two content boundaries, so each one's `trim` argument reaches the engine's trim branch rather than sitting inert), plus the four adapter mappings: 404 on an unknown build id, 400 on an unparseable playback order, a silent fall back to `FillerKind.None` on an unparseable filler kind, and the engine's no-progress `InvalidOperationException` translated to a 400. |
|
||||
| `ErsatzTV.Core.Tests/Scheduling/ContentEnumeratorBuilderTests.cs` | The enumerator-construction helper the Scripted and Sequential/YAML engines share (ersatztv#395). |
|
||||
| `ErsatzTV.Tests/Serialization/ApiJsonSettingsTests.cs` | What the replay's standalone binder does and does not share with the one MVC runs, so the paragraph below stays a measurement rather than a claim. |
|
||||
|
||||
@@ -93,17 +93,21 @@ names the concrete blind spot this leaves, and it is tracked as ersatztv#913.
|
||||
|
||||
The *serializer* inside that binding wrapper is covered rather than scoped out. The replay deserializes
|
||||
fixture bodies through `ErsatzTV.Serialization.ApiJsonSettings`, the same function `Startup` hands to
|
||||
`AddNewtonsoftJson`, and two tests witness that choice through the replay's own bind helper — the
|
||||
fixture's own bodies parse the same way under any of these serializers, so each needs a body built to
|
||||
separate them (mutants run 2026-09-05):
|
||||
`AddNewtonsoftJson`, and two tests witness that choice through the replay's own bind helper. The
|
||||
fixture's own bodies parse the same way under every plausible replacement, so each test carries a body
|
||||
built to separate one of them:
|
||||
|
||||
| Replace the replay binder with | Result |
|
||||
| Replacement the test separates the production binder from | The body that separates them |
|
||||
| --- | --- |
|
||||
| `System.Text.Json` with web defaults | 2 of 9 red. Newtonsoft has no notion of the C# `required` keyword, so a body omitting one deserializes to a default where System.Text.Json rejects it outright. |
|
||||
| a bare `new JsonSerializerSettings()` — still Newtonsoft, but without the production configuration | 1 of 9 red. `NullValueHandling.Ignore` keeps `ContentCollection.Order` at its declared `"shuffle"` over an explicit `"order": null`; Newtonsoft's own `Include` default writes the null through, and `AddCollection`'s `Enum.TryParse` then returns a 400. |
|
||||
| `System.Text.Json` with web defaults | A body omitting the C# `required` member `collection`. Newtonsoft has no notion of `required` and deserializes it to a default; System.Text.Json rejects the body outright. `Production_Body_Binder_Ignores_Required_Members` asserts both halves. |
|
||||
| a bare `new JsonSerializerSettings()` — still Newtonsoft, but without the production configuration | An explicit `"order": null`. `NullValueHandling.Ignore` keeps `ContentCollection.Order` at its declared `"shuffle"`; Newtonsoft's own `Include` default writes the null through, and `AddCollection`'s `Enum.TryParse` then returns a 400. `Production_Body_Binder_Keeps_Declared_Defaults_Over_An_Explicit_Null` asserts the production side end to end. |
|
||||
|
||||
Both are statements about the serializer and stop there — what MVC validation does with such a body
|
||||
belongs to the wrapper named above.
|
||||
belongs to the wrapper named above. What each test does to a suite when the binder is actually swapped
|
||||
was measured while building it and is recorded in the commit that did so, not here: under
|
||||
`testing.mutation-claims-are-executed` such a sentence is a `CLAIMS` entry in
|
||||
`scripts/tests/mutation_manifest.py` that executes every run or it is not written, and that harness runs
|
||||
pytest, so an NUnit proof cannot be declared in it.
|
||||
|
||||
What is shared with production is the *configuration*, not the settings **object**. MVC applies it to
|
||||
settings it has already configured; `ApiJsonSettings.Create()`, which every test outside the pipeline
|
||||
@@ -117,11 +121,12 @@ whole delta in both directions, so it fails rather than rots if either object mo
|
||||
|
||||
`ApiJsonSettings` exists so that binder is *defined* once; it is not a drift detector, and the difference
|
||||
is worth stating because it bounds what the suite can promise. Nothing here observes
|
||||
`Startup.ConfigureServices`, so re-inlining its `AddNewtonsoftJson` lambda as a hand-copy reddens no test
|
||||
— a byte-equal mirror is behaviourally indistinguishable by construction. A mirror that has *lost*
|
||||
something is caught: the table above on the read path, and `OpenApiSerializerContractTests` on the write
|
||||
path, where dropping `CustomContractResolver` turns all 4 cases red on PascalCase keys. Drift confined to
|
||||
`ReferenceLoopHandling` or the `StringEnumConverter` is witnessed by neither.
|
||||
`Startup.ConfigureServices`, and a byte-equal hand-copy of `Apply` is behaviourally indistinguishable
|
||||
from calling it — so the extraction removes the duplicate rather than detecting drift in one. A mirror
|
||||
that has *lost* something is what the suites separate: the table above on the read path, and
|
||||
`OpenApiSerializerContractTests` on the write path, whose four cases assert the camelCase keys
|
||||
`CustomContractResolver` produces and nothing else in the configuration supplies. Drift confined to
|
||||
`ReferenceLoopHandling` or the `StringEnumConverter` is separated by neither.
|
||||
|
||||
## Timezone independence
|
||||
|
||||
|
||||
Reference in New Issue
Block a user