Both issues are #701 deferrals, and they land together because both rewrite
the same decision record.
#823 -- can a null reach one of the six collection-valued scalar columns?
MEASURED against a real TvContext on BOTH providers (SQLite, and MySQL 8.4
on an ephemeral server), because the reasoning available beforehand pointed
the wrong way. The two converters differ on their read side --
IntCollectionValueConverter maps null-or-blank to Array.Empty<int>(), while
EnumCollectionJsonValueConverter would dereference the result of
JsonConvert.DeserializeObject -- so the expectation was that a NULL row
behaves differently per column. NEITHER RUNS: EF does not invoke a value
converter for a NULL column at all. All six materialize as CLR null, the
int converter's null-to-empty branch is dead on this path, and unguarded
each .Contains in AlternateScheduleSelector throws NullReferenceException.
A NULL reads as UNRESTRICTED -- the All*() sets -- not as empty. This is
the whole semantic question and the first draft got it backwards. It is
decided by the one NULL reachable WITHOUT any code writing one: Sqlite's
20240113140741_Add_PlayoutTemplate_DaysOfMonth adds the column with
nullable:true and NO defaultValue, so a PlayoutTemplate row inserted before
it holds NULL and by construction had no day-of-month restriction. Reading
that as empty INVERTS the row's meaning and silently stops the template
applying at all. All*() preserves it, and is how "no restriction recorded"
is already represented (GetPlayoutAlternateSchedulesHandler,
PreviewBlockPlayoutHandler). What does NOT decide it, and was wrongly cited
in the first draft: the API request records normalize an omitted field with
`?? []`, but that is a client omitting a field on a WRITE and says nothing
about what a legacy database NULL meant.
Two read sites, not one. Guarding only the selector would have left the
entity->DTO mappers unguarded, and those feed the SPA: PlayoutScheduleEditors
spreads the collection (`[...template.daysOfMonth]` -> TypeError on a JSON
null) and playoutTemplateCalendar's appliesToDate -- an exact port of
GetScheduleForDate -- calls .includes on it. Both mappers now substitute the
SAME defaults, so the preview agrees with what is actually scheduled. Neither
guard is assigned back onto the entity, which is the
media.nullable-primitive-collection-mutation mechanism.
Reachability, stated precisely rather than overclaimed. All six are
nullable:true on both providers, but a nullable column does not produce a
NULL row: five of the six were present at CreateTable, so a NULL there still
needs code to write one, and on MySQL there is NO code-path-free NULL for any
of the six. The write path ACCEPTS a null (SaveChanges succeeds, stores SQL
NULL) but no caller supplies one today -- every production construction of the
two commands goes through the request records. That is a property of the code,
not a live caller; claiming otherwise would be the banned "it's AsNoTracking
today" argument pointed the other way.
#824 -- ElasticSearchIndex.UpdateSong had no regression test
Issue option 1 (a non-network transport) shipped, and needed no new package:
Elastic.Transport.InMemoryRequestInvoker is public in the pinned version and
ElasticsearchClientSettings(NodePool, IRequestInvoker) accepts it, injected
into the private _client the way #701 injects the Lucene IndexWriter.
UpdateItems never runs `_client ??= CreateClient()`, so the injected instance
is the one used.
Two traps there are load-bearing, both measured: the canned response must
carry an `X-Elastic-Product: Elasticsearch` header or the client's product
check throws UnsupportedProductException INTO UpdateSong's catch, and an empty
body fails to deserialize the same way. Either turns the fixture into a green
measurement of the error path -- which is how it first failed here, caught by
the ThrowOnWarningLogger. The document id is asserted as the LAST PATH SEGMENT,
not by substring: the index name carries digits, so ShouldContain would stop
discriminating for a song whose id collided with one.
Six mutations executed, each disarming ITS OWN clause alone:
- `??=` restored in ElasticSearchIndex only -> the Elastic fixture reddens on
"metadata.Artists should be null but was []" while the LUCENE fixture stays
GREEN. The #824 hole demonstrated, not described.
- DaysOfWeek guard disarmed in the selector -> 4 red, 3 green (DaysOfMonth and
MonthsOfYear unaffected). Each clause is independently load-bearing.
- DaysOfMonth guard disarmed in Playouts.Mapper -> 1 red, 2 green.
- Elastic dropped from the covered set / mapped to the SAME fixture as Lucene /
mapped to a class with no [Test] -> SearchIndexMutationCoverageTests reddens
on each.
That coverage guard is the boundary fix the issue asked for: the covered set is
compared against an ISearchIndex population DERIVED FROM THE ASSEMBLY. Its claim
stops where the check does -- no static check can establish that a named fixture
actually DRIVES its indexer, so it forces a human to look rather than proving
coverage. ThrowOnWarningLogger moved to ErsatzTV.Tests/Support so both fixtures
share it; the Lucene fixture's assertions are otherwise untouched, since it is a
witnessed proof artifact.
No production change in ElasticSearchIndex.cs -- #824 is coverage only.
Docs: testing.md gains a "Provider-parity fixtures" section naming all THREE
opt-in-MySQL fixtures and recording that CI runs none of them (#627);
docs/README.md gains the matching task signal; guard-inventory.md's
hand-written C# guard list goes from five files to six. Scheduling/Mapper.cs
loses the UTF-8 BOM it inherited, per #311 fix-as-you-touch.
Local gate (with the MySQL lane armed): ErsatzTV.Tests 2096 passed / 0 skipped,
Core.Tests 693/1, Infrastructure.Tests 114, Architecture.Tests 7, Scanner.Tests
1504 -- 0 failures in each. scripts/tests 1228 passed / 2 skipped. dotnet format
whitespace --verify-no-changes clean; BOM check over the touched set with the
population COUNT asserted, because a bare zsh loop silently checks one
concatenated filename. decisions_validate OK.
Fixes#823Fixes#824
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019zUmJZHhVP7kXg5DV237TW
Adds parameterized invariant tests (Schedule_clock_padded_offline_multimode)
exercising schedule-level clock pad + offline advance across a 2-day window
(two midnight crossings) through Flood, Duration, and Multiple — previously
only PlayoutModeSchedulerOne had any coverage. Fixture uses sub-15-min content
so each padded item occupies one :15 slot and Duration's fill-the-block
contract tiles exactly (no off-boundary packing).
Part 2 (precision): replaces the day-boundary anchor-clamp magnitude heuristic
(overrun <= one pad interval on a padded schedule) with a precise signal — the
last scheduler now reports the exact offline-pad target it advanced CurrentTime
to (transient PlayoutSchedulerResult.ClockPadOfflineTarget, never persisted),
and the clamp exempts only when CurrentTime equals that target exactly. A
non-offline overrun (Duration/Flood/Multiple ending short of its natural end, a
hard-stop, a tail advance) changes CurrentTime away from the target and still
clamps, so persisted NextStart no longer shifts by up to an interval. No
persisted-schema/migration change. Output byte-identical for all existing
goldens.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the byte-identical PlaybackOrder -> IMediaCollectionEnumerator switch
shared by SchedulingEngine.EnumeratorForContent (Scripted) and
EnumeratorCache.GetEnumeratorForContent (Sequential/YAML) into one static
per-family seam, mirroring #380's ShuffleSourceBuilder. Each engine keeps its own
"not supported" warning on the None branch, so the per-engine message is unchanged.
Adds ContentEnumeratorBuilderTests pinning the block-shuffle-not-classic trap and
the unsupported-order -> None (#70) contract across all 8 unsupported orders.
Corrects the testing.scripted-playout-golden-deferred decision record: Scripted's
external-process + HTTP pipeline is integration-only (deferred to #563), but the
in-process SchedulingEngine it drives IS unit-testable (ScriptedScheduleController
is a 1:1 pass-through) -- the earlier "un-golden-able by construction" framing
conflated transport with engine. docs/testing.md reframed to match.
[decisions-edit]
fixes#395
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-ups (cold review of PR #457):
- Marathon can reach PlayoutBuilder's default arm via `goto default` when
its enumerator can't be built; the new warning claimed "Marathon is not
supported by classic scheduling", which is false. Distinguish
supported-but-failed-to-build (logs "could not build") from genuinely
unsupported using PlaybackOrderSupport, which also makes Classic a
runtime consumer of the matrix.
- PlaybackOrderSupportTests now asserts every SchedulingEngineKind has a
matrix entry, so a new engine kind fails the test instead of throwing
KeyNotFoundException at runtime.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adding a new PlaybackOrder was unsafe by construction: three build-time
dispatch sites turned an unknown value into an enumerator silently.
Classic substituted RandomizedMediaCollectionEnumerator (the // TODO
default arm), PlaylistEnumerator had no default arm so the item was
dropped, and BlockPlayoutBuilder's allow-list continue skipped it.
(#70 already made YAML/Scripted log a warning and MultiCollectionGroup
throws.)
- each silent site now logs a Warning naming the order + engine + the
fallback taken; the fallback itself is preserved so a live channel
never goes dark on one misconfigured item and scheduler goldens do
not move.
- PlaylistEnumerator.Create gained an optional Option<ILogger> (it was
static with no logger -- why the drop was unreportable); loggered
callers pass it.
- BlockPlayoutBuilder gained an explicit Random arm (it previously
reached an enumerator only via the coincidental _ => fallback) and a
loud defensive fallback.
- new PlaybackOrderSupport matrix (per SchedulingEngineKind) + tripwire
PlaybackOrderSupportTests: Supported ∪ Unsupported must partition the
enum for every engine, so a new order fails the test until classified.
BlockPlayoutBuilder consumes the matrix for its allow-list.
- write-path rejection left unchanged (#70 closed the persistence hole;
the perimeter has been wrong three times per decisions.md); reverse
_ => None mappings reviewed and deferred (different axis; making them
loud would warn on legit enumerator types).
docs/decisions.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI's Formatting job failed: 19 touched files carried a BOM, which .editorconfig
forbids (charset=utf-8). Pure encoding change — one byte per file, no semantic
diff (verified: every hunk is `-namespace` -> `+namespace`).
Self-inflicted. The patches that edited these legacy files wrote them back as
utf-8-sig to "preserve the existing style", but the #311 fix-as-you-touch gate
requires a file to be normalized when you touch it — that is the whole point of
scoping the gate to changed files instead of reformatting the ~2500 legacy BOM
files at once. dotnet format leaves the EF-generated Designer/snapshot files
alone as generated code, and its verify skips them the same way, so they stay as
ef emitted them.
Two corrections to what I believed going in:
- `dotnet format --include` does NOT no-op here. It reported `error CHARSET` for
each file and exit 2, reproducing CI exactly, and fixed them in place. The note
claiming otherwise is wrong for this invocation.
- My first BOM check reported all files clean. The od pattern was wrong; reading
the first three bytes directly found 19. A detector that can only say "ok" is
worse than no detector.
Core.Tests 565, ErsatzTV.Tests 1673, Architecture.Tests 5 — all passed. API
artifacts still in sync.
Refs #70
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review of PR #402 returned BLOCKED. It could not break the WRR math or
the stateless-restore claim (it probed restore across wraps at indices 12/13/20/37
— all held, and the clamp preserves a 1000:1 ratio exactly). What it broke was the
perimeter.
B1 — the validation gate had a hole, so the silent-drop bug shipped.
CreateChannelFromLineup is a THIRD writer of PlaylistItem.PlaybackOrder; its own
guard only covered MultiCollection entries, so a 2+ entry lineup of plain
collections persisted WeightedShuffle straight through to PlaylistEnumerator's
null-drop. My decisions.md claim that "the silent sites never see it" was false as
written — corrected in place, with the lesson recorded: grep every writer of the
field, the non-obvious composite handler is the one that gets missed. The
Add*ToPlaylist handlers are safe only because they hardcode their order.
B2 — Weight had no validation at all, and create/update disagreed on the same
input. EF's HasDefaultValue(1) substitutes 1 for a 0 on INSERT (0 reads as "not
set") but an UPDATE writes the 0 through — and a 0-weight source was filtered out
of the rotation, deleting it from the channel silently. Exactly the failure this
order is careful to avoid everywhere else. Now bounded 1..1000 by a shared
MultiCollectionItemWeight used by both paths so they cannot drift, and clamped
again in the enumerator for rows that predate the gate.
B3 — Sum(weights) is checked arithmetic, so two int.MaxValue weights threw
OverflowException from inside a playout build. Reachable through the API precisely
because of B2. The ceiling fixes both; the sum also widens to long.
M1 the lineup mirror now allows WeightedShuffle for multi collections, matching the
PlayoutModeMustBeValid change it claims to mirror. M3 ScheduleAsGroup is documented
as deliberately unread by this order. L1 MinimumDuration is computed over every
source instead of the current rotation — under the clamp a rotation is a strict
subset and is rebuilt each wrap, so caching over it went stale. L2 the retry guard
keys off the rotation, not the raw collection count.
N1 the tautological default test is gone: it built entities in C#, so it asserted
the property initializer, not the migration — it could not have failed. Replaced
with clamp, overflow, and cross-wrap restore cases (the property the review proved
but found unpinned).
H1 the two follow-ups the PR body claimed were "filed" did not exist. Now filed:
#403 (silent dispatch-fallback hardening) and #404 (SPA weight UI, blocked-by #388).
Core.Tests 565 passed, ErsatzTV.Tests 1643 passed, 0 failed.
Refs #70
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WeightedShuffle is implemented for classic schedule items only, and every other
engine mishandles an order it doesn't know *silently*: PlaylistEnumerator has no
default arm so the item is dropped from the playlist; BlockPlayoutBuilder filters
block items against an allow-list and `continue`s past the rest; YAML and Scripted
return None, which their callers' foreach reads as "no content". A weighted order
degrading to unweighted random or to nothing is the worst failure mode here,
because the output is supposed to look arbitrary — nobody would notice.
Rather than change those shipped fallbacks (a real defect class, but pre-existing
and wider than this feature — filed separately, non-goal here), this closes the
new exposure at the write path: ReplacePlaylistItems and ReplaceBlockItems reject
WeightedShuffle with an error naming where it is available. If it can't be
persisted where it isn't handled, the silent sites never see it.
Defence in depth for the two engines that address orders by name: YAML and
Scripted now log a warning when a parsed order falls through unhandled, so an
empty schedule explains itself. Enum.Parse accepts "weightedShuffle" the moment
the value exists, so the gate above can't cover them. EnumeratorForContent becomes
an instance method to reach the logger.
ProgramScheduleItemCommandBase lists WeightedShuffle explicitly as valid for multi
collections — it already passed by falling through the switch, and implicit-by-
omission is how this subsystem grew its silent paths.
Core.Tests: 558 passed, 0 failed.
Refs #70
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ten tests pin the distribution contract exactly rather than statistically,
because the sequence IS the product decision: 3:1 emits A A B A (spread, not
drained); equal weights air a 2-item source as often as a 20-item one; ties break
to the earliest source; a defaulted weight behaves as fair-share (guarding the
migration default); restoring at an index equals advancing to it (the stateless
contract that lets CollectionEnumeratorState carry this order).
Fixes a hang found by the non-vacuity control. When a rotation wraps, MoveNext
retried the rebuild to avoid an immediate repeat. ShuffleInOrder can do that
unbounded because its reshuffle randomizes the lead item — but this order's lead
is decided by weight, so the heaviest source always leads, and when it holds a
single item the lead never changes and the retry never terminates. Two single-item
collections with unequal weights would wedge the playout build. The retry is now
bounded: avoiding a back-to-back repeat is a nicety, not terminating is not.
Regression test walks several wraps under a timeout.
Non-vacuity proven per repo lore: inverting the real WRR pick (max -> min, never
if(true), which trips CS0219 under warnings-as-errors and silently serves a stale
dll to --no-build) failed 5 of 10 tests on a clean build (0 errors, so not a
stale-dll false pass). Reverted; 10/10 green.
Refs #70
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds PlaybackOrder.WeightedShuffle = 9 (Classic engine only) plus the
per-source weight it distributes by.
Why a new order rather than making ShuffleInOrder weight-aware: ShuffleInOrder's
balanced shuffle pads sources to equal length with Option.None spacers, and
spacers emit nothing — so one cycle plays every item exactly once and airtime
stays proportional to collection size. It prevents *clumping*, not *domination*.
Retrofitting weights onto it would silently change shipped users' output.
WeightedShuffleCollectionEnumerator picks a source by smooth weighted
round-robin (acc += weight; richest wins; pays the total), then takes that
source's next item. Weights 3:1 emit A A B A; ties break to the earliest source
in list order. Fair-share is the equal-weights default, so one mechanism covers
both behaviors in the issue.
One rotation is sized so the source needing the most picks works through all its
items at its share; smaller sources loop within it — that looping is what makes
equal weights mean equal airtime regardless of library size. The sequence stays a
pure function of (Seed, Index), so it restores by replay like its siblings and
needs no per-source persisted counters.
It consumes ShuffleSourceBuilder.GetCollectionItemsForShuffleInOrder unchanged —
the schedule-entity-free entry point #380 reserved for this issue. Source
identity is destroyed on the Shuffle path (MultiCollectionGrouper collapses to
GroupedMediaItem + Distinct), so the weight is only reachable via
CollectionWithItems on the ShuffleInOrder-shaped path.
Weight is persisted on MultiCollectionItem AND MultiCollectionSmartItem — the
mirror is mandatory; omitting the smart item silently un-weights smart-collection
members. Both carry a DB default of 1: without it existing rows would migrate to
0, and a 0-weight source is dropped by the enumerator.
Refs #70
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eliminate the one cross-engine reach-in in the scheduler: PlaylistEnumerator called
PlayoutBuilder.GetGroupedMediaItemsForShuffle / GetCollectionItemsForShuffleInOrder as
statics (one engine reaching into another engine's class). Move both helpers verbatim to
a new public static ShuffleSourceBuilder in ErsatzTV.Core/Scheduling (sibling to the
also-static MultiCollectionGrouper / MultiPartEpisodeGrouper; deps passed as parameters,
not DI). Classic (PlayoutBuilder) and Playlist (PlaylistEnumerator) now share this one
place to build shuffle sources.
One intentional signature change: GetGroupedMediaItemsForShuffle takes
(bool keepMultiPartEpisodesTogether, bool treatCollectionsAsShows) instead of a
ProgramSchedule (verified those are the only two properties it read). This deletes
PlaylistEnumerator's fake `new ProgramSchedule { KeepMultiPartEpisodesTogether = false }`
(its TODO becomes an honest false, false) and gives callers without a ProgramSchedule
(#176, #70) a schedule-entity-free entry point.
Scope is deliberately (a)-only: engine separation preserved, no god-factory. Block stays
its own family; the Scripted/YAML construction duplication is a separate follow-up gated
on #381. See docs/decisions.md 2026-07-17.
Behavior-preserving: the characterization net added in the previous commit (classic-shuffle
golden byte-identical, PlaylistEnumerator reach-in sequence unchanged) plus new direct
ShuffleSourceBuilder unit tests (multi-collection vs fake-multi-collection lookup;
multi-part grouping on/off) all green. Full Core.Tests: 546 passed. PlayoutBuilder.cs
also de-BOM'd + whitespace-normalized per the fix-as-you-touch convention (#311).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor classic and block schedules to use same alternate schedule selector
* handle start year and end year
* add migrations for classic and block schedules
* allow editing block template start and end year
* add tests that include years
* add date range editing to classic (alternate) schedules
* fix running tests locally
* restore media files load; needed for local folder scanners
* update changelog
* feedback
* fix effective block tests running on github
* update dependencies
* pass tz again
* use tzconvert for time zones in tests
* temporary logging
* maybe fix
* test cleanup
timothyandClaude Opus 5