feat(70): weighted / fair-share content distribution (WeightedShuffle playback order) #402

Merged

timothy merged 12 commits from feat/70-weighted-distribution into main 2026-07-17 19:00:48 +02:00

Conversation 7 Commits 12 Files Changed 42

+16133 -75

12 Commits

Author	SHA1	Message	Date
timothyandClaude Opus 4.8	4693aa14c3	docs(70): fix the last stale claim — found by reading every weight comment, not by grepping verbs ... Round-4 review returned BLOCKED, and its diagnosis is the sharpest of the four: my "re-derived by grep" claim was false. The reviewer's grep included `dropped`; mine had `drop(ped)? every` and missed the bare word — so MultiCollectionConfiguration.cs:38 ("a 0-weight source is dropped by the enumerator") never matched. That is the same failure as the previous round, one level up: I replaced a list of SITES with a list of VERBS I guessed and called it a class. Fourth consecutive incomplete completeness claim, and again the code twin of a doc line I had already corrected. The site is now accurate. The DB default of 1 is still right, but for the reason decisions.md already gives: a 0 backfill means nothing on a share-of-airtime scale. The enumerator clamps such a row to the floor, so it rotates rather than vanishing -- the drop it cited cannot happen since EffectiveWeight replaced the Weight > 0 filter, and A_Non_Positive_Weight_Does_Not_Delete_The_Source pins that. Method changed, not just the text: instead of guessing which verb a stale claim might use, enumerate EVERY comment/doc line mentioning weight across the diff and read them. 41 lines, one false. Both automated passes then produced false positives on different subjects -- a line-based check flagged EffectiveWeight's "must not drop..." (the framing verb wraps to the previous line), and a sentence-aware one flagged ReplacePlaylistItemsHandler's "the item is dropped from the playlist" (true, and about unknown ORDERS, not weights: PlaylistEnumerator really has 0 default arms and gates on `enumerator is not null` at :210). Both were read and cleared rather than counted, which is the only reason this message can say what it says. Verification: Build 0 errors; Core.Tests 566; ErsatzTV.Tests 1673; 0 failed; no BOM on any touched non-generated .cs; decisions-guard exit 0. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	17b39c8cff	docs(70): re-derive the stale-claim fix by grep instead of working the review's list ... Round-3 review returned BLOCKED: must-fix (b) was not closed. It was right, and the root cause it named is the point of this commit — the previous correction "was scoped to the four sites the reviewer listed rather than re-derived by grep". Fixing the list is not fixing the class. That is the same failure as B1, where the gate covered the two writers already in hand and missed CreateChannelFromLineup. Re-grepped the behavior class instead. Three survivors, two of them missed and one freshly introduced by the correction itself: - CreateMultiCollectionHandler.cs — the create twin of a comment whose UPDATE twin I corrected and whose create twin I never opened. Present tense, and contradicted by two tests in this same PR. - decisions.md — corrected one line in that file and left its sibling. - MultiCollectionItemWeight.cs (and its decisions.md mirror) — the ceiling rationale still claimed unbounded weights overflow the sum. They cannot: EffectiveWeight clamps before every sum and CycleLength widens to long. The earlier pass pattern-matched on the word "filtered" and left the identical defect on the ceiling. The ceiling's real job is the floor's argument — a billion is not a share of airtime any more than 0 is — so it now says that, and credits the clamp with the arithmetic safety it actually provides. Also corrected the writer claim to the right predicate: not "two persisting writers" (Add*ToPlaylist and Trakt persist it too, hardcoded) but two writers that persist a CALLER-SUPPLIED order. The full set is now classified persists-caller-value / persists-hardcoded / in-memory, including Engine/PlaylistHelper, which the previous "two Preview handlers" phrasing missed. That bullet has been wrong three times in the same shape; it now says so, since a lesson that keeps being re-learned is worth recording as a pattern rather than a fact. The BOM check caught this commit re-adding a BOM to the one file patched with utf-8-sig — the same trap, an hour after writing it down. Stripped; the mechanical pre-push check is what makes that survivable. Core.Tests 566, ErsatzTV.Tests 1673, 0 failed. Format verify exit 0. decisions.md +90/-0 (append-only guard green). Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	dff6aa5102	test(70): classic-weighted golden — weight beats collection size end-to-end ... The unit tests pin the enumerator's sequence in isolation; this pins that the real PlayoutBuilder actually distributes by weight, through the whole chain: MultiCollection -> MediaCollectionRepository -> CollectionWithItems.Weight -> ShuffleSourceBuilder -> WeightedShuffleCollectionEnumerator -> PlayoutItems. The fixture is deliberately lopsided: the HEAVY source (weight 3) is the SMALL collection (2 items) and the LIGHT source (weight 1) is the LARGE one (4). Over the pinned 2-day window that yields 81 : 27 = exactly 3.00 : 1 — the smaller collection taking the larger share, which is the entire point of the feature and is not reachable by any existing order: Shuffle is size-proportional (~2:4 here) and ShuffleInOrder plays every item exactly once per cycle. Longest consecutive same-source run is 3, so it interleaves smoothly rather than draining blocks. Same determinism contract as Classic_shuffle: pinned Playout.Seed + Continue mode (Reset randomizes the seed). Channel number/GUID 5 — every golden fixture shares one in-memory DB, so those must be globally unique. The net is armed, not decorative: with the golden absent the test fails (missing golden is a hard failure by design), and regenerating added only classic-weighted — the three existing goldens are byte-identical, so nothing was silently re-baselined. Core.Tests 566 passed, 0 failed. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	9e43621243	style(70): strip UTF-8 BOM from the .cs files this PR touches ... CI's Formatting job failed: 19 touched files carried a BOM, which .editorconfig forbids (charset=utf-8). Pure encoding change — one byte per file, no semantic diff (verified: every hunk is `-namespace` -> `+namespace`). Self-inflicted. The patches that edited these legacy files wrote them back as utf-8-sig to "preserve the existing style", but the #311 fix-as-you-touch gate requires a file to be normalized when you touch it — that is the whole point of scoping the gate to changed files instead of reformatting the ~2500 legacy BOM files at once. dotnet format leaves the EF-generated Designer/snapshot files alone as generated code, and its verify skips them the same way, so they stay as ef emitted them. Two corrections to what I believed going in: - `dotnet format --include` does NOT no-op here. It reported `error CHARSET` for each file and exit 2, reproducing CI exactly, and fixed them in place. The note claiming otherwise is wrong for this invocation. - My first BOM check reported all files clean. The od pattern was wrong; reading the first three bytes directly found 19. A detector that can only say "ok" is worse than no detector. Core.Tests 565, ErsatzTV.Tests 1673, Architecture.Tests 5 — all passed. API artifacts still in sync. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	e2189f0dee	fix(70): test the weight gate; correct rationale my own clamp made false ... Re-review of the fix commit returned MERGEABLE-WITH-NITS. It verified the gate is now complete by enumerating the writers itself (no fourth persisting writer) and proved B2's fix works by writing throwaway handler tests — which was also its point: the fix shipped with none. B2 was create and update silently DISAGREEING on the same input, and the fix re-established agreement with nothing pinning it. Both paths are now driven from one shared case list, plus an explicit test that create and update agree on every case — the per-path tests would both have passed while the two diverged, which is how the bug existed in the first place. Non-vacuity proven: inverting only the update path's validation fails 10 of 20 on a clean build (0 errors, so not a stale-dll pass), and the agreement test is among the failures. The rest is my own prose contradicting my own code. The commit that added EffectiveWeight removed the weight filter, then left four statements asserting a 0-weight source "is filtered out" — two of them authored by that same commit, including the stated justification for Minimum=1 in MultiCollectionItemWeight. A future agent could have read that and deleted the clamp or the floor as redundant; they are belt-and-braces and neither is. Corrected to describe what the code now does: the gate refuses input that means nothing on a share-of-airtime scale, the clamp protects rows predating the gate. Also corrected the writer count in the very bullet whose lesson is "grep every writer of the field": ReplaceBlockItems writes BlockItem.PlaybackOrder, not PlaylistItem.PlaybackOrder. There are TWO persisting writers of PlaylistItem's, and the correction itself had miscounted by conflating the two fields — so the lesson now says to grep each field separately. Core.Tests 565 passed, ErsatzTV.Tests 1673 passed, 0 failed. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	2ef7cb3fdf	fix(70): close the review blockers — third playlist writer, weight bounds, overflow ... Adversarial review of PR #402 returned BLOCKED. It could not break the WRR math or the stateless-restore claim (it probed restore across wraps at indices 12/13/20/37 — all held, and the clamp preserves a 1000:1 ratio exactly). What it broke was the perimeter. B1 — the validation gate had a hole, so the silent-drop bug shipped. CreateChannelFromLineup is a THIRD writer of PlaylistItem.PlaybackOrder; its own guard only covered MultiCollection entries, so a 2+ entry lineup of plain collections persisted WeightedShuffle straight through to PlaylistEnumerator's null-drop. My decisions.md claim that "the silent sites never see it" was false as written — corrected in place, with the lesson recorded: grep every writer of the field, the non-obvious composite handler is the one that gets missed. The Add*ToPlaylist handlers are safe only because they hardcode their order. B2 — Weight had no validation at all, and create/update disagreed on the same input. EF's HasDefaultValue(1) substitutes 1 for a 0 on INSERT (0 reads as "not set") but an UPDATE writes the 0 through — and a 0-weight source was filtered out of the rotation, deleting it from the channel silently. Exactly the failure this order is careful to avoid everywhere else. Now bounded 1..1000 by a shared MultiCollectionItemWeight used by both paths so they cannot drift, and clamped again in the enumerator for rows that predate the gate. B3 — Sum(weights) is checked arithmetic, so two int.MaxValue weights threw OverflowException from inside a playout build. Reachable through the API precisely because of B2. The ceiling fixes both; the sum also widens to long. M1 the lineup mirror now allows WeightedShuffle for multi collections, matching the PlayoutModeMustBeValid change it claims to mirror. M3 ScheduleAsGroup is documented as deliberately unread by this order. L1 MinimumDuration is computed over every source instead of the current rotation — under the clamp a rotation is a strict subset and is rebuilt each wrap, so caching over it went stale. L2 the retry guard keys off the rotation, not the raw collection count. N1 the tautological default test is gone: it built entities in C#, so it asserted the property initializer, not the migration — it could not have failed. Replaced with clamp, overflow, and cross-wrap restore cases (the property the review proved but found unpinned). H1 the two follow-ups the PR body claimed were "filed" did not exist. Now filed: #403 (silent dispatch-fallback hardening) and #404 (SPA weight UI, blocked-by #388). Core.Tests 565 passed, ErsatzTV.Tests 1643 passed, 0 failed. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	87538e4dd5	docs(70): record the WeightedShuffle design and correct the ShuffleInOrder misreading ... decisions.md entry covers why one enum value rather than two or a separate setting, why the weight lives on both multi-collection join entities with a DB default of 1, why it can only be applied on the ShuffleInOrder-shaped path (source identity is destroyed on the Shuffle path), why it is stateless, and why cross-engine exposure is closed by write-path validation instead of by changing five shipped fallbacks. The load-bearing part is the ShuffleInOrder clarification. Its balanced-shuffle padding reads as equalization, but the spacers emit nothing — one cycle plays every item once, so airtime stays proportional to collection size. It is anti-clumping, not fair-share. That was misread during this issue's own design pass and would have collapsed #70 to "already shipped"; the distinction is the whole justification for the feature, so it is now written down in both decisions.md and the domain-model glossary rather than left to be rediscovered. domain-model.md gains a Playback order row contrasting Shuffle / ShuffleInOrder / WeightedShuffle, and notes on the multi-collection row that the two join entities are mirrors. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:34 +02:00
timothyandClaude Opus 4.8	0e41503921	feat(70): accept per-source weight on the multi-collection API ... Without this the weight is only reachable by editing the database, so the enumerator has nothing to distribute by. Weight is threaded through create and update (all four handler branches: add and update, plain and smart) and defaults to 1, so it is optional on the wire and /api/v1 stays additive under the freeze. It is returned on the read path too, which is load-bearing rather than symmetry: the update replaces the item list, so a client that GETs, edits a name, and PUTs back would silently reset every weight to the default if the GET didn't carry it. Weight edits ride the existing MultiCollection Version token, so If-Match/412 concurrency needs no new design. Regenerated v1.json + v1.d.ts + endpoint-index via update-openapi.sh and generate:api (never hand-edited). The spec picks up weight on both request and response models and WeightedShuffle in the PlaybackOrder enum; weight is emitted optional. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:24 +02:00
timothyandClaude Opus 4.8	b808f8139e	test(70): prove per-source Weight reaches the scheduler for both member kinds ... A multi collection holds plain collections and smart collections through two separate join entities that build CollectionWithItems at separate call sites. Only the plain one is obvious, and forgetting the smart mirror un-weights smart-collection members with nothing reporting it — so it gets a test rather than a comment. Three tests: both weights survive the round trip; an unweighted row reads back as 1, not 0 (a 0-weight source is filtered out by the enumerator, so a 0 default would silently drop every pre-existing member from a weighted rotation); and the real MediaCollectionRepository hands the weight to the scheduler — the mapping tests alone would pass even if the repo dropped it on the floor. ErsatzTV.Tests: 1641 passed, 0 failed. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:24 +02:00
timothyandClaude Opus 4.8	20b56905ec	feat(70): refuse WeightedShuffle where it isn't implemented, report where it can't be ... WeightedShuffle is implemented for classic schedule items only, and every other engine mishandles an order it doesn't know *silently*: PlaylistEnumerator has no default arm so the item is dropped from the playlist; BlockPlayoutBuilder filters block items against an allow-list and `continue`s past the rest; YAML and Scripted return None, which their callers' foreach reads as "no content". A weighted order degrading to unweighted random or to nothing is the worst failure mode here, because the output is supposed to look arbitrary — nobody would notice. Rather than change those shipped fallbacks (a real defect class, but pre-existing and wider than this feature — filed separately, non-goal here), this closes the new exposure at the write path: ReplacePlaylistItems and ReplaceBlockItems reject WeightedShuffle with an error naming where it is available. If it can't be persisted where it isn't handled, the silent sites never see it. Defence in depth for the two engines that address orders by name: YAML and Scripted now log a warning when a parsed order falls through unhandled, so an empty schedule explains itself. Enum.Parse accepts "weightedShuffle" the moment the value exists, so the gate above can't cover them. EnumeratorForContent becomes an instance method to reach the logger. ProgramScheduleItemCommandBase lists WeightedShuffle explicitly as valid for multi collections — it already passed by falling through the switch, and implicit-by- omission is how this subsystem grew its silent paths. Core.Tests: 558 passed, 0 failed. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:24 +02:00
timothyandClaude Opus 4.8	2fda7d28f8	test(70): pin WeightedShuffle semantics; fix unbounded reshuffle retry on wrap ... Ten tests pin the distribution contract exactly rather than statistically, because the sequence IS the product decision: 3:1 emits A A B A (spread, not drained); equal weights air a 2-item source as often as a 20-item one; ties break to the earliest source; a defaulted weight behaves as fair-share (guarding the migration default); restoring at an index equals advancing to it (the stateless contract that lets CollectionEnumeratorState carry this order). Fixes a hang found by the non-vacuity control. When a rotation wraps, MoveNext retried the rebuild to avoid an immediate repeat. ShuffleInOrder can do that unbounded because its reshuffle randomizes the lead item — but this order's lead is decided by weight, so the heaviest source always leads, and when it holds a single item the lead never changes and the retry never terminates. Two single-item collections with unequal weights would wedge the playout build. The retry is now bounded: avoiding a back-to-back repeat is a nicety, not terminating is not. Regression test walks several wraps under a timeout. Non-vacuity proven per repo lore: inverting the real WRR pick (max -> min, never if(true), which trips CS0219 under warnings-as-errors and silently serves a stale dll to --no-build) failed 5 of 10 tests on a clean build (0 errors, so not a stale-dll false pass). Reverted; 10/10 green. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:24 +02:00
timothyandClaude Opus 4.8	8c7e81c863	feat(70): WeightedShuffle playback order — weighted / fair-share distribution ... Adds PlaybackOrder.WeightedShuffle = 9 (Classic engine only) plus the per-source weight it distributes by. Why a new order rather than making ShuffleInOrder weight-aware: ShuffleInOrder's balanced shuffle pads sources to equal length with Option.None spacers, and spacers emit nothing — so one cycle plays every item exactly once and airtime stays proportional to collection size. It prevents *clumping*, not *domination*. Retrofitting weights onto it would silently change shipped users' output. WeightedShuffleCollectionEnumerator picks a source by smooth weighted round-robin (acc += weight; richest wins; pays the total), then takes that source's next item. Weights 3:1 emit A A B A; ties break to the earliest source in list order. Fair-share is the equal-weights default, so one mechanism covers both behaviors in the issue. One rotation is sized so the source needing the most picks works through all its items at its share; smaller sources loop within it — that looping is what makes equal weights mean equal airtime regardless of library size. The sequence stays a pure function of (Seed, Index), so it restores by replay like its siblings and needs no per-source persisted counters. It consumes ShuffleSourceBuilder.GetCollectionItemsForShuffleInOrder unchanged — the schedule-entity-free entry point #380 reserved for this issue. Source identity is destroyed on the Shuffle path (MultiCollectionGrouper collapses to GroupedMediaItem + Distinct), so the weight is only reachable via CollectionWithItems on the ShuffleInOrder-shaped path. Weight is persisted on MultiCollectionItem AND MultiCollectionSmartItem — the mirror is mandatory; omitting the smart item silently un-weights smart-collection members. Both carry a DB default of 1: without it existing rows would migrate to 0, and a 0-weight source is dropped by the enumerator. Refs #70 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 18:31:24 +02:00