Search indexers mutate SongMetadata.Artists on a possibly-tracked entity (??= []) — same shape rejected in #691 #701

Closed
opened 2026-07-28 23:06:05 +02:00 by timothy · 2 comments
Owner

Found by adversarial review of PR #700 (#691). Pre-existing; not introduced by that PR, which correctly left these alone rather than widening scope.

Why this is worth a separate issue

#691 originally fixed the nullable SongMetadata.Artists/AlbumArtists by making the entity coalesce on read. That design was rejected after a reviewer probed the real TvContext: with EF's default PreferField access mode, mutating the field during a read on a tracked entity flips the entity to Modified, and the next SaveChanges writes [] over the NULL column.

Two places already do structurally the same thing, by assignment rather than by getter:

  • ErsatzTV.Infrastructure/Search/LuceneSearchIndex.cs:1321-1322
  • ErsatzTV.Infrastructure/Search/ElasticSearchIndex.cs:766-767
metadata.AlbumArtists ??= [];
metadata.Artists ??= [];

in UpdateSong. If the SongMetadata reached there came from a tracked query, indexing an untagged song sets the property, marks it Modified, and a later SaveChanges in the same scope silently rewrites NULL → [] across the table.

This may well be harmless today — it depends entirely on whether that path is AsNoTracking and whether anything saves in the same scope. That is exactly the point: it is currently unknown, and #691 established that "no test observes the column after a tracked read" means the existing suite cannot tell us. The guards themselves are also now redundant, since the read sites they were protecting are guarded at the point of use as of #691.

Second, smaller item

ErsatzTV.Application/Channels/Commands/RefreshChannelDataHandler.cs:610-611 assigns the raw nullable lists into the Scriban template context:

SongArtists = metadata.Artists
SongAlbumArtists = metadata.AlbumArtists

The assignment cannot throw, and the shipped _song.sbntxt only does array.join, so current output is unaffected. But a user's custom _song.sbntxt doing anything else with a null collection could fault on untagged songs. Worth guarding for consistency with the #691 sweep.

Done-when

  • Determine empirically whether the UpdateSong path holds a tracked SongMetadata (probe it; do not reason about it — reasoning is what got #691's first design wrong)
  • If tracked: remove the mutation, or make it provably safe
  • Decide whether the two ??= guards are now dead given #691's read-site guards, and remove them if so rather than leaving three contradictory stories about the same field
  • Guard the RefreshChannelDataHandler template fields
  • Adversarial review passed

Related

#691 (read-site guards), #671 / api.selection-projection-include-chain (the sweep-by-FIELD convention).

Box 3 — the issue's premise was wrong, recording it rather than ticking past it. This box assumed
the two ??= guards were "now redundant given #691's read-site guards". They were not: they were the
only thing protecting the four reads below them, and deleting them alone throws on every untagged song
(measured — NullReferenceException at the Lucene foreach; the Elastic .ToList() reads throw
ArgumentNullException). So they were not removed as dead; they were REPLACED by read-site guards that
move with their reads. Ticked as decided-and-acted-on, not as premise-confirmed.

Box 1 — the empirical answer is "not tracked today". Both feeds are AsNoTracking
(SearchRepository.GetItemToIndex, GetAllSongs), so no shipped caller loses data. The mutation was
removed regardless and the indexer's own contract pinned by test, because "today's two callers" is not
a property of the indexer — which is exactly what #691 recorded as a loaded gun.

Found by adversarial review of PR #700 (#691). **Pre-existing; not introduced by that PR**, which correctly left these alone rather than widening scope. ## Why this is worth a separate issue #691 originally fixed the nullable `SongMetadata.Artists`/`AlbumArtists` by making the entity coalesce on read. That design was rejected after a reviewer **probed the real `TvContext`**: with EF's default `PreferField` access mode, mutating the field during a read on a *tracked* entity flips the entity to `Modified`, and the next `SaveChanges` writes `[]` over the `NULL` column. Two places already do structurally the same thing, by assignment rather than by getter: - `ErsatzTV.Infrastructure/Search/LuceneSearchIndex.cs:1321-1322` - `ErsatzTV.Infrastructure/Search/ElasticSearchIndex.cs:766-767` ```csharp metadata.AlbumArtists ??= []; metadata.Artists ??= []; ``` in `UpdateSong`. If the `SongMetadata` reached there came from a tracked query, indexing an untagged song sets the property, marks it `Modified`, and a later `SaveChanges` in the same scope silently rewrites `NULL` → `[]` across the table. **This may well be harmless today** — it depends entirely on whether that path is `AsNoTracking` and whether anything saves in the same scope. That is exactly the point: it is currently unknown, and #691 established that "no test observes the column after a tracked read" means the existing suite cannot tell us. The guards themselves are also now redundant, since the read sites they were protecting are guarded at the point of use as of #691. ## Second, smaller item `ErsatzTV.Application/Channels/Commands/RefreshChannelDataHandler.cs:610-611` assigns the raw nullable lists into the Scriban template context: ```csharp SongArtists = metadata.Artists SongAlbumArtists = metadata.AlbumArtists ``` The assignment cannot throw, and the shipped `_song.sbntxt` only does `array.join`, so current output is unaffected. But a user's custom `_song.sbntxt` doing anything else with a null collection could fault on untagged songs. Worth guarding for consistency with the #691 sweep. ## Done-when - [x] Determine empirically whether the `UpdateSong` path holds a **tracked** `SongMetadata` (probe it; do not reason about it — reasoning is what got #691's first design wrong) - [x] If tracked: remove the mutation, or make it provably safe - [x] Decide whether the two `??=` guards are now dead given #691's read-site guards, and remove them if so rather than leaving three contradictory stories about the same field - [x] Guard the `RefreshChannelDataHandler` template fields - [x] Adversarial review passed ## Related #691 (read-site guards), #671 / `api.selection-projection-include-chain` (the sweep-by-FIELD convention). **Box 3 — the issue's premise was wrong, recording it rather than ticking past it.** This box assumed the two `??=` guards were "now redundant given #691's read-site guards". They were not: they were the only thing protecting the four reads below them, and deleting them alone throws on every untagged song (measured — `NullReferenceException` at the Lucene `foreach`; the Elastic `.ToList()` reads throw `ArgumentNullException`). So they were not removed as dead; they were REPLACED by read-site guards that move with their reads. Ticked as decided-and-acted-on, not as premise-confirmed. **Box 1 — the empirical answer is "not tracked today".** Both feeds are `AsNoTracking` (`SearchRepository.GetItemToIndex`, `GetAllSongs`), so no shipped caller loses data. The mutation was removed regardless and the indexer's own contract pinned by test, because "today's two callers" is not a property of the indexer — which is exactly what #691 recorded as a loaded gun.
timothy added the bugpriority: medium labels 2026-07-28 23:06:05 +02:00
timothy added the in-progress label 2026-08-22 23:31:55 +02:00
Author
Owner

Claiming this (Claude Code session, 2026-08-22). Pre-claim checks per process.parallel-session-claim: no open PR referencing #701 (only three Renovate PRs open), git ls-remote --heads origin '*701*' empty, no prior comments on the issue, fresh git fetch origin main.

Claiming this (Claude Code session, 2026-08-22). Pre-claim checks per `process.parallel-session-claim`: no open PR referencing #701 (only three Renovate PRs open), `git ls-remote --heads origin '*701*'` empty, no prior comments on the issue, fresh `git fetch origin main`.
Author
Owner

Closing record

Outcome: Shipped — PR #826, merged as 7966e0816. Both search indexers and RefreshChannelDataHandler no longer mutate SongMetadata.Artists/AlbumArtists; each site's reads moved to Optional(...).Flatten() locals with it. New fixture SongIndexerMetadataMutationTests pins the indexer's contract. New decision record media.nullable-primitive-collection-mutation.

Root cause: The ??= [] idiom is correct on navigation collections and appears 92 times across the app projects, so writing it two lines below metadata.Genres ??= [] looked right. The difference is invisible at the call site and lives in the model: Artists holds the whole list in ONE COLUMN, so the property IS the column value, and assigning it on a tracked entity flips the entry to Modified.

Decisions/conventions changed: Added media.nullable-primitive-collection-mutation. It complements api.selection-projection-include-chain (#671), which records the read-site guard; this one covers the write half — never assign the guard back onto the entity — and derives the collection-column population from the model configuration.

Reusable knowledge:

  1. The issue's premise on box 3 was wrong. The two ??= guards were not redundant given #691's read-site guards — they were the only thing protecting the four reads below them. Deleting them alone throws on every untagged song. Removal only works paired with a read-site guard.
  2. The exception type follows the read FORM, not the field. foreach over a null collection throws NullReferenceException; string.Join/Enumerable.ToList on a null source throw ArgumentNullException. This PR contains two of each (Lucene foreach, Elastic .ToList()), so no single exception-name grep characterises the class — which is the real reason a sweep must follow the FIELD. A clause asserting the ANE half only was inherited verbatim from #671, where it is correct, and was false here for the primary sites until measured.
  3. Deriving a collection-column population from the domain classes returns 2 of 8. IList<string> under ErsatzTV.Core/Domain misses the six value-converted collections (ProgramScheduleAlternate/PlayoutTemplate × DaysOfMonth, MonthsOfYear, DaysOfWeek), which are declared as plain ICollection<T> and become single columns only in Data/Configurations, and misses enum collections entirely. Authoritative source is the model configuration. Storage also differs — comma-separated text for the int converter, JSON for the enum one — so "JSON array" is wrong as a class-wide description.
  4. A test whose every assertion says "nothing changed" needs a positive control. This one passed with the code under test unreachable until writer.NumDocs.ShouldBe(1) was added; verified both ways.
  5. Phrase-grepping for retracted wording does not close a sweep. Three consecutive review rounds fixed one site and left its twin, because each searched for the previously challenged phrase. What closed it was enumerating every assertive clause and asking what would falsify each — 34 clauses, one experiment, one surviving falsehood (knowledge item 2).

Verification: ErsatzTV.Tests 2006 passed / 4 pre-existing skips · Core.Tests 685 / 1 skip · Infrastructure.Tests 114 · Architecture.Tests 7 · Scanner.Tests 1504 — 0 failures. scripts/tests 874 passed / 2 skipped. Format gate + BOM clean on all four touched .cs. CI green on b5dee2620, Build & test (.NET) 524s (a real run, not the docs-only fast path). Mutation proof witnessed: restoring only the ??= [] clause from the real predecessor file reddens the fixture; a probe variant reports STATE=Modified and the raw column moving NULL → "[]". Two independent cold reviews (Codex GPT-5.6 cross-family; Claude Opus 5 high, worktree-isolated) — C# cleared by both in round 1; four subsequent rounds were all evidence defects.

Deferred:

  • #823 — whether a runtime null is reachable on the six scheduling collections. Deliberately not asserted either way: no ??= exists on any of them, and the only constructors of those commands today are the two API request records, which normalize with ?? [].
  • #824 — ElasticSearchIndex.UpdateSong has no regression test (needs a stubbed transport). Its change here is by inspection against the Lucene one. A source-text guard was considered and rejected as brittle.

Docs updated: docs/decisions/records/media/nullable-primitive-collection-mutation.md (new) + docs/decisions/README.md (regenerated). No route or /api/* change, so blazor-route-parity.md / api-conventions.md / endpoint-index.md are not affected.

## Closing record **Outcome:** Shipped — PR #826, merged as `7966e0816`. Both search indexers and `RefreshChannelDataHandler` no longer mutate `SongMetadata.Artists`/`AlbumArtists`; each site's reads moved to `Optional(...).Flatten()` locals with it. New fixture `SongIndexerMetadataMutationTests` pins the indexer's contract. New decision record `media.nullable-primitive-collection-mutation`. **Root cause:** The `??= []` idiom is *correct* on navigation collections and appears 92 times across the app projects, so writing it two lines below `metadata.Genres ??= []` looked right. The difference is invisible at the call site and lives in the model: `Artists` holds the whole list in ONE COLUMN, so the property IS the column value, and assigning it on a tracked entity flips the entry to `Modified`. **Decisions/conventions changed:** Added `media.nullable-primitive-collection-mutation`. It complements `api.selection-projection-include-chain` (#671), which records the read-site guard; this one covers the write half — never assign the guard back onto the entity — and derives the collection-column population from the model configuration. **Reusable knowledge:** 1. **The issue's premise on box 3 was wrong.** The two `??=` guards were *not* redundant given #691's read-site guards — they were the only thing protecting the four reads below them. Deleting them alone throws on every untagged song. Removal only works paired with a read-site guard. 2. **The exception type follows the read FORM, not the field.** `foreach` over a null collection throws `NullReferenceException`; `string.Join`/`Enumerable.ToList` on a null *source* throw `ArgumentNullException`. This PR contains two of each (Lucene `foreach`, Elastic `.ToList()`), so no single exception-name grep characterises the class — which is the real reason a sweep must follow the FIELD. A clause asserting the ANE half only was inherited verbatim from #671, where it is correct, and was false here for the primary sites until measured. 3. **Deriving a collection-column population from the domain classes returns 2 of 8.** `IList<string>` under `ErsatzTV.Core/Domain` misses the six value-converted collections (`ProgramScheduleAlternate`/`PlayoutTemplate` × `DaysOfMonth`, `MonthsOfYear`, `DaysOfWeek`), which are declared as plain `ICollection<T>` and become single columns only in `Data/Configurations`, and misses enum collections entirely. Authoritative source is the model configuration. Storage also differs — comma-separated text for the int converter, JSON for the enum one — so "JSON array" is wrong as a class-wide description. 4. **A test whose every assertion says "nothing changed" needs a positive control.** This one passed with the code under test unreachable until `writer.NumDocs.ShouldBe(1)` was added; verified both ways. 5. **Phrase-grepping for retracted wording does not close a sweep.** Three consecutive review rounds fixed one site and left its twin, because each searched for the *previously challenged* phrase. What closed it was enumerating every assertive clause and asking what would falsify each — 34 clauses, one experiment, one surviving falsehood (knowledge item 2). **Verification:** `ErsatzTV.Tests` 2006 passed / 4 pre-existing skips · `Core.Tests` 685 / 1 skip · `Infrastructure.Tests` 114 · `Architecture.Tests` 7 · `Scanner.Tests` 1504 — 0 failures. `scripts/tests` 874 passed / 2 skipped. Format gate + BOM clean on all four touched `.cs`. CI green on `b5dee2620`, `Build & test (.NET)` **524s** (a real run, not the docs-only fast path). Mutation proof witnessed: restoring only the `??= []` clause from the real predecessor file reddens the fixture; a probe variant reports `STATE=Modified` and the raw column moving `NULL` → `"[]"`. Two independent cold reviews (Codex GPT-5.6 cross-family; Claude Opus 5 `high`, worktree-isolated) — C# cleared by both in round 1; four subsequent rounds were all evidence defects. **Deferred:** - #823 — whether a runtime null is reachable on the six scheduling collections. Deliberately *not* asserted either way: no `??=` exists on any of them, and the only constructors of those commands today are the two API request records, which normalize with `?? []`. - #824 — `ElasticSearchIndex.UpdateSong` has no regression test (needs a stubbed transport). Its change here is by inspection against the Lucene one. A source-text guard was considered and rejected as brittle. **Docs updated:** `docs/decisions/records/media/nullable-primitive-collection-mutation.md` (new) + `docs/decisions/README.md` (regenerated). No route or `/api/*` change, so `blazor-route-parity.md` / `api-conventions.md` / `endpoint-index.md` are not affected.
timothy removed the in-progress label 2026-08-23 01:25:37 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#701