Rebasing onto #612 exposed a silent gap rather than a conflict. #612 added a
`projectionFailureCount` parameter to MediaServerReconciliationGuard.ShouldFlagMissing that
refuses the sweep when the enumeration reported swallowed projection exceptions — but the
parameter is OPTIONAL with a default of 0, so this scanner compiled unchanged while opting
out of the protection entirely. ProjectToMusicVideo has exactly the swallowing catch #484
exists to defend against, so the music-video sweep would have been the only one unguarded.
- MediaServerMusicVideoLibraryScanner creates one MediaServerProjectionFailureCounter per
enumeration (never a field on the singleton api client, so concurrent scans of different
libraries can't leak failures into each other's sweep decision), passes it to
GetMusicVideoLibraryItems, and feeds its Count to ShouldFlagMissing.
- The single guard call also gates the #496 legacy path diff, which is more exposed: a
legacy row has no etag to fall back on.
- ProjectToMusicVideo now returns MediaServerProjectionResult<JellyfinMusicVideo>, combining
#612's Skipped/Failed distinction with #496's identity type.
- main's own #484 music-video test targeted the pre-#496 hard-delete scanner (FindMusicVideoPaths
/DeleteByPath) and no longer applies; it is replaced by two tests in the new architecture
covering the identity sweep and the legacy sweep. Both proven non-vacuous — removing
projectionFailures.Count from the guard call fails exactly those two.
4,277 tests green; format/BOM clean; decisions validator OK.
Independent cold review (Codex) returned BLOCKED. Findings 1, 3 and 4 are fixed here;
each has a regression test proven non-vacuous by a negative control.
1. Blocker — the replaced local path was discarded. The scanner computed localPath but
GetOrAdd only received `incoming`, so the repository re-derived the path from the
UNREPLACED projection. On any install with path replacements, adoption hashed the
server-side path, missed the existing row, ALSO slipped past MediaFileAlreadyExists
(which hashes that same wrong string) and inserted a duplicate row under a server path,
leaving the original collection-linked row identity-less forever. The test harness hid
this because its path-replacement stub was an identity function.
→ GetOrAdd now takes localPath explicitly and never reads the projection's path;
BuildPathReplacement takes a real mapping and the new test genuinely replaces.
3. Medium — GetByItemId matched on ItemId alone, so two media sources presenting the same
item id (cloned Jellyfin DB) resolved to each other's row, letting one library repoint
another's. → filtered by LibraryPath.LibraryId.
4. Medium — a row predating the identity that the server had ALREADY stopped reporting was
never adopted (adoption only runs for an incoming item) and carried no identity, so the
itemId diff could not see it either: it sat Normal and schedulable forever, strictly
worse than the hard delete it replaced. → GetExistingLegacyMusicVideoPaths +
FlagFileNotFoundByPaths reconcile legacy rows by local path, and they are counted into
the #477 empty-fetch guard (on the first scan after this ships they ARE the whole
library, so a guard counting only identity rows would sweep all of them on a transient
empty fetch).
Finding 2 (the issue's Done-when #2) is a scope question, not a defect, and is unchanged:
one file path is still one MediaItem row globally, so this lands music videos at parity
with movies rather than eliminating shared-row trashing. Recorded honestly in the decision
record; raised for an explicit call before the issue is closed.
fixes#496
Music videos carried no server identity, so JellyfinMusicVideoLibraryScanner had to
reconcile by a (LibraryPathId, path) diff and HARD-delete the remainder. A file served
by two libraries with overlapping local paths is a single row owned by whichever library
scanned it first, so that owner's sweep destroyed a row another library still served —
taking collection membership and playout references with it, irreversibly.
This is #494's deferred "option 2":
- New JellyfinMusicVideo : MusicVideo (ItemId/Etag), mirroring JellyfinMovie — TPT table,
varchar(36), ItemId index. Dual-provider migration Add_JellyfinMusicVideo.
- New IMediaServerMusicVideoRepository + JellyfinMusicVideoRepository: itemId-keyed
existing-set/lookup and Flag{Normal,Unavailable,FileNotFound} seams, all scoped per
library via LibraryPath.LibraryId.
- New MediaServerMusicVideoLibraryScanner base; JellyfinMusicVideoLibraryScanner folds
onto it and keeps the #177/#488/#497/#500 metadata-reconcile logic verbatim.
- The sweep now soft-trashes (FileNotFound) instead of deleting, so removal is reversible
and EmptyTrash-governed. DeleteEmptyArtists consequently no longer fires from a sweep.
- Pre-identity rows are ADOPTED in place: the identity row is inserted against the same
MediaItem id, scoped to the scanned library's own LibraryPath, so collection membership
survives and a local/second-library row is never hijacked.
- AddMusicVideo normalizes Path/PathHash to the path-REPLACED local path; the projection
fills them from the server-reported path, which would break every later PathHash lookup.
Docs: scan.musicvideo-reconciliation relocated to docs/decisions/archive/scan.md as
superseded; new active record scan.musicvideo-server-identity.
fixes#496
Review finding 1 (blocking). ScanSeasons' FlagFileNotFoundSeasons and ScanEpisodes'
FlagFileNotFoundEpisodes had no guard at all — neither #477's nor #484's — so ProjectToSeason /
ProjectToEpisode returning Failed() was computed and discarded.
#477 scoped those out because "the blast radius is one show's seasons / one season's episodes",
which holds for a per-parent EMPTY fetch but not for a projection failure: that is systematic by
construction. One bad code path fires on every parent, so every season enumerates zero episodes,
existing.Except([]) is the whole episode library, and EmptyTrashHandler deletes it permanently.
Threads the counter into GetSeasonLibraryItems / GetEpisodeLibraryItems(WithoutPeople) for
Jellyfin and Emby using the same optional-trailing-param shape, and guards both sweeps with
MediaServerReconciliationGuard.ShouldFlagMissingDescendants — the same class and the same private
failure predicate as ShouldFlagMissing, deliberately WITHOUT #477's empty-fetch branch so
per-parent empty behaviour (and #476's cascade, which depends on it) is unchanged.
Also from the review:
- finding 3: tests now pin the same-instance JOIN at every level (movie, show, season, episode,
music video) by driving the real ScanLibrary entry point and recording the failure from inside
the enumeration, so a refactor handing the api client a fresh counter goes red.
- finding 4: the missing-library Failed() branch is documented as defensive and unreachable.
- finding 2: the mass-Skip residual (Emby's response-shape-dependent MediaSources guard, Plex's
pre-projection filter) is stated as a known limitation in the decision record.
- finding 5: the log-contract change (only the #484 message when both refusals apply) is noted.
fixes#484
Extends MediaServerReconciliationGuard (#477) with a second deterministic refusal: when the
enumeration that produced the incoming set silently dropped items whose projection THREW, the
file-not-found sweep is refused. A dropped item the server did return is indistinguishable
from a deletion at the reconcile step, so a projection regression could otherwise mass-flag a
healthy library FileNotFound (which EmptyTrash then deletes permanently).
Deliberate guard-clause skips (STRM files, virtual items, unsupported types) are explicitly NOT
failures and never suppress a sweep — counting them would permanently disable reconciliation for
any library holding a single STRM file.
The ratio / missing-fraction threshold is REJECTED, not deferred: it is a two-sided heuristic
with no tunable default and no telemetry, and the failure it approximates is exactly observable
via the projection-failure count (a genuine bulk deletion produces zero failures).
Seam is deliberately narrow — the private ProjectTo* contract inside each api client changed from
Option<T> to MediaServerProjectionResult<T> (projected/skipped/failed), the paged helper counts
IsFailure in one place, and the scanner reads it through an optional trailing
MediaServerProjectionFailureCounter on only the five library-level methods that feed a sweep.
The counter is per-enumeration state created by the scanner, never a field on an api client.
fixes#484
The remove-stale + add-new reconcile idiom materializes its add set with
.ToList() BEFORE the loop mutates the existing collection, so the add filter
(`incoming.All(x2 => x2.Name != x.Name)`) is evaluated against a snapshot. Two
identically-named incoming entries whose name is not yet on the existing item
therefore BOTH passed the filter and BOTH inserted — a duplicate row.
Deduplicate the incoming set on the same key the filter compares (Name; Guid
for Guids), in both copies of the idiom:
- PlexMovieLibraryScanner.UpdateMetadata (the original) — genres, studios,
actors, directors, writers, guids, tags.
- JellyfinMusicVideoLibraryScanner.Reconcile{Genres,Tags,Studios,Artists}
(added in #497, mirrors the Plex pattern verbatim).
Plex ACTORS are the exception and get an artwork-preferring dedup hoisted out
and shared with the remove filter, because that filter is keyed on
(Name, artwork-presence) — it is the mechanism that drops an artwork-less actor
so the add loop can re-add it WITH artwork. A bare DistinctBy(a => a.Name)
there keeps the FIRST duplicate, so Plex listing the artwork-less copy first
discarded the artwork; worse, the remove filter would still see the
artwork-less duplicate, making its upgrade clause false, so the stale row was
never removed and the artwork never arrived on ANY later scan either. Actor
also carries Role/Order, which first-wins would silently drop too. Caught by
the cold review of the first version of this commit.
For the Jellyfin scanner the dedup sits at the incoming-list declaration, which
also covers the remove filter — safe because all four of those filters only ask
"is this name present at all", an answer duplicates cannot change.
Tests: duplicate-collapse for both paths, the two Actors cases above, and a
POSITIVE CONTROL proving distinct entries are still all added and stale ones
still removed (without it, a mis-keyed dedup that collapsed genuinely different
entries would pass every other assertion). Each proven non-vacuous.
The Plex tests drive the protected UpdateMetadata through a minimal test-only
subclass, as MediaServerMovieLibraryScannerTests already does.
Low likelihood in practice (a media server emitting two identically-named
genres for one item is unusual); this is defensive, with no observed occurrence.
The same idiom is copied into 8 further scanners/repositories that this change
deliberately does not touch (the issue scoped it to two paths) — filed as #600
so the class of bug is tracked rather than silently left in the majority of its
instances. The dedup rule is recorded under scan.musicvideo-reconciliation.
fixes#500
Jellyfin-sourced music videos rendered weaker MTV-style credits than local
NFO libraries: the Scriban credits templates expose Album/Track, and
MusicVideoNfoReader has always mapped both, but the Jellyfin projection
never did. ChronologicalMediaComparer orders music videos by the same two
fields, so they were also ordering worse.
Verified against the live server (1437 MusicVideo items): Album comes back
on 111 and IndexNumber on 4, both WITHOUT being named in the `fields` query
param -- Album is a plain BaseItemDto property, not an ItemFields value, so
no Refit `fields` change is needed (and adding one would be wrong).
ParentIndexNumber is deliberately NOT used for Track: on live data, where
both are present ParentIndexNumber is 1 while IndexNumber carries the real
ordinal, and where only ParentIndexNumber is present it is a collection/disc
grouping that tracks the Album ("Glastonbury: 2022" -> 230).
The fix is two layers, not one. The projection alone would only ever reach
music videos ADDED after it -- UpdateMetadata copies scalars field by field,
so an existing item whose album/track is set or corrected in Jellyfin would
keep a stale value forever. That is the same class of bug #497 fixed for
child collections, one layer up.
Also strips a pre-existing UTF-8 BOM from JellyfinLibraryItemResponse.cs,
which the format gate flags once the file is touched (format-as-you-touch).
fixes#177
JellyfinMusicVideoLibraryScanner.UpdateMetadata copied only scalar fields
for an EXISTING music video, so genres/tags/studios/artists edited in
Jellyfin never reached ErsatzTV — the update path silently dropped every
child collection (only the Add path ever persisted them). Root cause is
inherited from upstream: unlike the movie/episode Jellyfin path, which
reconciles collections inside the tracked repository GetOrAdd,
MusicVideoRepository.GetOrAdd is AsNoTracking and the scanner never
reconciled the collections itself.
Fix mirrors PlexMovieLibraryScanner.UpdateMetadata's remove-stale +
add-new idiom, reconciling exactly the collections that BOTH the Add path
persists AND GetOrAdd eager-loads: Genres, Tags, Studios, Artists. Guids
(add-persisted but not eager-loaded — would duplicate) and Directors
(eager-loaded but not add-persisted for music videos) are deliberately
out of scope.
Movie/Episode paths do NOT have this gap (they reconcile in the tracked
repo GetOrAdd), so no separate fix is needed there.
Test is an interaction test (substituted repos, canned existing item)
verifying the exact reconcile calls; proven non-vacuous. The real-DB
double-scan approach can't drive this: the in-memory harness shares one
SQLite connection across contexts and mid-scan GetOrAdd's First()-nav
path predicate mis-resolves once the existing item carries metadata
children — a harness-only quirk (prod uses per-context pooled
connections).
fixes#497
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(494): reconcile removed music videos in Jellyfin scanner (#495)
JellyfinMusicVideoLibraryScanner did add/update only, so a music video or
artist deleted on the Jellyfin side lingered forever. Add a library-scoped
trash sweep (TrashMissingMusicVideos) gated by the #477 empty-fetch guard.
fixes#494
Co-authored-by: Timothy <timothy.look@gmail.com>
Co-committed-by: Timothy <timothy.look@gmail.com>
A successful fetch returning zero items made existing.Except([]) flag the ENTIRE
library FileNotFound in one scan — feeding EmptyTrashHandler's permanent delete
and emptying every affected collection (dead channels). Add a shared
MediaServerReconciliationGuard that skips (and logs a Warning) the sweep when
incoming==0 while items exist, wired into the three library-level sweeps
(Television shows / Movie / OtherVideo).
An empty incoming set is indistinguishable at scan time from a mid-restore /
emptied-upstream error (both report a zero total), so this deliberately overrides
#476's degenerate "last item removed => empty incoming => flag" case. #476's
cascade still fires for partial deletions (survivors present); its characterization
test moves from an empty incoming to a survivor+removed partial-deletion case.
Tests: policy table (MediaServerReconciliationGuardTests) + per-scanner integration
proving the wiring (empty incoming + non-empty existing flags/reindexes nothing).
Proven non-vacuous by neutralizing the guard. Nested TV season/episode sweeps left
unguarded (bounded blast radius); ratio-threshold + projection-failure detection
deferred to a follow-up. docs/decisions.md updated.
Fixes#477
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The media-server television scanner reconciles removed items at three levels
via existing.Except(incoming), but the season and episode sweeps live inside
the per-parent loops (ScanSeasons inside the show loop, ScanEpisodes inside the
season loop). Those loops only iterate parents present in the incoming list, so
a show (or season) that is gone from the media server is never visited and its
descendants are never swept — they keep their last state. On Jellyfin/Emby that
is RemoteOnly, which PlayoutBuilder does NOT skip even with PlayoutSkipMissingItems
on, so every orphaned episode keeps getting scheduled as a guaranteed tune-in
failure (the #473 reproduction; 717 stale prod rows across 10 removed shows).
Fix: cascade the flag by parent MediaItem.Id. Two provider-agnostic repo helpers
(Season.ShowId / Episode.SeasonId are on the base tables) flag descendants and the
scanner drives them after each parent sweep — show → seasons → episodes, and
season → episodes for the show-present case. Shared abstract base fixes
Jellyfin/Plex/Emby at once; Movie/OtherVideo are flat and have no such gap.
Tests: a Scanner.Tests case asserts the scanner cascades through the (substituted)
repository (non-vacuous — fails if the cascade calls are removed), and Integration
tests exercise the real cascade SQL against the schema, proving it flags only the
targeted subtree and no-ops on empty input.
fixes#476
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* move plex artwork removal to its own repository
* clean up old local movie artwork
* clien up old music video/artist artwork
* clean up old remote stream artwork
* clean up old song artwork
* clean up old show artwork; properly update season artwork
* adds .aiff to supported audio file extension in local folder scanner
* add support for aif, aifc, aiff
---------
Co-authored-by: Jason Dove <1695733+jasongdove@users.noreply.github.com>
* add ability to deep scan just a single tv show for Plex, Emby, and Jellyfin
Including "/api/libraries/{id:int}/scan-show" REST API endpoint to
trigger.
* restrict plex search results to the intended library
* restrict scanning to media server libraries that are marked to sync with etv
* fix previous commit
* also guard library scan api
* add scan buttons to show ui
* scan single plex show by id
* scan jellyfin and emby single shows by id
* update changelog
---------
Co-authored-by: Jeff Slutter <MrMustard@gmail.com>
Co-authored-by: Jason Dove <1695733+jasongdove@users.noreply.github.com>
* add external chapter file scanning
Support Matroska chapter xml files next to media file with extension .xml or .chapters
* only update chapters in db
---------
Co-authored-by: Jeff Slutter <MrMustard@gmail.com>
Co-authored-by: Jason Dove <1695733+jasongdove@users.noreply.github.com>
* fix validation in new form layout
* pin mediatr to last oss version
* update dependencies
* cleanup code in core
* cleanup code in ffmpeg
* cleanup code in infra
* cleanup code in scanner
* cleanup code in application
* cleanup main code
* cleanup test code
* solution-wide code cleanup
* initial plumbing
* scan for plex networks
* save network contents to db as tags
* eliminate network tag id churn
* add network and show_network to search index
* update last networks scan
* show networks on tv show page
* update changelog
timothy
