SQLite's LOWER() folds ASCII only -- lower('Édith') is 'Édith' unchanged --
so the EF-sourced facet fields UNDER-matched any stored value whose prefix
carries an uppercase non-ASCII character. An under-match is unrecoverable:
no later stage can reintroduce a row SQL never returned.
Adds a SECOND, ADDITIVE query taken only when the provider is SQLite and q
contains a non-ASCII character: raw Dapper SQL folding through etv_upper(),
a SqliteConnection.CreateFunction scalar implementing ToUpperInvariant.
Every other case -- all-ASCII q, and MySQL for all q -- runs the existing
EF query byte-identically.
MySQL needed no change and gets none: verified on MySQL 8.4 that its LOWER()
is Unicode-aware and its ci collation makes the predicate OVER-match, which
the existing ordinal filter already discards.
The fold is ToUpperInvariant because OrdinalIgnoreCase equality is a strict
SUBSET of invariant-uppercase equality, so the SQL stage yields a superset of
the final filter's matches and can never under-match. Note OrdinalIgnoreCase
is NOT "invariant-upper then ordinal": ToUpperInvariant('ſ') is 'S', yet
"ſweet".StartsWith("S", OrdinalIgnoreCase) is false. Tests pin that.
No migration, no model change; both provider snapshots are untouched.
Refs #668
Decisions-Edit: yes
Review of 491f5099 found the fix inert in the only process that runs it, plus
a MySQL collation defect in the lookup.
B1 — TvContext.IsUniqueConstraintViolation was assigned only in ErsatzTV/
Startup.cs, but ErsatzTV.Scanner is a separate executable and every production
caller of GetOrAddFolder/SetEtag lives there. The classifier kept its '_ =>
false' default, so the catch never ran and the DbUpdateException failed the
whole scan - worse than the duplicate row it replaced. Wire both provider
branches in ErsatzTV.Scanner/Program.cs, and add ProviderStaticsWiringTests
(architecture) asserting the scanner assigns every TvContext static the host
assigns, with IsSqlite documented as the one exemption.
H1 — GetFolder's 'Path == folder' is case-insensitive on MySQL while PathHash
is case-sensitive, and FirstOrDefault was unordered: a scan of '/x/foo' could
resolve the '/x/Foo' row and stamp the wrong hash onto it (verified on MySQL
8.4: the WHERE matches both, LIMIT 1 returns the wrong one). Treat the SQL
equality as a narrowing filter, order by Id, and settle identity ordinally.
Route the heal through EF and drop a classified violation, so an opportunistic
maintenance write can never abort a scan.
Also: run the dedupe migration test with foreign keys ON (matching prod), clear
the keeper's etag, null out a self-parent, and document the cleanup's limits
(NULL paths excluded, Down does not restore deleted rows, CI's fresh-DB apply
covers none of the data mutation).
Refs #488#308fix#491
Rebasing onto #612 exposed a silent gap rather than a conflict. #612 added a
`projectionFailureCount` parameter to MediaServerReconciliationGuard.ShouldFlagMissing that
refuses the sweep when the enumeration reported swallowed projection exceptions — but the
parameter is OPTIONAL with a default of 0, so this scanner compiled unchanged while opting
out of the protection entirely. ProjectToMusicVideo has exactly the swallowing catch #484
exists to defend against, so the music-video sweep would have been the only one unguarded.
- MediaServerMusicVideoLibraryScanner creates one MediaServerProjectionFailureCounter per
enumeration (never a field on the singleton api client, so concurrent scans of different
libraries can't leak failures into each other's sweep decision), passes it to
GetMusicVideoLibraryItems, and feeds its Count to ShouldFlagMissing.
- The single guard call also gates the #496 legacy path diff, which is more exposed: a
legacy row has no etag to fall back on.
- ProjectToMusicVideo now returns MediaServerProjectionResult<JellyfinMusicVideo>, combining
#612's Skipped/Failed distinction with #496's identity type.
- main's own #484 music-video test targeted the pre-#496 hard-delete scanner (FindMusicVideoPaths
/DeleteByPath) and no longer applies; it is replaced by two tests in the new architecture
covering the identity sweep and the legacy sweep. Both proven non-vacuous — removing
projectionFailures.Count from the guard call fails exactly those two.
4,277 tests green; format/BOM clean; decisions validator OK.
Independent cold review (Codex) returned BLOCKED. Findings 1, 3 and 4 are fixed here;
each has a regression test proven non-vacuous by a negative control.
1. Blocker — the replaced local path was discarded. The scanner computed localPath but
GetOrAdd only received `incoming`, so the repository re-derived the path from the
UNREPLACED projection. On any install with path replacements, adoption hashed the
server-side path, missed the existing row, ALSO slipped past MediaFileAlreadyExists
(which hashes that same wrong string) and inserted a duplicate row under a server path,
leaving the original collection-linked row identity-less forever. The test harness hid
this because its path-replacement stub was an identity function.
→ GetOrAdd now takes localPath explicitly and never reads the projection's path;
BuildPathReplacement takes a real mapping and the new test genuinely replaces.
3. Medium — GetByItemId matched on ItemId alone, so two media sources presenting the same
item id (cloned Jellyfin DB) resolved to each other's row, letting one library repoint
another's. → filtered by LibraryPath.LibraryId.
4. Medium — a row predating the identity that the server had ALREADY stopped reporting was
never adopted (adoption only runs for an incoming item) and carried no identity, so the
itemId diff could not see it either: it sat Normal and schedulable forever, strictly
worse than the hard delete it replaced. → GetExistingLegacyMusicVideoPaths +
FlagFileNotFoundByPaths reconcile legacy rows by local path, and they are counted into
the #477 empty-fetch guard (on the first scan after this ships they ARE the whole
library, so a guard counting only identity rows would sweep all of them on a transient
empty fetch).
Finding 2 (the issue's Done-when #2) is a scope question, not a defect, and is unchanged:
one file path is still one MediaItem row globally, so this lands music videos at parity
with movies rather than eliminating shared-row trashing. Recorded honestly in the decision
record; raised for an explicit call before the issue is closed.
fixes#496
Music videos carried no server identity, so JellyfinMusicVideoLibraryScanner had to
reconcile by a (LibraryPathId, path) diff and HARD-delete the remainder. A file served
by two libraries with overlapping local paths is a single row owned by whichever library
scanned it first, so that owner's sweep destroyed a row another library still served —
taking collection membership and playout references with it, irreversibly.
This is #494's deferred "option 2":
- New JellyfinMusicVideo : MusicVideo (ItemId/Etag), mirroring JellyfinMovie — TPT table,
varchar(36), ItemId index. Dual-provider migration Add_JellyfinMusicVideo.
- New IMediaServerMusicVideoRepository + JellyfinMusicVideoRepository: itemId-keyed
existing-set/lookup and Flag{Normal,Unavailable,FileNotFound} seams, all scoped per
library via LibraryPath.LibraryId.
- New MediaServerMusicVideoLibraryScanner base; JellyfinMusicVideoLibraryScanner folds
onto it and keeps the #177/#488/#497/#500 metadata-reconcile logic verbatim.
- The sweep now soft-trashes (FileNotFound) instead of deleting, so removal is reversible
and EmptyTrash-governed. DeleteEmptyArtists consequently no longer fires from a sweep.
- Pre-identity rows are ADOPTED in place: the identity row is inserted against the same
MediaItem id, scoped to the scanned library's own LibraryPath, so collection membership
survives and a local/second-library row is never hijacked.
- AddMusicVideo normalizes Path/PathHash to the path-REPLACED local path; the projection
fills them from the server-reported path, which would break every later PathHash lookup.
Docs: scan.musicvideo-reconciliation relocated to docs/decisions/archive/scan.md as
superseded; new active record scan.musicvideo-server-identity.
fixes#496
Review finding 1 (blocking). ScanSeasons' FlagFileNotFoundSeasons and ScanEpisodes'
FlagFileNotFoundEpisodes had no guard at all — neither #477's nor #484's — so ProjectToSeason /
ProjectToEpisode returning Failed() was computed and discarded.
#477 scoped those out because "the blast radius is one show's seasons / one season's episodes",
which holds for a per-parent EMPTY fetch but not for a projection failure: that is systematic by
construction. One bad code path fires on every parent, so every season enumerates zero episodes,
existing.Except([]) is the whole episode library, and EmptyTrashHandler deletes it permanently.
Threads the counter into GetSeasonLibraryItems / GetEpisodeLibraryItems(WithoutPeople) for
Jellyfin and Emby using the same optional-trailing-param shape, and guards both sweeps with
MediaServerReconciliationGuard.ShouldFlagMissingDescendants — the same class and the same private
failure predicate as ShouldFlagMissing, deliberately WITHOUT #477's empty-fetch branch so
per-parent empty behaviour (and #476's cascade, which depends on it) is unchanged.
Also from the review:
- finding 3: tests now pin the same-instance JOIN at every level (movie, show, season, episode,
music video) by driving the real ScanLibrary entry point and recording the failure from inside
the enumeration, so a refactor handing the api client a fresh counter goes red.
- finding 4: the missing-library Failed() branch is documented as defensive and unreachable.
- finding 2: the mass-Skip residual (Emby's response-shape-dependent MediaSources guard, Plex's
pre-projection filter) is stated as a known limitation in the decision record.
- finding 5: the log-contract change (only the #484 message when both refusals apply) is noted.
fixes#484
Extends MediaServerReconciliationGuard (#477) with a second deterministic refusal: when the
enumeration that produced the incoming set silently dropped items whose projection THREW, the
file-not-found sweep is refused. A dropped item the server did return is indistinguishable
from a deletion at the reconcile step, so a projection regression could otherwise mass-flag a
healthy library FileNotFound (which EmptyTrash then deletes permanently).
Deliberate guard-clause skips (STRM files, virtual items, unsupported types) are explicitly NOT
failures and never suppress a sweep — counting them would permanently disable reconciliation for
any library holding a single STRM file.
The ratio / missing-fraction threshold is REJECTED, not deferred: it is a two-sided heuristic
with no tunable default and no telemetry, and the failure it approximates is exactly observable
via the projection-failure count (a genuine bulk deletion produces zero failures).
Seam is deliberately narrow — the private ProjectTo* contract inside each api client changed from
Option<T> to MediaServerProjectionResult<T> (projected/skipped/failed), the paged helper counts
IsFailure in one place, and the scanner reads it through an optional trailing
MediaServerProjectionFailureCounter on only the five library-level methods that feed a sweep.
The counter is per-enumeration state created by the scanner, never a field on an api client.
fixes#484
The remove-stale + add-new reconcile idiom materializes its add set with
.ToList() BEFORE the loop mutates the existing collection, so the add filter
(`incoming.All(x2 => x2.Name != x.Name)`) is evaluated against a snapshot. Two
identically-named incoming entries whose name is not yet on the existing item
therefore BOTH passed the filter and BOTH inserted — a duplicate row.
Deduplicate the incoming set on the same key the filter compares (Name; Guid
for Guids), in both copies of the idiom:
- PlexMovieLibraryScanner.UpdateMetadata (the original) — genres, studios,
actors, directors, writers, guids, tags.
- JellyfinMusicVideoLibraryScanner.Reconcile{Genres,Tags,Studios,Artists}
(added in #497, mirrors the Plex pattern verbatim).
Plex ACTORS are the exception and get an artwork-preferring dedup hoisted out
and shared with the remove filter, because that filter is keyed on
(Name, artwork-presence) — it is the mechanism that drops an artwork-less actor
so the add loop can re-add it WITH artwork. A bare DistinctBy(a => a.Name)
there keeps the FIRST duplicate, so Plex listing the artwork-less copy first
discarded the artwork; worse, the remove filter would still see the
artwork-less duplicate, making its upgrade clause false, so the stale row was
never removed and the artwork never arrived on ANY later scan either. Actor
also carries Role/Order, which first-wins would silently drop too. Caught by
the cold review of the first version of this commit.
For the Jellyfin scanner the dedup sits at the incoming-list declaration, which
also covers the remove filter — safe because all four of those filters only ask
"is this name present at all", an answer duplicates cannot change.
Tests: duplicate-collapse for both paths, the two Actors cases above, and a
POSITIVE CONTROL proving distinct entries are still all added and stale ones
still removed (without it, a mis-keyed dedup that collapsed genuinely different
entries would pass every other assertion). Each proven non-vacuous.
The Plex tests drive the protected UpdateMetadata through a minimal test-only
subclass, as MediaServerMovieLibraryScannerTests already does.
Low likelihood in practice (a media server emitting two identically-named
genres for one item is unusual); this is defensive, with no observed occurrence.
The same idiom is copied into 8 further scanners/repositories that this change
deliberately does not touch (the issue scoped it to two paths) — filed as #600
so the class of bug is tracked rather than silently left in the majority of its
instances. The dedup rule is recorded under scan.musicvideo-reconciliation.
fixes#500
Jellyfin-sourced music videos rendered weaker MTV-style credits than local
NFO libraries: the Scriban credits templates expose Album/Track, and
MusicVideoNfoReader has always mapped both, but the Jellyfin projection
never did. ChronologicalMediaComparer orders music videos by the same two
fields, so they were also ordering worse.
Verified against the live server (1437 MusicVideo items): Album comes back
on 111 and IndexNumber on 4, both WITHOUT being named in the `fields` query
param -- Album is a plain BaseItemDto property, not an ItemFields value, so
no Refit `fields` change is needed (and adding one would be wrong).
ParentIndexNumber is deliberately NOT used for Track: on live data, where
both are present ParentIndexNumber is 1 while IndexNumber carries the real
ordinal, and where only ParentIndexNumber is present it is a collection/disc
grouping that tracks the Album ("Glastonbury: 2022" -> 230).
The fix is two layers, not one. The projection alone would only ever reach
music videos ADDED after it -- UpdateMetadata copies scalars field by field,
so an existing item whose album/track is set or corrected in Jellyfin would
keep a stale value forever. That is the same class of bug #497 fixed for
child collections, one layer up.
Also strips a pre-existing UTF-8 BOM from JellyfinLibraryItemResponse.cs,
which the format gate flags once the file is touched (format-as-you-touch).
fixes#177
Jellyfin libraries typed `mixed` were dropped by JellyfinApiClient.Project's
`_ => None` with no log line, so music and standup content could not be
ingested without a local-library workaround that bypassed Jellyfin entirely.
Adds LibraryMediaKind.Mixed, maps "mixed"/absent/blank CollectionType onto it,
and gives SynchronizeJellyfinLibraryByIdHandler a Mixed arm composing the three
existing per-kind scanners. Jellyfin classifies items server-side via
includeItemTypes, so the passes see disjoint sets; reconciliation is type-scoped
and cannot cross-delete. No new scanner and no DB migration -- MediaItem is TPT
keyed on LibraryPathId, so heterogeneous contents were already legal.
Segregation falls out of the model: a library is a place (one path <-> one
Jellyfin library <-> one ErsatzTV library), so music/standup cannot leak into
Movies or TV Shows.
Also removes the silent-success `_ => Unit.Default` from both scanner
dispatchers, which returned Right for an unhandled kind and stamped LastScan as
though a scan had run, and rejects Mixed for local libraries at the API.
Deliberately Jellyfin-only: local scanners share one video extension list and
would claim each other's files, and LibraryFolder etags are keyed by
LibraryPathId with no notion of kind.
Verified by live E2E against a real Jellyfin, including the interaction with
#494's reconciliation sweep. Four cold review rounds, all MERGEABLE.
fixes#489
Co-authored-by: Timothy <timothy.look@gmail.com>
Co-committed-by: Timothy <timothy.look@gmail.com>
JellyfinMusicVideoLibraryScanner.UpdateMetadata copied only scalar fields
for an EXISTING music video, so genres/tags/studios/artists edited in
Jellyfin never reached ErsatzTV — the update path silently dropped every
child collection (only the Add path ever persisted them). Root cause is
inherited from upstream: unlike the movie/episode Jellyfin path, which
reconciles collections inside the tracked repository GetOrAdd,
MusicVideoRepository.GetOrAdd is AsNoTracking and the scanner never
reconciled the collections itself.
Fix mirrors PlexMovieLibraryScanner.UpdateMetadata's remove-stale +
add-new idiom, reconciling exactly the collections that BOTH the Add path
persists AND GetOrAdd eager-loads: Genres, Tags, Studios, Artists. Guids
(add-persisted but not eager-loaded — would duplicate) and Directors
(eager-loaded but not add-persisted for music videos) are deliberately
out of scope.
Movie/Episode paths do NOT have this gap (they reconcile in the tracked
repo GetOrAdd), so no separate fix is needed there.
Test is an interaction test (substituted repos, canned existing item)
verifying the exact reconcile calls; proven non-vacuous. The real-DB
double-scan approach can't drive this: the in-memory harness shares one
SQLite connection across contexts and mid-scan GetOrAdd's First()-nav
path predicate mis-resolves once the existing item carries metadata
children — a harness-only quirk (prod uses per-context pooled
connections).
fixes#497
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(494): reconcile removed music videos in Jellyfin scanner (#495)
JellyfinMusicVideoLibraryScanner did add/update only, so a music video or
artist deleted on the Jellyfin side lingered forever. Add a library-scoped
trash sweep (TrashMissingMusicVideos) gated by the #477 empty-fetch guard.
fixes#494
Co-authored-by: Timothy <timothy.look@gmail.com>
Co-committed-by: Timothy <timothy.look@gmail.com>
A successful fetch returning zero items made existing.Except([]) flag the ENTIRE
library FileNotFound in one scan — feeding EmptyTrashHandler's permanent delete
and emptying every affected collection (dead channels). Add a shared
MediaServerReconciliationGuard that skips (and logs a Warning) the sweep when
incoming==0 while items exist, wired into the three library-level sweeps
(Television shows / Movie / OtherVideo).
An empty incoming set is indistinguishable at scan time from a mid-restore /
emptied-upstream error (both report a zero total), so this deliberately overrides
#476's degenerate "last item removed => empty incoming => flag" case. #476's
cascade still fires for partial deletions (survivors present); its characterization
test moves from an empty incoming to a survivor+removed partial-deletion case.
Tests: policy table (MediaServerReconciliationGuardTests) + per-scanner integration
proving the wiring (empty incoming + non-empty existing flags/reindexes nothing).
Proven non-vacuous by neutralizing the guard. Nested TV season/episode sweeps left
unguarded (bounded blast radius); ratio-threshold + projection-failure detection
deferred to a follow-up. docs/decisions.md updated.
Fixes#477
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The media-server television scanner reconciles removed items at three levels
via existing.Except(incoming), but the season and episode sweeps live inside
the per-parent loops (ScanSeasons inside the show loop, ScanEpisodes inside the
season loop). Those loops only iterate parents present in the incoming list, so
a show (or season) that is gone from the media server is never visited and its
descendants are never swept — they keep their last state. On Jellyfin/Emby that
is RemoteOnly, which PlayoutBuilder does NOT skip even with PlayoutSkipMissingItems
on, so every orphaned episode keeps getting scheduled as a guaranteed tune-in
failure (the #473 reproduction; 717 stale prod rows across 10 removed shows).
Fix: cascade the flag by parent MediaItem.Id. Two provider-agnostic repo helpers
(Season.ShowId / Episode.SeasonId are on the base tables) flag descendants and the
scanner drives them after each parent sweep — show → seasons → episodes, and
season → episodes for the show-present case. Shared abstract base fixes
Jellyfin/Plex/Emby at once; Movie/OtherVideo are flat and have no such gap.
Tests: a Scanner.Tests case asserts the scanner cascades through the (substituted)
repository (non-vacuous — fails if the cascade calls are removed), and Integration
tests exercise the real cascade SQL against the schema, proving it flags only the
targeted subtree and no-ops on empty input.
fixes#476
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review (MERGEABLE, no blockers) raised two items worth
folding in rather than deferring:
Medium — ScanLocalLibraryHandler silently swallowed path scan errors,
while the three remote scanners it mirrors all log result.LeftToSeq().
That mattered less when a failed path only skipped the path-level
LastScan, but the previous commit makes a failed path suppress the
library-level scan time too — so the user would see exactly the #264
symptom ("Never scanned") with nothing in the log explaining why. That
is a diagnosis dead-end of the same class as the bug being fixed, so
log it here rather than file a follow-up. No test: the sibling
Synchronize*LibraryByIdHandlerTests don't assert on logging either, and
LogError is an extension method that NSubstitute can't cleanly verify.
Low — docs/testing.md bills itself as the authoritative map of what each
test project covers but omitted ErsatzTV.Scanner.Tests entirely (1471
pre-existing tests). This PR adds a file to that project, so add the row
and include it in the per-PR verification gate.
Local libraries permanently showed "Never scanned" in the SPA libraries
hub regardless of successful scans, even as item counts updated.
Root cause: ScanLocalLibraryHandler wrote only the path-level
LibraryPath.LastScan (which gates the per-path refresh interval) and
never the library-level Library.LastScan. The read API
(GetAllMediaSourcesForApiHandler) populates the hub's scan-time badge
from Library.LastScan, so that value stayed null forever. The three
remote scanners (Jellyfin/Emby/Plex) already set the library-level
value; only the local scanner did not. Both sides predate #202 — the
SPA hub merely made the missing value visible.
Mirror the remote scanners' semantics: record the library-level scan
time only when the scan actually ran and every path that ran succeeded,
so a skipped (unforced, interval not elapsed) or partially-failed scan
does not claim a successful scan time.
Also de-BOM the touched handler per the fix-as-you-touch charset gate
(#311).
fixes#264
Centralize SDK and threading analyzers, baseline the .NET 10 All rule inventory at suggestion severity, and promote S3981 repo-wide. Fix the always-true worker count predicate and cover the idle/active branches.
Fixes#15
Co-Authored-By: Codex <codex@openai.com>
Move all 154 PackageReference versions (96 distinct packages) out of the 14
project files into a single central Directory.Packages.props with
ManagePackageVersionsCentrally=true. No version changes — every package was
already pinned identically across projects (no conflicts detected), so this is a
pure relocation: updates become one-line and cross-project version drift is
structurally impossible.
Also copy Directory.Packages.props into the Docker image build before restore:
with CPM the csproj carry no versions, so the image's `dotnet restore` fails
without the central manifest (verified: NU1015 across every project).
Restore + Release build verified locally, plus a simulation of the image's
restore layer under linux-x64 (0 errors; only the pre-existing
NCalcSync/SQLitePCLRaw advisories remain, demoted to warnings, tracked in #8).
Part of #14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* move plex artwork removal to its own repository
* clean up old local movie artwork
* clien up old music video/artist artwork
* clean up old remote stream artwork
* clean up old song artwork
* clean up old show artwork; properly update season artwork
* fix effective block tests running on github
* update dependencies
* pass tz again
* use tzconvert for time zones in tests
* temporary logging
* maybe fix
* test cleanup
timothy
