SQLite's LOWER() folds ASCII only -- lower('Édith') is 'Édith' unchanged --
so the EF-sourced facet fields UNDER-matched any stored value whose prefix
carries an uppercase non-ASCII character. An under-match is unrecoverable:
no later stage can reintroduce a row SQL never returned.
Adds a SECOND, ADDITIVE query taken only when the provider is SQLite and q
contains a non-ASCII character: raw Dapper SQL folding through etv_upper(),
a SqliteConnection.CreateFunction scalar implementing ToUpperInvariant.
Every other case -- all-ASCII q, and MySQL for all q -- runs the existing
EF query byte-identically.
MySQL needed no change and gets none: verified on MySQL 8.4 that its LOWER()
is Unicode-aware and its ci collation makes the predicate OVER-match, which
the existing ordinal filter already discards.
The fold is ToUpperInvariant because OrdinalIgnoreCase equality is a strict
SUBSET of invariant-uppercase equality, so the SQL stage yields a superset of
the final filter's matches and can never under-match. Note OrdinalIgnoreCase
is NOT "invariant-upper then ordinal": ToUpperInvariant('ſ') is 'S', yet
"ſweet".StartsWith("S", OrdinalIgnoreCase) is false. Tests pin that.
No migration, no model change; both provider snapshots are untouched.
Refs #668
Decisions-Edit: yes
Review of 491f5099 found the fix inert in the only process that runs it, plus
a MySQL collation defect in the lookup.
B1 — TvContext.IsUniqueConstraintViolation was assigned only in ErsatzTV/
Startup.cs, but ErsatzTV.Scanner is a separate executable and every production
caller of GetOrAddFolder/SetEtag lives there. The classifier kept its '_ =>
false' default, so the catch never ran and the DbUpdateException failed the
whole scan - worse than the duplicate row it replaced. Wire both provider
branches in ErsatzTV.Scanner/Program.cs, and add ProviderStaticsWiringTests
(architecture) asserting the scanner assigns every TvContext static the host
assigns, with IsSqlite documented as the one exemption.
H1 — GetFolder's 'Path == folder' is case-insensitive on MySQL while PathHash
is case-sensitive, and FirstOrDefault was unordered: a scan of '/x/foo' could
resolve the '/x/Foo' row and stamp the wrong hash onto it (verified on MySQL
8.4: the WHERE matches both, LIMIT 1 returns the wrong one). Treat the SQL
equality as a narrowing filter, order by Id, and settle identity ordinally.
Route the heal through EF and drop a classified violation, so an opportunistic
maintenance write can never abort a scan.
Also: run the dedupe migration test with foreign keys ON (matching prod), clear
the keeper's etag, null out a self-parent, and document the cleanup's limits
(NULL paths excluded, Down does not restore deleted rows, CI's fresh-DB apply
covers none of the data mutation).
Refs #488#308fix#491
Music videos carried no server identity, so JellyfinMusicVideoLibraryScanner had to
reconcile by a (LibraryPathId, path) diff and HARD-delete the remainder. A file served
by two libraries with overlapping local paths is a single row owned by whichever library
scanned it first, so that owner's sweep destroyed a row another library still served —
taking collection membership and playout references with it, irreversibly.
This is #494's deferred "option 2":
- New JellyfinMusicVideo : MusicVideo (ItemId/Etag), mirroring JellyfinMovie — TPT table,
varchar(36), ItemId index. Dual-provider migration Add_JellyfinMusicVideo.
- New IMediaServerMusicVideoRepository + JellyfinMusicVideoRepository: itemId-keyed
existing-set/lookup and Flag{Normal,Unavailable,FileNotFound} seams, all scoped per
library via LibraryPath.LibraryId.
- New MediaServerMusicVideoLibraryScanner base; JellyfinMusicVideoLibraryScanner folds
onto it and keeps the #177/#488/#497/#500 metadata-reconcile logic verbatim.
- The sweep now soft-trashes (FileNotFound) instead of deleting, so removal is reversible
and EmptyTrash-governed. DeleteEmptyArtists consequently no longer fires from a sweep.
- Pre-identity rows are ADOPTED in place: the identity row is inserted against the same
MediaItem id, scoped to the scanned library's own LibraryPath, so collection membership
survives and a local/second-library row is never hijacked.
- AddMusicVideo normalizes Path/PathHash to the path-REPLACED local path; the projection
fills them from the server-reported path, which would break every later PathHash lookup.
Docs: scan.musicvideo-reconciliation relocated to docs/decisions/archive/scan.md as
superseded; new active record scan.musicvideo-server-identity.
fixes#496
* move plex artwork removal to its own repository
* clean up old local movie artwork
* clien up old music video/artist artwork
* clean up old remote stream artwork
* clean up old song artwork
* clean up old show artwork; properly update season artwork
* add external chapter file scanning
Support Matroska chapter xml files next to media file with extension .xml or .chapters
* only update chapters in db
---------
Co-authored-by: Jeff Slutter <MrMustard@gmail.com>
Co-authored-by: Jason Dove <1695733+jasongdove@users.noreply.github.com>
* fix validation in new form layout
* pin mediatr to last oss version
* update dependencies
* cleanup code in core
* cleanup code in ffmpeg
* cleanup code in infra
* cleanup code in scanner
* cleanup code in application
* cleanup main code
* cleanup test code
* solution-wide code cleanup
* initial plumbing
* scan for plex networks
* save network contents to db as tags
* eliminate network tag id churn
* add network and show_network to search index
* update last networks scan
* show networks on tv show page
* update changelog
* Allow Other Videos Library Type on Plex to be sync
* Migrating database: Creating PlexOtherVideo table
* Using Plex Media path to create tags for OtherVideos
* missed these in the merge
* Getting PlexLibrary for Tag set on OtherVideo
* fix migrations
* set tag metadata on plex other videos
* update changelog
---------
Co-authored-by: Jason Dove <1695733+jasongdove@users.noreply.github.com>
* wip
* first pass at elasticsearch; movies kind of work
* use field name constants
* properly sort search results
* fix some crashes
* fix page map/jump letters
* optimize page map using terms aggregation
* index all item types
* optionally use elastic search
* code cleanup
* automatically rebuild lucene index after improper shutdown
* update changelog
* start moving local scans to separate process
* send progress updates to main process
* move scanners and tests
* simplify dependencies; sync search index
* commit search index more often when scanning
* support forced scan and cancellation
* use scanner process for plex libraries
* update changelog
* update dockerfiles
* fix search index for local folder scanning
* rework plex scanners
* rework scanner handlers
* emby works again
* sync jellyfin
* cleanup
* update build
* update changelog
* remove scanner dependency in pr and artifacts workflows
* fix mac sed syntax
* fix pr build
timothy
