The dedupe DML had zero automated coverage on MySql: the migrations job only
applies migrations to a fresh EMPTY database, so no dedupe row ever executed
there. Two MySql-only collation defects escaped that gate in this session and
were caught only by hand-run containers.
Parameterize LibraryFolderDedupeMigrationTests over both providers from ONE
fixture body - same seeded rows, same expected survivors - rather than adding a
MySql-only copy that would drift and recreate the gap. Assertions no longer use
WHERE Path = '...', which is itself collation-dependent and would quietly mean
something different per provider; rows are read once and compared ordinally in
memory. A new step in the existing migrations job runs it against that job's
mysql:8.4 service, on a per-test database of its own.
Proven red when the collation is wrong: restoring COLLATE utf8mb4_bin fails the
MySql half with survivors [1,4,5,6,7,9] - the trailing-space sibling deleted -
while SQLite stays green. Proven non-skippable: without
ETV_TEST_MYSQL_CONNECTION the fixture ignores visibly, and with
ETV_REQUIRE_MYSQL_TESTS=1 (which CI sets) that skip becomes a hard failure, so
it cannot pass having connected to nothing. Local runs need no MySql.
Also correct an overstated comment. The schema pins only the utf8mb4 charset,
never a collation, so the effective comparison is the server default: always
case-insensitive, but PAD SPACE only on utf8mb4_general_ci - 8.4's default
utf8mb4_0900_ai_ci is NO PAD, verified on the real column. The migration bug was
independent of that because the old code applied an EXPLICIT utf8mb4_bin, which
is PAD SPACE everywhere; the runtime simply tolerates both.
Refs #488#308fix#491
Cross-family review of 1b4dd6d6 found that utf8mb4_bin - chosen to keep the
dedupe case-exact - is a PAD SPACE collation, so trailing spaces are
insignificant under it. Verified on MySQL 8.4: '/media/Foo' = '/media/Foo ' is
TRUE, while case correctly compares unequal. Two distinct legal directories
therefore grouped together and the second was DELETED irreversibly, even though
PathUtils.GetPathHash hashes them differently and the unique index about to be
created would have accepted both. The dedupe destroyed data the constraint
never required it to destroy.
Group and join on CONVERT(Path USING binary) instead - NO PAD and byte-exact,
matching the hash. utf8mb4_0900_bin is also NO PAD but carries a server-version
floor. This is the only path comparison in either migration (every other
predicate keys off an integer id), so there is no mix of padded and unpadded
comparisons across the keeper-selection, repoint and delete steps.
SQLite's = on TEXT is byte-exact with no padding, so that migration was already
correct - which is exactly why a SQLite-only test could not see the divergence.
The two providers are now semantically equivalent, and the dedupe fixture is
shared: same rows, same expected survivors (1,4,5,6,7,9,10), asserted by the
SQLite test and reproduced by hand on MySQL 8.4.
Runtime was never affected, and this is now stated and tested rather than
assumed: GetFolder's SQL equality is a superset narrowing (both collation quirks
make it more permissive, never less, so it cannot miss a byte-exact match) and
ResolveExact settles identity with StringComparison.Ordinal, which compares
length first. Added ResolveExact coverage for the trailing-space axis.
Refs #488#308fix#491
Review follow-up. ResolveExact was inserted between GetFolder's summary
and GetFolder itself, leaving ResolveExact with two summaries and
GetFolder with none -- so the H1 rationale (SQL equality is only a
narrowing filter; OrderBy(Id) for determinism) described neither of the
things ResolveExact does. On a fix that turns on exactly that reasoning,
a misattached explanation is what misleads the next reader.
Final low-severity items from the re-review of ee10f932.
L2: DbUpdateConcurrencyException derives from DbUpdateException but carries no
provider exception, so IsUniqueConstraintViolation does not classify it. A row
deleted by a concurrent library edit between the heal's read and its save would
propagate and fail the scan, contradicting the invariant stated directly above
it. Admit it in the filter.
L1: lift the in-memory ordinal settle into LibraryRepository.ResolveExact and
unit-test it with both spellings in the candidate list. No SQLite-backed test
can exercise it (SQLite's = on TEXT is already binary), so this converts the
half that rested on hand-run MySQL evidence into automated coverage. The
end-to-end companion test's comment no longer claims to be provider-independent.
L4: assert PRAGMA foreign_keys is 1 before migrating, so the enforcement guard
cannot silently degrade into the weak pre-fix form it was added to replace.
L3: detach the failed heal, matching the insert path.
N3: the heal's inner predicate now matches its IsNullOrEmpty outer guard, so a
PathHash = '' row cannot enter the branch and silently never heal.
N4: record that GetFolder returning null for a case-differing spelling makes
MySQL insert a second row where it used to reuse one — correct, and now matching
SQLite, but a real behaviour change on a case-insensitive filesystem.
Refs #488#308fix#491
Review of 491f5099 found the fix inert in the only process that runs it, plus
a MySQL collation defect in the lookup.
B1 — TvContext.IsUniqueConstraintViolation was assigned only in ErsatzTV/
Startup.cs, but ErsatzTV.Scanner is a separate executable and every production
caller of GetOrAddFolder/SetEtag lives there. The classifier kept its '_ =>
false' default, so the catch never ran and the DbUpdateException failed the
whole scan - worse than the duplicate row it replaced. Wire both provider
branches in ErsatzTV.Scanner/Program.cs, and add ProviderStaticsWiringTests
(architecture) asserting the scanner assigns every TvContext static the host
assigns, with IsSqlite documented as the one exemption.
H1 — GetFolder's 'Path == folder' is case-insensitive on MySQL while PathHash
is case-sensitive, and FirstOrDefault was unordered: a scan of '/x/foo' could
resolve the '/x/Foo' row and stamp the wrong hash onto it (verified on MySQL
8.4: the WHERE matches both, LIMIT 1 returns the wrong one). Treat the SQL
equality as a narrowing filter, order by Id, and settle identity ordinally.
Route the heal through EF and drop a classified violation, so an opportunistic
maintenance write can never abort a scan.
Also: run the dedupe migration test with foreign keys ON (matching prod), clear
the keeper's etag, null out a self-parent, and document the cleanup's limits
(NULL paths excluded, Down does not restore deleted rows, CI's fresh-DB apply
covers none of the data mutation).
Refs #488#308fix#491
GetOrAddFolder was a check-then-insert with no unique constraint behind it,
so two callers racing the same folder could both miss the lookup and both
insert. Enforce identity in the schema and make the loser adopt the winner.
- LibraryFolder gains a SHA-256 PathHash (the MediaFile.Path/PathHash
precedent): Path is MySQL longtext, which cannot be indexed without a
prefix length and collates case-insensitively, so the unique index is on
(LibraryPathId, PathHash) instead.
- GetOrAddFolder and SetEtag catch a classified unique violation via the
existing TvContext.IsUniqueConstraintViolation seam (#308) and re-read.
- Dual-provider migration audits and collapses pre-existing duplicates
(repointing MediaFile, ParentId and ImageFolderDuration) before creating
the index; legacy rows keep a null hash and heal on the next scan.
- Tests: deterministic cross-connection race, 8x10 barrier stress with an
insert-attempt vacuity guard, classifier-inversion negative control, and
a real-migration dedupe test.
Refs #488#308fix#491
GetOrAddFolder read libraryPath.LibraryFolders, a navigation collection only
eager-loaded on the local scan path (via GetLibrary). The Jellyfin remote sync
path takes its LibraryPath off the JellyfinLibrary entity, where LibraryFolders
is null, so .Filter(null) threw ArgumentNullException('source') on the first
item of every Jellyfin music-video scan — a feature that had therefore never
run in prod, CI, or locally.
Look the folder up from the DB by (LibraryPathId, Path) instead, removing the
implicit eager-load contract entirely (correct for all nine callers) and
documenting it on ILibraryRepository. null != empty is preserved so a re-scan
does not insert duplicate LibraryFolder rows. No new hot-path cost: local
scanners already query GetParentFolderId per folder just before this call.
Tests:
- LibraryRepositoryTests: GetOrAddFolder with a null LibraryFolders (the exact
remote-path shape) creates the folder, is idempotent on re-scan, and persists
a supplied ParentId.
- JellyfinMusicVideoLibraryScannerTests: an end-to-end scan of one synthetic
music video, wiring the REAL LibraryRepository/ArtistRepository/
MusicVideoRepository against in-memory SQLite (the existing MediaServer*
scanner tests mock every repo, which is why the bug escaped), asserts the scan
completes and creates Artist + MusicVideo rows with a real LibraryFolder.
Both proven non-vacuous against the reverted fix (each reproduces the issue's
ArgumentNullException). decisions.md entry added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds LocalLibrariesController (L1-L7: list/get/create/update/delete/move-path/
path-exists) wrapping the existing local-library MediatR commands, mapping to
the shared S0 response DTOs. Per design #202 §A.1/§C5/§C6:
- 404 for L4/L5/L6 comes from a controller pre-check (GetLocalLibraryById is
None), not the handler -- .Apply/.ToEitherAsync both .Join() a NotFoundError
into a plain 422, so relying on the handler would be dead code. This is
check-then-act; a delete racing the pre-check falls through to the handler's
422, documented in the controller.
- L4/L5 409 via IEntityLocker.IsLibraryLocked(id); L6 resolves the source
library from the path id (new ILibraryRepository.GetLibraryIdForPath) before
its own lock check.
- MoveLocalLibraryPathHandler gains same-MediaKind and different-library
validation (finding 3) -- Blazor only filtered these client-side in the move
dialog, so an API/MCP client could bypass them.
- CreateLocalLibraryHandler/UpdateLocalLibraryHandler gain a shared
NewPathsMustExist validation (LocalLibraryHandlerBase) that Directory.Exists-
checks only new paths (Id < 1); existing rows stay exempt so an unmounted
share doesn't block a rename. L7 (path-exists) is a controller-local
IFileSystem check with no command.
Tests: controller route/mediator-arg tests incl. 404-pre-check vs
fall-through-422 and 409-lock cases; handler tests for the move-path
cross-kind/same-library 422s, new-path 422 (missing/mixed), and a lossless
round-trip proving local paths are identified by normalized path string, not
id. Full solution test suite (Scanner/Core/Architecture/Tests/Infrastructure)
green, 0 regressions.
Deviations: none from the S1 slice description. Did not touch
MediaSourceRepository.cs or any Plex/Jellyfin/Emby file (S2/S3 scope). Did not
run the OpenAPI regen scripts (separate gate after S1-S3 merge per design §E).
* improve scanner detection when developing on macos
* support ogv files in local libraries
* improve health check wording
* try to properly restore other videos that are no longer missing
* flag local movies as file not found
* show warning icon on cards
* unflag movie that is found during scan
* skip missing files when building playouts
* add state to search index
* add file not found health check
* link to search from file not found health check
* support flagging other media kinds as file not found
* continue to schedule missing items
* support episode files not found
* wip trash page
* fix trash url
* trash page is functional
* update changelog
* fix changelog merge
* use etags to optimize local movie scanner
* use etags to optimize local television scanner
* use etags to optimize local music video scanner
* code cleanup
* add music videos library
* add music video tables
* first pass at music video library scan
* support music videos in playouts
* display music videos in search results and collections
* fix music video thumbnails
* remove some obsolete fields
* add letter bar with no links
* use lucene for search, add paged search results
* add search index version
* index library_name; rebuild index when folder is missing
* maintain index as local movies change
* fix tests
* maintain index as local shows change
* maintain index as plex movies change
* maintain index as plex shows change
* code cleanup
* add duplicate filter to search
* add links to letter bar
* code cleanup
* starting database redesign
* set season and episode numbers
* use datetimes in db (utc); update movie metadata
* get movie cards from new table
* copy show/episode metadata
* remove old movie metadata type
* rename new movie metadata type
* code cleanup
* start to remove old television classes
* remove old television tables from database
* fix playout building
* fix collection views
* fix show/season views
* clean up movie metadata table
* fix scanner tests
* add libraries ui
* code cleanup
* fix movie scanning/metadata
* add library scan button to ui
* delete library path from ui
* temp disable movie scanning
* remove orphan media items and prevent duplicate paths
* attach artwork to metadata
* fix split show/season display
* fix television artwork
* store year distinct from release date
* fix collections ui
* code cleanup
* add library paths from ui
* fix adding to collections from ui
* fix schedule items loading
* schedule editing works again
* remove some todos
* more cleanup
* fix unit tests
* fix episode sorting
* fix deleting show library paths
* remove unused class
* fix playout list in ui
* fix log viewer
* start to use version/file instead of statistics
* clean up old columns
* fix playout display (time zone)
* fix playback
* fix channel guide time zone
* cascade more deletes
* fix compiler warnings
* fix adding new seasons
* use artwork for channel logo
* clean cache folder on startup (move channel logos, delete everything else)
* log database migration
* update homepage docs for libraries
* fix adding new channel with logo
* fix episode numbers in epg
timothy
