timothy/ersatztv
1
0
Fork 0
Code Issues 54 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

fix(491): make the MySql dedupe byte-exact, not just case-exact (PAD SPACE)

Browse Source 
Cross-family review of 1b4dd6d6 found that utf8mb4_bin - chosen to keep the
dedupe case-exact - is a PAD SPACE collation, so trailing spaces are
insignificant under it. Verified on MySQL 8.4: '/media/Foo' = '/media/Foo ' is
TRUE, while case correctly compares unequal. Two distinct legal directories
therefore grouped together and the second was DELETED irreversibly, even though
PathUtils.GetPathHash hashes them differently and the unique index about to be
created would have accepted both. The dedupe destroyed data the constraint
never required it to destroy.

Group and join on CONVERT(Path USING binary) instead - NO PAD and byte-exact,
matching the hash. utf8mb4_0900_bin is also NO PAD but carries a server-version
floor. This is the only path comparison in either migration (every other
predicate keys off an integer id), so there is no mix of padded and unpadded
comparisons across the keeper-selection, repoint and delete steps.

SQLite's = on TEXT is byte-exact with no padding, so that migration was already
correct - which is exactly why a SQLite-only test could not see the divergence.
The two providers are now semantically equivalent, and the dedupe fixture is
shared: same rows, same expected survivors (1,4,5,6,7,9,10), asserted by the
SQLite test and reproduced by hand on MySQL 8.4.

Runtime was never affected, and this is now stated and tested rather than
assumed: GetFolder's SQL equality is a superset narrowing (both collation quirks
make it more permissive, never less, so it cannot miss a byte-exact match) and
ResolveExact settles identity with StringComparison.Ordinal, which compares
length first. Added ResolveExact coverage for the trailing-space axis.

Refs #488 #308
fix #491
This commit is contained in:
timothy
2026-07-25 21:13:31 +02:00
parent 50eff83628
commit 48d41b9235
4 changed files with 75 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ErsatzTV.Infrastructure.MySql/Migrations/20260725121841_Add_LibraryFolder_PathHash_UniqueIndex.cs
+15 -7
View File
@@ -12,10 +12,18 @@ namespace ErsatzTV.Infrastructure.MySql.Migrations
{
// ersatztv#491 — audit/clean pre-existing duplicate LibraryFolder rows before the unique index.
// Mirrors the Sqlite migration; see it for the full rationale. The one provider difference is
// the explicit utf8mb4_bin collation on every Path comparison: MySql's default collation is
// case-INsensitive, and grouping under it would treat sibling folders differing only in case
// (legal on a case-sensitive filesystem) as duplicates and delete one. The unique index is on
// the SHA-256 PathHash precisely so identity stays byte-exact on both providers.
// that every Path comparison here is forced BYTE-EXACT with CONVERT(... USING binary), because
// MySql's string comparison differs from Sqlite's on two independent axes and the dedupe
// deletes rows irreversibly:
// * case — the server default (utf8mb4_general_ci) is case-INsensitive, so grouping under it
// would collapse sibling folders differing only in case, legal on a case-sensitive fs;
// * trailing spaces — utf8mb4_bin, the obvious fix for the case half, is a PAD SPACE
// collation (verified on 8.4: '/media/Foo' = '/media/Foo ' is TRUE under it), so it would
// still collapse "/media/Foo" and "/media/Foo ", two distinct legal directories.
// Binary comparison is NO PAD and byte-exact, which is exactly what PathUtils.GetPathHash does
// — so the dedupe now destroys only rows the unique index would actually have rejected, and
// the two providers' migrations are semantically equivalent. (utf8mb4_0900_bin is also NO PAD
// but carries a server-version floor; CONVERT USING binary does not.)
// DROP TABLE IF EXISTS makes a retry after a partial failure safe (DDL implicitly commits on
// MySql, so the migration is not atomic).
migrationBuilder.Sql("DROP TABLE IF EXISTS `__LibraryFolderDedupe`");
@@ -33,10 +41,10 @@ namespace ErsatzTV.Infrastructure.MySql.Migrations
SELECT l.Id, k.KeeperId
FROM LibraryFolder l
INNER JOIN (
SELECT LibraryPathId, Path COLLATE utf8mb4_bin AS BinPath, MIN(Id) AS KeeperId
SELECT LibraryPathId, CONVERT(Path USING binary) AS BinPath, MIN(Id) AS KeeperId
FROM LibraryFolder
GROUP BY LibraryPathId, Path COLLATE utf8mb4_bin
) k ON k.LibraryPathId = l.LibraryPathId AND k.BinPath = l.Path COLLATE utf8mb4_bin
GROUP BY LibraryPathId, CONVERT(Path USING binary)
) k ON k.LibraryPathId = l.LibraryPathId AND k.BinPath = CONVERT(l.Path USING binary)
WHERE l.Id <> k.KeeperId
""");
ErsatzTV.Infrastructure/Data/Repositories/LibraryRepository.cs
+10 -7
View File
@@ -321,13 +321,16 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
/// <summary>
/// Resolve a folder by its exact path within a library path.
/// <para>
/// The SQL equality is only a *narrowing* filter, not the identity test: on MySQL, `Path` is a
/// `longtext` under the case-INsensitive server default (the same collation the codebase names
/// in <see cref="TvContext.CaseInsensitiveCollation" />), so `Path = @folder` also matches
/// sibling folders differing only in case — which are legal on a case-sensitive filesystem and
/// which the #491 migration deliberately preserves. Identity is settled in memory by
/// <see cref="ResolveExact" /> with an ORDINAL comparison, matching <c>PathUtils.GetPathHash</c>,
/// which hashes the exact bytes. Without this the case-insensitive lookup and the case-sensitive
/// The SQL equality is only a *narrowing* filter, not the identity test. On MySQL, `Path` is a
/// `longtext` compared under the server default, which differs from byte equality on two axes:
/// it is case-INsensitive (the collation the codebase names in
/// <see cref="TvContext.CaseInsensitiveCollation" />), and it is PAD SPACE, so trailing spaces
/// are insignificant. `Path = @folder` therefore also matches siblings differing only in case
/// or in trailing whitespace — all legal on a case-sensitive filesystem, and all preserved by
/// the #491 migration. Crucially the SQL predicate is a *superset*: both quirks make it more
/// permissive, never less, so it cannot miss a byte-exact match. Identity is then settled in
/// memory by <see cref="ResolveExact" /> with an ORDINAL comparison, matching
/// <c>PathUtils.GetPathHash</c>, which hashes the exact bytes. Without this the lookup and the
/// hash disagree, and the PathHash heal could stamp one sibling's hash onto the other's row.
/// </para>
/// <para>
ErsatzTV.Tests/Integration/LibraryFolderDedupeMigrationTests.cs
+27 -5
View File
@@ -90,7 +90,14 @@ public class LibraryFolderDedupeMigrationTests
-- duplicate 3 is parented on duplicate 2: repointing both would make the survivor
-- its own parent, the cycle the ParentId null-out guards
(7, 1, '/data/music/artist3', 8, NULL),
(8, 1, '/data/music/artist3', NULL, NULL)
(8, 1, '/data/music/artist3', NULL, NULL),
-- a sibling differing only by a TRAILING SPACE. Two distinct legal directories on
-- Linux, and PathUtils.GetPathHash hashes them differently, so the unique index would
-- accept both — the dedupe must not collapse them. MySql's string comparison is PAD
-- SPACE (verified on 8.4, for utf8mb4_bin as well as the ci default), so grouping
-- under any collation rather than binary would delete row 10 irreversibly.
(9, 1, '/data/music/pad', NULL, NULL),
(10, 1, '/data/music/pad ', NULL, NULL)
""");
// a media file on each duplicate, and an image-folder-duration on a duplicate only
@@ -99,7 +106,10 @@ public class LibraryFolderDedupeMigrationTests
INSERT INTO MediaFile (Id, Path, PathHash, MediaVersionId, LibraryFolderId) VALUES
(1, '/data/music/artist1/a.mkv', 'hash-a', 1, 1),
(2, '/data/music/artist1/b.mkv', 'hash-b', 2, 2),
(3, '/data/music/artist1/c.mkv', 'hash-c', 3, 3)
(3, '/data/music/artist1/c.mkv', 'hash-c', 3, 3),
-- dependents of the trailing-space pair: must stay attached to their OWN folder
(4, '/data/music/pad/d.mkv', 'hash-d', 4, 9),
(5, '/data/music/pad /e.mkv', 'hash-e', 5, 10)
""");
await context.Connection.ExecuteAsync(
"""
@@ -124,16 +134,28 @@ public class LibraryFolderDedupeMigrationTests
folderIds.ShouldBe([1]);
// survivors: keeper 1, artist2 (4), the child (5), the case-differing sibling (6), artist3's
// keeper (7). Losers 2, 3 and 8 are gone.
// keeper (7), and BOTH halves of the trailing-space pair (9, 10). Losers 2, 3 and 8 are gone.
List<int> survivors =
(await context.Connection.QueryAsync<int>(
"SELECT Id FROM LibraryFolder ORDER BY Id")).ToList();
survivors.ShouldBe([1, 4, 5, 6, 7]);
survivors.ShouldBe([1, 4, 5, 6, 7, 9, 10]);
// a folder differing only in case is NOT a duplicate — it must survive untouched
(await context.Connection.ExecuteScalarAsync<int>(
"SELECT COUNT(*) FROM LibraryFolder WHERE Path = '/data/music/ARTIST2'")).ShouldBe(1);
// nor is one differing only by a trailing space, and each keeps its OWN dependents
List<string> padded =
(await context.Connection.QueryAsync<string>(
"SELECT Id || '=[' || Path || ']' FROM LibraryFolder WHERE Id IN (9, 10) ORDER BY Id"))
.ToList();
padded.ShouldBe(["9=[/data/music/pad]", "10=[/data/music/pad ]"]);
List<int> padMediaFolderIds =
(await context.Connection.QueryAsync<int>(
"SELECT LibraryFolderId FROM MediaFile WHERE Id IN (4, 5) ORDER BY Id")).ToList();
padMediaFolderIds.ShouldBe([9, 10]);
// the keeper's etag is cleared: which duplicate the scanner was writing to was arbitrary, so
// MIN(Id)'s etag could suppress the rescan that repairs the collapsed folder
(await context.Connection.ExecuteScalarAsync<string>(
@@ -146,7 +168,7 @@ public class LibraryFolderDedupeMigrationTests
// every media file follows the keeper — nothing orphaned, nothing deleted
List<int> mediaFolderIds =
(await context.Connection.QueryAsync<int>(
"SELECT LibraryFolderId FROM MediaFile ORDER BY Id")).ToList();
"SELECT LibraryFolderId FROM MediaFile WHERE Id IN (1, 2, 3) ORDER BY Id")).ToList();
mediaFolderIds.ShouldBe([1, 1, 1]);
// the child folder is reparented off the deleted duplicate onto the keeper
ErsatzTV.Tests/Integration/LibraryRepositoryTests.cs
+23
View File
@@ -216,6 +216,29 @@ public class LibraryRepositoryTests
LibraryRepository.ResolveExact([], "/x/foo").ShouldBeNull();
}
// MySQL's default collation is not only case-insensitive, it is also PAD SPACE (verified on 8.4:
// '/x/foo' = '/x/foo ' is TRUE under utf8mb4_bin and utf8mb4_general_ci alike), so the SQL narrowing
// hands back trailing-space siblings too. Those are distinct directories on Linux and hash
// differently, so the ordinal settle has to keep them apart — same defect class as the case variant,
// second axis. Ordinal compares length first, so this holds; pin it.
[Test]
public void ResolveExact_Should_Distinguish_Paths_Differing_Only_In_Trailing_Space()
{
var candidates = new List<LibraryFolder>
{
new() { Id = 20, Path = "/x/foo" },
new() { Id = 21, Path = "/x/foo " }
};
LibraryRepository.ResolveExact(candidates, "/x/foo").Id.ShouldBe(20);
LibraryRepository.ResolveExact(candidates, "/x/foo ").Id.ShouldBe(21);
LibraryRepository.ResolveExact(candidates, "/x/foo ").ShouldBeNull();
// the two spellings must also hash differently, or the unique index would reject one of them and
// the migration's decision to keep both would be wrong
PathUtils.GetPathHash("/x/foo").ShouldNotBe(PathUtils.GetPathHash("/x/foo "));
}
// Candidates arrive ordered by Id, and the first ordinal match wins — so true duplicates resolve to
// MIN(Id), the same row the #491 migration keeps.
[Test]