fix(491): unique index on LibraryFolder(LibraryPathId, PathHash) + tolerate concurrent insert #613

Merged
timothy merged 11 commits from fix/491-libraryfolder-unique-index into main 2026-07-25 22:00:45 +02:00
18 changed files with 16316 additions and 26 deletions
+10
View File
@@ -408,6 +408,16 @@ jobs:
done
echo "::endgroup::"
# NOTE (ersatztv#491 -> #627): running the LibraryFolder dedupe fixture against the live `mysql`
# service was implemented here and then REMOVED. The coverage gap it closes is real — the two
# checks above only ever apply migrations to a fresh EMPTY database, so they execute no rows of any
# data-migration logic, and two MySql-only collation defects escaped exactly this gate. But the
# fixture proved non-deterministic in CI across three attempts (stale pooled session after a drop,
# then lost isolation from a shared database name, then a connect-before-create), and an
# intermittently-red gate is worse than none: it trains everyone to re-run instead of read, which is
# how the original defects escaped. The fixture itself is retained and is opt-in via
# ETV_TEST_MYSQL_CONNECTION (skipped, visibly, without it). Re-arming it here is tracked by #627.
functional-e2e:
name: Functional E2E (curl + UI contracts)
runs-on: ubuntu-latest
@@ -0,0 +1,104 @@
using System.Text.RegularExpressions;
using NUnit.Framework;
using Shouldly;
namespace ErsatzTV.Architecture.Tests;
/// <summary>
/// ersatztv#491: <c>TvContext</c> carries settable provider statics (<c>LastInsertedRowId</c>,
/// <c>CaseInsensitiveCollation</c>, <c>IsUniqueConstraintViolation</c>, …) that Infrastructure code
/// reads at runtime. There are TWO composition roots that execute that Infrastructure code —
/// <c>ErsatzTV/Startup.cs</c> (the host) and <c>ErsatzTV.Scanner/Program.cs</c> (a separate
/// executable launched per scan by <c>CallLibraryScannerHandler</c>) — and each wires the statics in
/// its own copy of the provider branch.
/// <para>
/// The failure mode this guards is "a static nobody assigned": #491 added
/// <c>IsUniqueConstraintViolation</c> to <c>Startup</c> only, so every production caller of
/// <c>GetOrAddFolder</c> (all of which live in the scanner) silently kept the conservative
/// <c>_ =&gt; false</c> default and the new catch was inert. Nothing about that is visible in a
/// unit test, because every test harness wires the classifier itself.
/// </para>
/// <para>
/// Source-level rather than reflective on purpose: the wiring lives inside a host-builder
/// lambda that cannot be invoked without standing up a real application, and the thing being
/// asserted is precisely that a line of code exists in both files.
/// </para>
/// </summary>
[TestFixture]
public class ProviderStaticsWiringTests
{
/// <summary>
/// Statics the host wires that the scanner deliberately does not. Add to this only with a reason:
/// the default must be provably harmless in the scanner process.
/// </summary>
private static readonly Dictionary<string, string> ScannerExemptions = new()
{
// Only read by DbInitializer / DatabaseMigratorService, which run in the host exclusively; no
// Infrastructure code on a scan path reads it. Pre-dates #491.
["IsSqlite"] = "read only by DbInitializer + DatabaseMigratorService, both host-only"
};
private static string HostSource => ReadRepoFile(Path.Combine("ErsatzTV", "Startup.cs"));
private static string ScannerSource => ReadRepoFile(Path.Combine("ErsatzTV.Scanner", "Program.cs"));
[Test]
public void Scanner_should_wire_every_TvContext_provider_static_the_host_wires()
{
HashSet<string> host = AssignedStatics(HostSource);
HashSet<string> scanner = AssignedStatics(ScannerSource);
// sanity: the parser found the wiring at all, so a rename can't turn this test into a no-op
host.ShouldContain("LastInsertedRowId");
host.ShouldContain("IsUniqueConstraintViolation");
scanner.ShouldContain("LastInsertedRowId");
List<string> missing = host
.Except(scanner)
.Except(ScannerExemptions.Keys)
.OrderBy(name => name, StringComparer.Ordinal)
.ToList();
missing.ShouldBeEmpty(
"ErsatzTV.Scanner/Program.cs does not assign TvContext static(s) that ErsatzTV/Startup.cs "
+ $"assigns: {string.Join(", ", missing)}. The scanner is a separate process, so an unassigned "
+ "static keeps its default in every library scan. Wire it in BOTH provider branches, or add "
+ "it to ScannerExemptions with a reason if the default is provably harmless there.");
}
[Test]
public void Both_hosts_should_wire_the_unique_constraint_classifier_for_both_providers()
{
// The specific #491 regression, asserted directly rather than via set arithmetic: the classifier
// must be pointed at a real provider implementation on BOTH branches of BOTH composition roots.
foreach ((string name, string source) in new[] { ("host", HostSource), ("scanner", ScannerSource) })
{
source.ShouldContain(
"TvContext.IsUniqueConstraintViolation = SqliteErrorClassifier.IsUniqueConstraintViolation",
customMessage: $"{name} does not wire the Sqlite unique-constraint classifier");
source.ShouldContain(
"TvContext.IsUniqueConstraintViolation = MySqlErrorClassifier.IsUniqueConstraintViolation",
customMessage: $"{name} does not wire the MySql unique-constraint classifier");
}
}
private static HashSet<string> AssignedStatics(string source) =>
Regex.Matches(source, @"\bTvContext\.(?<name>[A-Za-z_][A-Za-z0-9_]*)\s*=[^=]")
.Select(m => m.Groups["name"].Value)
.ToHashSet(StringComparer.Ordinal);
private static string ReadRepoFile(string relativePath)
{
var directory = new DirectoryInfo(AppContext.BaseDirectory);
while (directory is not null && !File.Exists(Path.Combine(directory.FullName, "ErsatzTV.sln")))
{
directory = directory.Parent;
}
directory.ShouldNotBeNull("could not locate the repository root (no ErsatzTV.sln above the test binary)");
string path = Path.Combine(directory!.FullName, relativePath);
File.Exists(path).ShouldBeTrue($"expected source file not found: {path}");
return File.ReadAllText(path);
}
}
+11 -1
View File
@@ -1,9 +1,19 @@
namespace ErsatzTV.Core.Domain;
namespace ErsatzTV.Core.Domain;
public class LibraryFolder
{
public int Id { get; set; }
public string Path { get; set; }
/// <summary>
/// SHA-256 hex of <see cref="Path" /> (<see cref="ErsatzTV.Core.PathUtils.GetPathHash" />), the
/// indexable stand-in for the unbounded <see cref="Path" /> column that backs the unique
/// <c>(LibraryPathId, PathHash)</c> constraint — the same shape as <c>MediaFile.PathHash</c>.
/// Nullable: rows created before ersatztv#491 carry <c>null</c> until a scan heals them, and a
/// unique index treats nulls as distinct so those legacy rows never collide.
/// </summary>
public string PathHash { get; set; }
public int LibraryPathId { get; set; }
public LibraryPath LibraryPath { get; set; }
public int? ParentId { get; set; }
@@ -0,0 +1,172 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ErsatzTV.Infrastructure.MySql.Migrations
{
/// <inheritdoc />
public partial class Add_LibraryFolder_PathHash_UniqueIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
// ersatztv#491 — audit/clean pre-existing duplicate LibraryFolder rows before the unique index.
// Mirrors the Sqlite migration; see it for the full rationale. The one provider difference is
// that every Path comparison here is forced BYTE-EXACT with CONVERT(... USING binary), because
// MySql's string comparison differs from Sqlite's on two independent axes and the dedupe
// deletes rows irreversibly:
// * case — the server default (utf8mb4_general_ci) is case-INsensitive, so grouping under it
// would collapse sibling folders differing only in case, legal on a case-sensitive fs;
// * trailing spaces — utf8mb4_bin, the obvious fix for the case half, is a PAD SPACE
// collation (verified on 8.4: '/media/Foo' = '/media/Foo ' is TRUE under it), so it would
// still collapse "/media/Foo" and "/media/Foo ", two distinct legal directories.
// Binary comparison is NO PAD and byte-exact, which is exactly what PathUtils.GetPathHash does
// — so the dedupe now destroys only rows the unique index would actually have rejected, and
// the two providers' migrations are semantically equivalent. (utf8mb4_0900_bin is also NO PAD
// but carries a server-version floor; CONVERT USING binary does not.)
// DROP TABLE IF EXISTS makes a retry after a partial failure safe (DDL implicitly commits on
// MySql, so the migration is not atomic).
migrationBuilder.Sql("DROP TABLE IF EXISTS `__LibraryFolderDedupe`");
migrationBuilder.Sql(
"""
CREATE TABLE `__LibraryFolderDedupe` (
LoserId INT NOT NULL PRIMARY KEY,
KeeperId INT NOT NULL
)
""");
migrationBuilder.Sql(
"""
INSERT INTO `__LibraryFolderDedupe` (LoserId, KeeperId)
SELECT l.Id, k.KeeperId
FROM LibraryFolder l
INNER JOIN (
SELECT LibraryPathId, CONVERT(Path USING binary) AS BinPath, MIN(Id) AS KeeperId
FROM LibraryFolder
GROUP BY LibraryPathId, CONVERT(Path USING binary)
) k ON k.LibraryPathId = l.LibraryPathId AND k.BinPath = CONVERT(l.Path USING binary)
WHERE l.Id <> k.KeeperId
""");
// media files recorded against a duplicate folder follow the keeper (MediaFile.LibraryFolderId
// is Restrict, so the delete below would fail otherwise)
migrationBuilder.Sql(
"""
UPDATE MediaFile
SET LibraryFolderId = (
SELECT KeeperId FROM `__LibraryFolderDedupe` WHERE LoserId = MediaFile.LibraryFolderId)
WHERE LibraryFolderId IN (SELECT LoserId FROM `__LibraryFolderDedupe`)
""");
// child folders parented on a duplicate follow the keeper (ParentId is Restrict as well)
migrationBuilder.Sql(
"""
UPDATE LibraryFolder
SET ParentId = (
SELECT KeeperId FROM `__LibraryFolderDedupe` WHERE LoserId = LibraryFolder.ParentId)
WHERE ParentId IN (SELECT LoserId FROM `__LibraryFolderDedupe`)
""");
// A folder parented on its OWN duplicate would become its own parent above. Unreachable from
// any code path today, but this is a tree the scanner walks, so remove the cycle class rather
// than reason about it.
migrationBuilder.Sql("UPDATE LibraryFolder SET ParentId = NULL WHERE ParentId = Id");
// Clear the survivor's etag. Which duplicate the scanner was actually writing to was
// arbitrary, so MIN(Id)'s etag may describe a stale view of the folder and would suppress the
// next rescan. A null etag costs exactly one rescan and cannot be wrong.
migrationBuilder.Sql(
"UPDATE LibraryFolder SET Etag = NULL WHERE Id IN (SELECT KeeperId FROM `__LibraryFolderDedupe`)");
// ImageFolderDuration is 1:1 with LibraryFolder (unique index on LibraryFolderId), so the
// duplicates' rows cannot all be repointed. Keep the keeper's own setting when it has one;
// otherwise promote exactly one loser's (lowest Id) and drop the rest.
migrationBuilder.Sql("DROP TABLE IF EXISTS `__LibraryFolderDedupeIfd`");
migrationBuilder.Sql(
"""
CREATE TABLE `__LibraryFolderDedupeIfd` (
KeeperId INT NOT NULL PRIMARY KEY,
IfdId INT NOT NULL
)
""");
migrationBuilder.Sql(
"""
INSERT INTO `__LibraryFolderDedupeIfd` (KeeperId, IfdId)
SELECT d.KeeperId, MIN(i.Id)
FROM `__LibraryFolderDedupe` d
INNER JOIN ImageFolderDuration i ON i.LibraryFolderId = d.LoserId
WHERE NOT EXISTS (
SELECT 1 FROM ImageFolderDuration ki WHERE ki.LibraryFolderId = d.KeeperId)
GROUP BY d.KeeperId
""");
migrationBuilder.Sql(
"""
DELETE FROM ImageFolderDuration
WHERE LibraryFolderId IN (SELECT LoserId FROM `__LibraryFolderDedupe`)
AND Id NOT IN (SELECT IfdId FROM `__LibraryFolderDedupeIfd`)
""");
migrationBuilder.Sql(
"""
UPDATE ImageFolderDuration
SET LibraryFolderId = (
SELECT KeeperId FROM `__LibraryFolderDedupeIfd` WHERE IfdId = ImageFolderDuration.Id)
WHERE Id IN (SELECT IfdId FROM `__LibraryFolderDedupeIfd`)
""");
migrationBuilder.Sql(
"DELETE FROM LibraryFolder WHERE Id IN (SELECT LoserId FROM `__LibraryFolderDedupe`)");
migrationBuilder.Sql("DROP TABLE `__LibraryFolderDedupeIfd`");
migrationBuilder.Sql("DROP TABLE `__LibraryFolderDedupe`");
migrationBuilder.AddColumn<string>(
name: "PathHash",
table: "LibraryFolder",
type: "varchar(64)",
maxLength: 64,
nullable: true)
.Annotation("MySql:CharSet", "utf8mb4");
// Existing rows keep a null hash on purpose: a unique index treats nulls as distinct, so the
// index applies cleanly to any database, and LibraryRepository.GetOrAddFolder heals each row
// (SHA-256 of Path) the first time a scan touches it. Those rows are deduplicated above and are
// still found by the Path lookup, so no insert can race them in the meantime.
//
// Order matters on MySql, and EF scaffolds it the other way round: InnoDB refuses to drop the
// FK's only backing index ("Cannot drop index 'IX_LibraryFolder_LibraryPathId': needed in a
// foreign key constraint"). Create the composite first — LibraryPathId is its leftmost column,
// so it takes over as the FK's backing index — then drop the now-redundant single-column one.
migrationBuilder.CreateIndex(
name: "IX_LibraryFolder_LibraryPathId_PathHash",
table: "LibraryFolder",
columns: new[] { "LibraryPathId", "PathHash" },
unique: true);
migrationBuilder.DropIndex(
name: "IX_LibraryFolder_LibraryPathId",
table: "LibraryFolder");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
// mirror of Up: restore the single-column index before dropping the composite one, so the
// foreign key is never left without a backing index
migrationBuilder.CreateIndex(
name: "IX_LibraryFolder_LibraryPathId",
table: "LibraryFolder",
column: "LibraryPathId");
migrationBuilder.DropIndex(
name: "IX_LibraryFolder_LibraryPathId_PathHash",
table: "LibraryFolder");
migrationBuilder.DropColumn(
name: "PathHash",
table: "LibraryFolder");
}
}
}
@@ -1353,12 +1353,17 @@ namespace ErsatzTV.Infrastructure.MySql.Migrations
b.Property<string>("Path")
.HasColumnType("longtext");
b.Property<string>("PathHash")
.HasMaxLength(64)
.HasColumnType("varchar(64)");
b.HasKey("Id");
b.HasIndex("LibraryPathId");
b.HasIndex("ParentId");
b.HasIndex("LibraryPathId", "PathHash")
.IsUnique();
b.ToTable("LibraryFolder", (string)null);
});
@@ -0,0 +1,157 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ErsatzTV.Infrastructure.Sqlite.Migrations
{
/// <inheritdoc />
public partial class Add_LibraryFolder_PathHash_UniqueIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
// ersatztv#491 — audit/clean pre-existing duplicate LibraryFolder rows before the unique index.
// Duplicates were reachable before #488 (the folder lookup read a scan-start in-memory snapshot,
// so a folder created earlier in the SAME scan was invisible and inserted again) and via the
// check-then-insert race the index now closes. Keep the lowest Id per (LibraryPathId, Path) and
// repoint every dependent row at it before deleting the losers. The helper tables keep the
// statements readable; DROP TABLE IF EXISTS makes a retry after a partial failure safe.
migrationBuilder.Sql("DROP TABLE IF EXISTS __LibraryFolderDedupe");
migrationBuilder.Sql(
"""
CREATE TABLE __LibraryFolderDedupe (
LoserId INTEGER NOT NULL PRIMARY KEY,
KeeperId INTEGER NOT NULL
)
""");
migrationBuilder.Sql(
"""
INSERT INTO __LibraryFolderDedupe (LoserId, KeeperId)
SELECT l.Id, k.KeeperId
FROM LibraryFolder l
INNER JOIN (
SELECT LibraryPathId, Path, MIN(Id) AS KeeperId
FROM LibraryFolder
GROUP BY LibraryPathId, Path
) k ON k.LibraryPathId = l.LibraryPathId AND k.Path = l.Path
WHERE l.Id <> k.KeeperId
""");
// media files recorded against a duplicate folder follow the keeper (MediaFile.LibraryFolderId
// is Restrict, so the delete below would fail otherwise)
migrationBuilder.Sql(
"""
UPDATE MediaFile
SET LibraryFolderId = (
SELECT KeeperId FROM __LibraryFolderDedupe WHERE LoserId = MediaFile.LibraryFolderId)
WHERE LibraryFolderId IN (SELECT LoserId FROM __LibraryFolderDedupe)
""");
// child folders parented on a duplicate follow the keeper (ParentId is Restrict as well)
migrationBuilder.Sql(
"""
UPDATE LibraryFolder
SET ParentId = (
SELECT KeeperId FROM __LibraryFolderDedupe WHERE LoserId = LibraryFolder.ParentId)
WHERE ParentId IN (SELECT LoserId FROM __LibraryFolderDedupe)
""");
// A folder parented on its OWN duplicate would become its own parent above. Unreachable from
// any code path today, but this is a tree the scanner walks, so remove the cycle class rather
// than reason about it.
migrationBuilder.Sql("UPDATE LibraryFolder SET ParentId = NULL WHERE ParentId = Id");
// Clear the survivor's etag. Which duplicate the scanner was actually writing to was
// arbitrary, so MIN(Id)'s etag may describe a stale view of the folder and would suppress the
// next rescan. A null etag costs exactly one rescan and cannot be wrong.
migrationBuilder.Sql(
"UPDATE LibraryFolder SET Etag = NULL WHERE Id IN (SELECT KeeperId FROM __LibraryFolderDedupe)");
// ImageFolderDuration is 1:1 with LibraryFolder (unique index on LibraryFolderId), so the
// duplicates' rows cannot all be repointed. Keep the keeper's own setting when it has one;
// otherwise promote exactly one loser's (lowest Id) and drop the rest.
migrationBuilder.Sql("DROP TABLE IF EXISTS __LibraryFolderDedupeIfd");
migrationBuilder.Sql(
"""
CREATE TABLE __LibraryFolderDedupeIfd (
KeeperId INTEGER NOT NULL PRIMARY KEY,
IfdId INTEGER NOT NULL
)
""");
migrationBuilder.Sql(
"""
INSERT INTO __LibraryFolderDedupeIfd (KeeperId, IfdId)
SELECT d.KeeperId, MIN(i.Id)
FROM __LibraryFolderDedupe d
INNER JOIN ImageFolderDuration i ON i.LibraryFolderId = d.LoserId
WHERE NOT EXISTS (
SELECT 1 FROM ImageFolderDuration ki WHERE ki.LibraryFolderId = d.KeeperId)
GROUP BY d.KeeperId
""");
migrationBuilder.Sql(
"""
DELETE FROM ImageFolderDuration
WHERE LibraryFolderId IN (SELECT LoserId FROM __LibraryFolderDedupe)
AND Id NOT IN (SELECT IfdId FROM __LibraryFolderDedupeIfd)
""");
migrationBuilder.Sql(
"""
UPDATE ImageFolderDuration
SET LibraryFolderId = (
SELECT KeeperId FROM __LibraryFolderDedupeIfd WHERE IfdId = ImageFolderDuration.Id)
WHERE Id IN (SELECT IfdId FROM __LibraryFolderDedupeIfd)
""");
migrationBuilder.Sql(
"DELETE FROM LibraryFolder WHERE Id IN (SELECT LoserId FROM __LibraryFolderDedupe)");
migrationBuilder.Sql("DROP TABLE __LibraryFolderDedupeIfd");
migrationBuilder.Sql("DROP TABLE __LibraryFolderDedupe");
migrationBuilder.AddColumn<string>(
name: "PathHash",
table: "LibraryFolder",
type: "TEXT",
maxLength: 64,
nullable: true);
// Existing rows keep a null hash on purpose: a unique index treats nulls as distinct, so the
// index applies cleanly to any database, and LibraryRepository.GetOrAddFolder heals each row
// (SHA-256 of Path) the first time a scan touches it. Those rows are deduplicated above and are
// still found by the Path lookup, so no insert can race them in the meantime.
//
// Create-then-drop rather than EF's scaffolded drop-then-create, matching the MySql copy, where
// InnoDB refuses to drop the foreign key's only backing index.
migrationBuilder.CreateIndex(
name: "IX_LibraryFolder_LibraryPathId_PathHash",
table: "LibraryFolder",
columns: new[] { "LibraryPathId", "PathHash" },
unique: true);
migrationBuilder.DropIndex(
name: "IX_LibraryFolder_LibraryPathId",
table: "LibraryFolder");
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateIndex(
name: "IX_LibraryFolder_LibraryPathId",
table: "LibraryFolder",
column: "LibraryPathId");
migrationBuilder.DropIndex(
name: "IX_LibraryFolder_LibraryPathId_PathHash",
table: "LibraryFolder");
migrationBuilder.DropColumn(
name: "PathHash",
table: "LibraryFolder");
}
}
}
@@ -1298,12 +1298,17 @@ namespace ErsatzTV.Infrastructure.Sqlite.Migrations
b.Property<string>("Path")
.HasColumnType("TEXT");
b.Property<string>("PathHash")
.HasMaxLength(64)
.HasColumnType("TEXT");
b.HasKey("Id");
b.HasIndex("LibraryPathId");
b.HasIndex("ParentId");
b.HasIndex("LibraryPathId", "PathHash")
.IsUnique();
b.ToTable("LibraryFolder", (string)null);
});
@@ -1,4 +1,4 @@
using ErsatzTV.Core.Domain;
using ErsatzTV.Core.Domain;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
@@ -10,6 +10,19 @@ public class LibraryFolderConfiguration : IEntityTypeConfiguration<LibraryFolder
{
builder.ToTable("LibraryFolder");
// ersatztv#491: GetOrAddFolder is a check-then-insert, so two callers racing the same folder both
// miss the lookup and both insert. The unique index makes the duplicate impossible at the storage
// layer (the loser gets a constraint violation the repository catches and re-reads).
// Indexed on PathHash rather than Path because Path is unbounded (MySQL longtext, which cannot be
// indexed without a prefix length, and whose default collation is case-INsensitive — a prefix
// index would also false-collide sibling folders differing only in case on a case-sensitive
// filesystem). This mirrors the existing MediaFile.Path/PathHash pair.
builder.Property(f => f.PathHash)
.HasMaxLength(64);
builder.HasIndex(f => new { f.LibraryPathId, f.PathHash })
.IsUnique();
builder.HasOne(f => f.Parent)
.WithMany(p => p.Children)
.HasForeignKey(f => f.ParentId)
@@ -1,5 +1,6 @@
using System.IO.Abstractions;
using Dapper;
using ErsatzTV.Core;
using ErsatzTV.Core.Domain;
using ErsatzTV.Core.Interfaces.Repositories;
using ErsatzTV.Infrastructure.Extensions;
@@ -110,15 +111,35 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
if (knownFolder.IsNone)
{
await dbContext.LibraryFolders.AddAsync(
new LibraryFolder
{
Path = path,
Etag = etag,
LibraryPathId = libraryPath.Id
});
var newFolder = new LibraryFolder
{
Path = path,
PathHash = PathUtils.GetPathHash(path),
Etag = etag,
LibraryPathId = libraryPath.Id
};
await dbContext.SaveChangesAsync();
try
{
await dbContext.LibraryFolders.AddAsync(newFolder);
await dbContext.SaveChangesAsync();
}
catch (DbUpdateException ex) when (TvContext.IsUniqueConstraintViolation(ex))
{
// ersatztv#491: a concurrent caller created this folder between the caller's lookup and
// this insert. The etag write is the whole point of the call, so apply it to the winner's
// row rather than failing the scan.
dbContext.Entry(newFolder).State = EntityState.Detached;
LibraryFolder winner = await GetFolder(dbContext, libraryPath.Id, path);
if (winner is null)
{
throw;
}
await dbContext.Connection.ExecuteAsync(
"UPDATE LibraryFolder SET Etag = @Etag WHERE Id = @Id",
new { winner.Id, Etag = etag });
}
}
}
@@ -174,11 +195,74 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
// local scan path (via GetLibrary) and is null on the remote (Jellyfin) sync path, which used
// to NRE every Jellyfin music-video scan here (ersatztv#488). The local scanners already hit
// the db once per folder via GetParentFolderId, so this adds no new query pattern.
LibraryFolder knownFolder = await dbContext.LibraryFolders
.AsNoTracking()
.Filter(f => f.LibraryPathId == libraryPath.Id && f.Path == folder)
.FirstOrDefaultAsync()
?? CreateNewFolder(libraryPath, maybeParentFolder, folder);
LibraryFolder knownFolder = await GetFolder(dbContext, libraryPath.Id, folder);
// add new folder to library path
if (knownFolder is null)
{
LibraryFolder newFolder = CreateNewFolder(libraryPath, maybeParentFolder, folder);
try
{
await dbContext.LibraryFolders.AddAsync(newFolder);
await dbContext.SaveChangesAsync();
knownFolder = newFolder;
}
catch (DbUpdateException ex) when (TvContext.IsUniqueConstraintViolation(ex))
{
// ersatztv#491: the lookup above is not atomic with this insert, so a concurrent caller
// scanning the same folder can slip its row in between. The unique index on
// (LibraryPathId, PathHash) turns that lost race into a constraint violation instead of a
// duplicate row; adopt the winner's row rather than failing the scan. Detach first so the
// failed insert is not retried by anything reusing this context.
dbContext.Entry(newFolder).State = EntityState.Detached;
knownFolder = await GetFolder(dbContext, libraryPath.Id, folder);
if (knownFolder is null)
{
// no winner to adopt — the violation came from somewhere else, so don't swallow it
throw;
}
}
}
else if (string.IsNullOrEmpty(knownFolder.PathHash))
{
// Heal a row created before the PathHash column existed, so it participates in the unique
// index from here on (a null hash is distinct from every other value, so it does not).
//
// This is opportunistic maintenance on a hot scan path, so it must never be able to abort a
// scan. It goes through EF rather than a raw Dapper UPDATE precisely so a collision surfaces
// as a classifiable DbUpdateException instead of a bare provider exception, and a lost heal
// is simply left for the next scan. Reachable only if some other row already owns
// (LibraryPathId, hash) — a legacy duplicate the migration's dedupe could not see (e.g. one
// with a NULL Path, which `NULL = NULL` excludes from its grouping).
string pathHash = PathUtils.GetPathHash(folder);
LibraryFolder tracked = null;
try
{
// the predicate must agree with the IsNullOrEmpty guard above, or a PathHash = '' row would
// enter this branch, match nothing, and silently never heal
tracked = await dbContext.LibraryFolders
.FirstOrDefaultAsync(f => f.Id == knownFolder.Id && (f.PathHash == null || f.PathHash == ""));
if (tracked is not null)
{
tracked.PathHash = pathHash;
await dbContext.SaveChangesAsync();
knownFolder.PathHash = pathHash;
}
}
catch (DbUpdateException ex) when (
TvContext.IsUniqueConstraintViolation(ex) || ex is DbUpdateConcurrencyException)
{
// Either another row already owns this hash, or the row was deleted out from under us by a
// concurrent library edit (DbUpdateConcurrencyException derives from DbUpdateException but
// carries no provider exception, so the classifier does NOT recognize it). Both mean "the
// heal is moot" — leave the row unhealed rather than fail the scan, per the invariant above.
if (tracked is not null)
{
// drop the failed change so it cannot be replayed by a later save on this context
dbContext.Entry(tracked).State = EntityState.Detached;
}
}
}
// update parent folder if not present
foreach (int parentFolder in maybeParentFolder)
@@ -193,13 +277,6 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
}
}
// add new folder to library path
if (knownFolder.Id < 1)
{
await dbContext.LibraryFolders.AddAsync(knownFolder);
await dbContext.SaveChangesAsync();
}
return knownFolder;
}
@@ -221,6 +298,71 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
new { Path = normalizedLibraryPath, libraryPath.Id });
}
/// <summary>
/// The in-memory half of <see cref="GetFolder" />, lifted out so the ordinal decision is pinned by a
/// test with no database at all: the collation behaviour that makes it necessary is MySQL-only, so a
/// SQLite-backed test cannot exercise it (SQLite's <c>=</c> on TEXT is already binary and never
/// returns the case-differing candidate). Given the candidates a case-INsensitive server may return,
/// pick the one whose path matches ordinally; callers pass them lowest <c>Id</c> first.
/// </summary>
public static LibraryFolder ResolveExact(IReadOnlyList<LibraryFolder> candidates, string folder)
{
for (var i = 0; i < candidates.Count; i++)
{
if (string.Equals(candidates[i].Path, folder, StringComparison.Ordinal))
{
return candidates[i];
}
}
return null;
}
/// <summary>
/// Resolve a folder by its exact path within a library path.
/// <para>
/// The SQL equality is only a *narrowing* filter, not the identity test. On MySQL, `Path` is a
/// `longtext` whose collation the schema does not pin — only the `utf8mb4` charset — so the
/// effective comparison is whatever the server defaults to, and it differs from byte equality:
/// <list type="bullet">
/// <item>
/// always case-INsensitive: both plausible defaults are `_ci` (8.4 verified:
/// `utf8mb4_0900_ai_ci`; older servers `utf8mb4_general_ci`), which is why
/// <see cref="TvContext.CaseInsensitiveCollation" /> exists at all;
/// </item>
/// <item>
/// possibly PAD SPACE, making trailing spaces insignificant — true of
/// `utf8mb4_general_ci`, but NOT of `utf8mb4_0900_ai_ci`, which is NO PAD. So this axis
/// is server-dependent rather than guaranteed, and must be tolerated rather than
/// assumed either way.
/// </item>
/// </list>
/// `Path = @folder` can therefore also match siblings differing only in case, or (on a PAD
/// SPACE server) in trailing whitespace — all legal on a case-sensitive filesystem, and all
/// preserved by the #491 migration. Crucially the SQL predicate is a *superset*: every such
/// quirk makes it more permissive, never less, so it cannot miss a byte-exact match. Identity
/// is then settled in memory by <see cref="ResolveExact" /> with an ORDINAL comparison,
/// matching <c>PathUtils.GetPathHash</c>, which hashes the exact bytes. Without this the lookup
/// and the hash disagree, and the PathHash heal could stamp one sibling's hash onto the other's
/// row.
/// </para>
/// <para>
/// Ordered by Id so the result is deterministic: an unordered <c>FirstOrDefault</c> may return a
/// different candidate run to run as the query plan changes (adding the composite index alone
/// can flip it), which would make the heal non-idempotent.
/// </para>
/// </summary>
private static async Task<LibraryFolder> GetFolder(TvContext dbContext, int libraryPathId, string folder)
{
List<LibraryFolder> candidates = await dbContext.LibraryFolders
.AsNoTracking()
.Filter(f => f.LibraryPathId == libraryPathId && f.Path == folder)
.OrderBy(f => f.Id)
.ToListAsync();
return ResolveExact(candidates, folder);
}
private static LibraryFolder CreateNewFolder(LibraryPath libraryPath, Option<int> maybeParentFolder, string folder)
{
int? parentId = null;
@@ -232,6 +374,7 @@ public class LibraryRepository(IFileSystem fileSystem, IDbContextFactory<TvConte
return new LibraryFolder
{
Path = folder,
PathHash = PathUtils.GetPathHash(folder),
Etag = null,
LibraryPathId = libraryPath.Id,
ParentId = parentId
+7
View File
@@ -24,6 +24,7 @@ using ErsatzTV.Infrastructure.Emby;
using ErsatzTV.Infrastructure.Images;
using ErsatzTV.Infrastructure.Jellyfin;
using ErsatzTV.Infrastructure.Metadata;
using ErsatzTV.Infrastructure.MySql.Data;
using ErsatzTV.Infrastructure.Plex;
using ErsatzTV.Infrastructure.Runtime;
using ErsatzTV.Infrastructure.Search;
@@ -152,10 +153,15 @@ public class Program
}
});
// Keep this block in sync with ErsatzTV/Startup.cs — the scanner is a SEPARATE process
// (launched by CallLibraryScannerHandler), so any TvContext provider static the host wires
// has to be wired here too or it silently keeps its default in every scan.
// ProviderStaticsWiringTests enforces that parity.
if (databaseProvider == Provider.Sqlite.Name)
{
TvContext.LastInsertedRowId = "last_insert_rowid()";
TvContext.CaseInsensitiveCollation = "NOCASE";
TvContext.IsUniqueConstraintViolation = SqliteErrorClassifier.IsUniqueConstraintViolation;
SqlMapper.AddTypeHandler(new DateTimeOffsetHandler());
SqlMapper.AddTypeHandler(new GuidHandler());
@@ -166,6 +172,7 @@ public class Program
{
TvContext.LastInsertedRowId = "last_insert_id()";
TvContext.CaseInsensitiveCollation = "utf8mb4_general_ci";
TvContext.IsUniqueConstraintViolation = MySqlErrorClassifier.IsUniqueConstraintViolation;
}
services.AddHttpClient();
@@ -0,0 +1,302 @@
using ErsatzTV.Core;
using ErsatzTV.Core.Domain;
using ErsatzTV.Infrastructure.Data;
using ErsatzTV.Infrastructure.Data.Repositories;
using ErsatzTV.Infrastructure.Sqlite.Data;
using ErsatzTV.Tests.Support;
using LanguageExt;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
using NSubstitute;
using NUnit.Framework;
using Shouldly;
using IFileSystem = System.IO.Abstractions.IFileSystem;
namespace ErsatzTV.Tests.Integration;
/// <summary>
/// ersatztv#491: <c>ILibraryRepository.GetOrAddFolder</c> is a check-then-insert, so two callers
/// racing the same <c>(LibraryPathId, Path)</c> both miss the lookup and both insert. The fix is a
/// unique index on <c>(LibraryPathId, PathHash)</c> plus a catch-and-re-read in the repository, so
/// the loser adopts the winner's row instead of creating a duplicate.
/// </summary>
[TestFixture]
public class LibraryFolderConcurrencyTests
{
private const string LibraryPathValue = "/data/music";
private const string FolderPath = "/data/music/artist1";
private static LibraryRepository Repository(IDbContextFactory<TvContext> factory) =>
new(Substitute.For<IFileSystem>(), factory);
private static async Task<int> SeedLibraryPath(Func<TvContext> createContext)
{
await using TvContext context = createContext();
var libraryPath = new LibraryPath { Path = LibraryPathValue };
await context.LibraryPaths.AddAsync(libraryPath);
await context.SaveChangesAsync();
return libraryPath.Id;
}
private static async Task<int> FolderCount(Func<TvContext> createContext, int libraryPathId, string path)
{
await using TvContext context = createContext();
return await context.LibraryFolders.CountAsync(f => f.LibraryPathId == libraryPathId && f.Path == path);
}
private static async Task InsertFolderRaw(
SharedCacheTvContext db,
int libraryPathId,
string path,
string pathHash,
CancellationToken cancellationToken = default)
{
await using SqliteConnection connection = db.OpenConnection();
await using SqliteCommand command = connection.CreateCommand();
command.CommandText =
"INSERT INTO \"LibraryFolder\" (\"LibraryPathId\", \"Path\", \"PathHash\", \"Etag\", \"ParentId\") " +
"VALUES ($libraryPathId, $path, $pathHash, NULL, NULL)";
command.Parameters.AddWithValue("$libraryPathId", libraryPathId);
command.Parameters.AddWithValue("$path", path);
command.Parameters.AddWithValue("$pathHash", (object?)pathHash ?? DBNull.Value);
await command.ExecuteNonQueryAsync(cancellationToken);
}
/// <summary>
/// Simulates the concurrent "winner": exactly once, on a SEPARATE connection, insert the same
/// folder and commit — AFTER the intercepted context read the (stale) absent lookup but BEFORE its
/// own INSERT runs. This interposes the race deterministically instead of hoping for a timing
/// window. <see cref="Fired" /> proves the race actually happened (non-vacuity).
/// </summary>
private sealed class InsertConflictingFolderOnce(SharedCacheTvContext db, int libraryPathId, string path)
: SaveChangesInterceptor
{
private int _fired;
public int Fired => _fired;
public override async ValueTask<InterceptionResult<int>> SavingChangesAsync(
DbContextEventData eventData,
InterceptionResult<int> result,
CancellationToken cancellationToken = default)
{
if (Interlocked.Exchange(ref _fired, 1) == 0)
{
await InsertFolderRaw(db, libraryPathId, path, PathUtils.GetPathHash(path), cancellationToken);
}
return result;
}
}
/// <summary>Counts insert attempts so the multi-threaded test can prove it really raced.</summary>
private sealed class CountSaveAttempts : SaveChangesInterceptor
{
private int _attempts;
public int Attempts => _attempts;
public override ValueTask<InterceptionResult<int>> SavingChangesAsync(
DbContextEventData eventData,
InterceptionResult<int> result,
CancellationToken cancellationToken = default)
{
Interlocked.Increment(ref _attempts);
return ValueTask.FromResult(result);
}
}
// ----- Negative control #1: the index itself. Without the new unique index this test fails, because
// the second insert simply succeeds and there is no violation to classify. -----
[Test]
public async Task Duplicate_LibraryFolder_Insert_Throws_A_Classified_UniqueViolation()
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-index");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
string hash = PathUtils.GetPathHash(FolderPath);
await InsertFolderRaw(db, libraryPathId, FolderPath, hash);
await using TvContext context = db.CreateContext();
await context.LibraryFolders.AddAsync(
new LibraryFolder { LibraryPathId = libraryPathId, Path = FolderPath, PathHash = hash });
DbUpdateException ex = await Should.ThrowAsync<DbUpdateException>(() => context.SaveChangesAsync());
SqliteErrorClassifier.IsUniqueConstraintViolation(ex).ShouldBeTrue();
// and the classifier is not a blanket "true"
SqliteErrorClassifier.IsUniqueConstraintViolation(
new DbUpdateException("nope", new InvalidOperationException())).ShouldBeFalse();
}
// The migration leaves pre-#491 rows with a null hash; a unique index treats nulls as distinct, so
// applying the index to an existing database can never fail on them. Documents that premise.
[Test]
public async Task Legacy_Null_PathHash_Rows_Do_Not_Collide()
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-nulls");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
await InsertFolderRaw(db, libraryPathId, "/data/music/a", null);
await InsertFolderRaw(db, libraryPathId, "/data/music/b", null);
await using TvContext context = db.CreateContext();
(await context.LibraryFolders.CountAsync(f => f.LibraryPathId == libraryPathId)).ShouldBe(2);
}
// ----- The deterministic cross-connection race through the real repository -----
[Test]
public async Task GetOrAddFolder_Losing_The_Race_Adopts_The_Winner_Instead_Of_Duplicating()
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-race");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
var libraryPath = new LibraryPath { Id = libraryPathId, Path = LibraryPathValue, LibraryFolders = null };
var racer = new InsertConflictingFolderOnce(db, libraryPathId, FolderPath);
LibraryRepository repository = Repository(db.Factory(racer));
LibraryFolder result = await repository.GetOrAddFolder(libraryPath, Option<int>.None, FolderPath);
racer.Fired.ShouldBe(1); // the race genuinely occurred — this assertion is the vacuity guard
result.ShouldNotBeNull();
result.Id.ShouldBeGreaterThan(0);
result.Path.ShouldBe(FolderPath);
(await FolderCount(db.CreateContext, libraryPathId, FolderPath)).ShouldBe(1);
// the returned row is the winner's persisted row, not a phantom
await using TvContext context = db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.LibraryPathId == libraryPathId);
persisted.Id.ShouldBe(result.Id);
persisted.PathHash.ShouldBe(PathUtils.GetPathHash(FolderPath));
}
// The loser must still apply the parent id it was asked to set — to the WINNER's row.
[Test]
public async Task GetOrAddFolder_Losing_The_Race_Still_Persists_The_ParentId()
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-race-parent");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
var libraryPath = new LibraryPath { Id = libraryPathId, Path = LibraryPathValue, LibraryFolders = null };
LibraryFolder parent = await Repository(db.Factory())
.GetOrAddFolder(libraryPath, Option<int>.None, LibraryPathValue);
var racer = new InsertConflictingFolderOnce(db, libraryPathId, FolderPath);
LibraryFolder result = await Repository(db.Factory(racer))
.GetOrAddFolder(libraryPath, Option<int>.Some(parent.Id), FolderPath);
racer.Fired.ShouldBe(1);
result.ParentId.ShouldBe(parent.Id);
await using TvContext context = db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.Path == FolderPath);
persisted.Id.ShouldBe(result.Id);
persisted.ParentId.ShouldBe(parent.Id);
}
// ----- Negative control #2: invert the real condition (the provider classifier) and the SAME race
// must blow up, proving the catch in GetOrAddFolder is load-bearing rather than decorative. -----
[Test]
public async Task GetOrAddFolder_Rethrows_When_The_Provider_Does_Not_Classify_The_Violation()
{
Func<DbUpdateException, bool> original = TvContext.IsUniqueConstraintViolation;
try
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-negctl");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
var libraryPath = new LibraryPath { Id = libraryPathId, Path = LibraryPathValue, LibraryFolders = null };
var racer = new InsertConflictingFolderOnce(db, libraryPathId, FolderPath);
LibraryRepository repository = Repository(db.Factory(racer));
TvContext.IsUniqueConstraintViolation = _ => false;
await Should.ThrowAsync<DbUpdateException>(
() => repository.GetOrAddFolder(libraryPath, Option<int>.None, FolderPath));
racer.Fired.ShouldBe(1);
}
finally
{
TvContext.IsUniqueConstraintViolation = original;
}
}
// ----- N threads x rounds over a single (LibraryPathId, Path) -----
[Test]
public async Task Concurrent_GetOrAddFolder_Never_Produces_Duplicate_Rows()
{
const int threads = 8;
const int rounds = 10;
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-threads");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
var counter = new CountSaveAttempts();
IDbContextFactory<TvContext> factory = db.Factory(counter);
for (var round = 0; round < rounds; round++)
{
string path = $"{LibraryPathValue}/round{round}";
using var gate = new Barrier(threads);
var tasks = new Task<LibraryFolder>[threads];
for (var thread = 0; thread < threads; thread++)
{
tasks[thread] = Task.Run(async () =>
{
// every thread carries its own detached LibraryPath, as the scanners do
var libraryPath = new LibraryPath
{
Id = libraryPathId, Path = LibraryPathValue, LibraryFolders = null
};
gate.SignalAndWait();
return await Repository(factory).GetOrAddFolder(libraryPath, Option<int>.None, path);
});
}
LibraryFolder[] results = await Task.WhenAll(tasks);
// every caller got the same single row back...
results.Select(f => f.Id).Distinct().Count().ShouldBe(1);
results[0].Id.ShouldBeGreaterThan(0);
// ...and exactly one row exists for it
(await FolderCount(db.CreateContext, libraryPathId, path)).ShouldBe(1);
}
await using TvContext context = db.CreateContext();
(await context.LibraryFolders.CountAsync(f => f.LibraryPathId == libraryPathId)).ShouldBe(rounds);
// Vacuity guard: one insert attempt per round would mean the threads never actually collided and
// the test proved nothing. More attempts than rounds means at least one caller lost the race and
// was rescued by the index + catch.
counter.Attempts.ShouldBeGreaterThan(rounds);
}
// ----- SetEtag is the repository's other check-then-insert on LibraryFolder -----
[Test]
public async Task SetEtag_Losing_The_Race_Updates_The_Winner_Instead_Of_Duplicating()
{
await using SharedCacheTvContext db = await SharedCacheTvContext.CreateAsync("etv491-setetag");
int libraryPathId = await SeedLibraryPath(db.CreateContext);
var libraryPath = new LibraryPath { Id = libraryPathId, Path = LibraryPathValue, LibraryFolders = null };
var racer = new InsertConflictingFolderOnce(db, libraryPathId, FolderPath);
LibraryRepository repository = Repository(db.Factory(racer));
await repository.SetEtag(libraryPath, Option<LibraryFolder>.None, FolderPath, "etag-1");
racer.Fired.ShouldBe(1);
await using TvContext context = db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.Path == FolderPath);
persisted.Etag.ShouldBe("etag-1");
}
}
@@ -0,0 +1,445 @@
using Dapper;
using ErsatzTV.Infrastructure;
using ErsatzTV.Infrastructure.Data;
using ErsatzTV.Infrastructure.MySql.Data;
using ErsatzTV.Infrastructure.Sqlite.Data;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging.Abstractions;
using MySqlConnector;
using NUnit.Framework;
using Shouldly;
namespace ErsatzTV.Tests.Integration;
public enum TestProvider
{
Sqlite,
MySql
}
/// <summary>
/// ersatztv#491: the unique index on <c>LibraryFolder(LibraryPathId, PathHash)</c> ships with an
/// audit/cleanup of pre-existing duplicate rows (reachable before #488, when the folder lookup read a
/// scan-start in-memory snapshot and could not see a folder created earlier in the same scan). This
/// drives the REAL migration against a database seeded at the previous migration, so the cleanup SQL
/// is exercised rather than restated.
/// <para>
/// Runs against BOTH providers from ONE fixture body. The dedupe deletes rows irreversibly and its
/// correctness turns on string-comparison semantics that differ per provider — two MySQL-only
/// collation defects escaped review in the #491 session (a case-insensitive grouping, then a PAD
/// SPACE one), and neither was reachable from a SQLite-only test, nor from CI's MySQL job, which
/// only applies migrations to a fresh EMPTY database and so executes no dedupe rows at all.
/// Parameterizing one fixture is what makes "the two providers agree" a checked property rather
/// than an assumption; a separate MySQL-only copy would drift and recreate the gap.
/// </para>
/// <para>
/// MySQL needs a live server, supplied via <c>ETV_TEST_MYSQL_CONNECTION</c>. Without it the MySQL
/// fixture <b>ignores</b> — a visible skip, never a silent pass — so local runs need no MySQL. CI
/// sets <c>ETV_REQUIRE_MYSQL_TESTS=1</c>, which turns that skip into a hard failure, so the gate
/// cannot quietly degrade into "connected to nothing and passed".
/// </para>
/// </summary>
[TestFixture(TestProvider.Sqlite)]
[TestFixture(TestProvider.MySql)]
[NonParallelizable]
public class LibraryFolderDedupeMigrationTests(TestProvider provider)
{
// the migration immediately preceding Add_LibraryFolder_PathHash_UniqueIndex
private const string PreviousMigration = "Add_Channel_Origin";
private const string MySqlConnectionVariable = "ETV_TEST_MYSQL_CONNECTION";
private const string MySqlRequiredVariable = "ETV_REQUIRE_MYSQL_TESTS";
private string _databasePath = null!;
private string? _mySqlConnectionString;
// Seeding writes deliberately partial object graphs (a LibraryPath with no Library, a MediaFile with
// no MediaVersion), so it runs with foreign keys OFF.
private DbContextOptions<TvContext> _seedOptions = null!;
// The migration itself runs with foreign keys ON, matching production. This matters: the single most
// dangerous statement in the #491 migration is DELETE FROM LibraryFolder against two Restrict foreign
// keys (MediaFile.LibraryFolderId, LibraryFolder.ParentId). With enforcement off, a wrong repoint
// order would still pass; with it on, the delete fails loudly.
private DbContextOptions<TvContext> _migrateOptions = null!;
[SetUp]
public async Task SetUp()
{
if (provider is TestProvider.Sqlite)
{
TvContext.IsSqlite = true;
TvContext.LastInsertedRowId = "last_insert_rowid()";
TvContext.CaseInsensitiveCollation = "NOCASE";
TvContext.IsUniqueConstraintViolation = SqliteErrorClassifier.IsUniqueConstraintViolation;
_databasePath = Path.Combine(Path.GetTempPath(), $"etv491-{Guid.NewGuid():N}.sqlite3");
_seedOptions = SqliteOptions(foreignKeys: false);
_migrateOptions = SqliteOptions(foreignKeys: true);
return;
}
string? baseConnectionString = Environment.GetEnvironmentVariable(MySqlConnectionVariable);
if (string.IsNullOrWhiteSpace(baseConnectionString))
{
string message =
$"{MySqlConnectionVariable} is not set, so the MySql half of the #491 dedupe fixture cannot "
+ "run. The dedupe deletes rows irreversibly and its correctness is provider-specific, so "
+ "this coverage is not optional in CI.";
// A skip is fine locally; in CI it is the very failure mode this fixture exists to prevent.
if (IsTrue(Environment.GetEnvironmentVariable(MySqlRequiredVariable)))
{
Assert.Fail($"{message} {MySqlRequiredVariable} is set, so this is a failure, not a skip.");
}
Assert.Ignore($"{message} Set it to run this locally.");
}
// A database of our own, NOT the one the surrounding CI step migrates, and a FRESH one per test:
// isolation by construction. A name that has never been used cannot contain another test's rows,
// so no wipe has to succeed for the fixture to be correct. It is not created here — the test's own
// MigrateAsync(PreviousMigration) creates it, which keeps EF the single owner of the schema.
_mySqlConnectionString =
new MySqlConnectionStringBuilder(baseConnectionString) { Database = $"etv491_{Guid.NewGuid():N}" }
.ConnectionString;
TvContext.IsSqlite = false;
TvContext.LastInsertedRowId = "last_insert_id()";
TvContext.CaseInsensitiveCollation = "utf8mb4_general_ci";
TvContext.IsUniqueConstraintViolation = MySqlErrorClassifier.IsUniqueConstraintViolation;
// AutoDetect opens its own connection, so resolve the version once and share it between the two
// option sets instead of connecting twice per test.
ServerVersion serverVersion = ServerVersion.AutoDetect(_mySqlConnectionString);
_seedOptions = MySqlOptions(serverVersion);
_migrateOptions = MySqlOptions(serverVersion);
}
/// <summary>
/// Drop this test's database and clear the connection pool that was keyed to it.
/// <para>
/// Both halves are required, and earlier revisions of this fixture each got one wrong. Measured
/// against a real 8.4 server:
/// </para>
/// <list type="number">
/// <item>
/// <b>A pooled session outlives <c>DROP DATABASE</c>.</b> Reopening the dropped database's
/// connection string succeeds — MySqlConnector hands back the still-alive session whose
/// default schema is gone — so whether a later caller sees success or <c>Unknown database</c>
/// depends on whether the pool reuses that session or opens a fresh one (a fresh handshake
/// names the dropped schema and fails 1049). <c>ClearPool</c> after the drop removes it.
/// Note this hazard needs the connection string to be REUSED after the drop, which a
/// never-repeated database name already makes impossible; clearing the pool is the belt to
/// that brace, and closes the leak below.
/// </item>
/// <item>
/// <b>An uncleared pool leaks a server connection per test.</b> MySqlConnector keys pools by
/// connection string, so a fresh database name means a fresh pool; left uncleared it leaked
/// ~1 server thread per iteration and eventually exhausted <c>max_connections</c>.
/// <c>ClearPoolAsync</c> on that exact connection string fixes it completely — measured at
/// 0 leaked threads over 30 iterations — so per-test isolation costs nothing. A previous
/// revision instead collapsed to one shared database to stop the leak; that traded isolation
/// for a wipe that has to succeed, and when it silently did not, the second test seeded on
/// top of the first's rows and failed with a duplicate primary key.
/// </item>
/// </list>
/// <para>
/// EF owns the drop: <c>EnsureDeletedAsync</c> is guarded (a no-op when the database is absent,
/// unlike a raw <c>DROP DATABASE</c>) and uses the same connection string EF migrated with.
/// </para>
/// </summary>
private async Task DropMySqlDatabase()
{
await using (TvContext context = MigrateContext())
{
await context.Database.EnsureDeletedAsync();
}
await using var probe = new MySqlConnection(_mySqlConnectionString);
await MySqlConnection.ClearPoolAsync(probe);
}
[TearDown]
public async Task TearDown()
{
if (provider is TestProvider.Sqlite)
{
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (string path in new[] { _databasePath, $"{_databasePath}-wal", $"{_databasePath}-shm" })
{
if (File.Exists(path))
{
File.Delete(path);
}
}
return;
}
if (_mySqlConnectionString is not null)
{
await DropMySqlDatabase();
_mySqlConnectionString = null;
}
}
[Test]
public async Task Migration_Collapses_Duplicate_Folders_And_Repoints_Their_Dependents()
{
await using (TvContext context = SeedContext())
{
await context.Database.MigrateAsync(PreviousMigration);
await using SeedSession seed = await SeedSession.OpenAsync(context, provider);
// one library path with the SAME folder recorded three times (ids 1, 2, 3), an unrelated
// folder (4), a child parented on one of the duplicates (5), a case-differing sibling (6),
// a folder parented on its own duplicate (7/8 — the cycle the ParentId null-out guards), and
// a sibling differing only by a TRAILING SPACE (9/10). The last two pairs are distinct legal
// directories on Linux that hash differently, so the unique index accepts both and the dedupe
// must not collapse them. On MySql a case-insensitive grouping deletes 6, and a PAD SPACE one
// deletes 10 — utf8mb4_bin, the obvious fix for the first, is itself PAD SPACE, which is why
// the migration groups on CONVERT(Path USING binary) instead.
await seed.ExecuteAsync(
"INSERT INTO LibraryPath (Id, LibraryId, Path) VALUES (1, 1, '/data/music')");
await seed.ExecuteAsync(
"""
INSERT INTO LibraryFolder (Id, LibraryPathId, Path, ParentId, Etag) VALUES
(1, 1, '/data/music/artist1', NULL, 'etag-keeper'),
(2, 1, '/data/music/artist1', NULL, 'etag-dupe-a'),
(3, 1, '/data/music/artist1', NULL, 'etag-dupe-b'),
(4, 1, '/data/music/artist2', NULL, NULL),
(5, 1, '/data/music/artist1/album', 3, NULL),
(6, 1, '/data/music/ARTIST2', NULL, NULL),
(7, 1, '/data/music/artist3', 8, NULL),
(8, 1, '/data/music/artist3', NULL, NULL),
(9, 1, '/data/music/pad', NULL, NULL),
(10, 1, '/data/music/pad ', NULL, NULL)
""");
// a media file on each duplicate and on each half of the trailing-space pair, plus an
// image-folder-duration on duplicates only
await seed.ExecuteAsync(
"""
INSERT INTO MediaFile (Id, Path, PathHash, MediaVersionId, LibraryFolderId) VALUES
(1, '/data/music/artist1/a.mkv', 'hash-a', 1, 1),
(2, '/data/music/artist1/b.mkv', 'hash-b', 2, 2),
(3, '/data/music/artist1/c.mkv', 'hash-c', 3, 3),
(4, '/data/music/pad/d.mkv', 'hash-d', 4, 9),
(5, '/data/music/pad /e.mkv', 'hash-e', 5, 10)
""");
await seed.ExecuteAsync(
"""
INSERT INTO ImageFolderDuration (Id, LibraryFolderId, DurationSeconds) VALUES
(1, 2, 30.0),
(2, 3, 45.0)
""");
}
await using (TvContext context = MigrateContext())
{
await AssertForeignKeysEnforced(context);
await context.Database.MigrateAsync();
}
await using (TvContext context = MigrateContext())
{
// Read the surviving rows once and assert in memory. Deliberately NOT `WHERE Path = '...'`:
// that predicate is itself collation-dependent (on MySQL it would also match the
// case-differing and trailing-space siblings), so an assertion written that way would quietly
// mean something different on each provider — the exact class of bug this fixture guards.
List<FolderRow> folders = (await context.Connection.QueryAsync<FolderRow>(
"SELECT Id, Path, ParentId, Etag FROM LibraryFolder ORDER BY Id")).ToList();
// survivors: keeper 1, artist2 (4), the child (5), the case-differing sibling (6), artist3's
// keeper (7), and BOTH halves of the trailing-space pair (9, 10). Losers 2, 3 and 8 are gone.
folders.Select(f => f.Id).ToList().ShouldBe([1, 4, 5, 6, 7, 9, 10]);
// and their paths survive BYTE-exactly (ordinal comparison here, matching the hash)
folders.Select(f => f.Path).ToList().ShouldBe(
[
"/data/music/artist1",
"/data/music/artist2",
"/data/music/artist1/album",
"/data/music/ARTIST2",
"/data/music/artist3",
"/data/music/pad",
"/data/music/pad "
]);
// the keeper's etag is cleared: which duplicate the scanner was writing to was arbitrary, so
// MIN(Id)'s etag could suppress the rescan that repairs the collapsed folder
folders.Single(f => f.Id == 1).Etag.ShouldBeNull();
// the child folder is reparented off the deleted duplicate onto the keeper
folders.Single(f => f.Id == 5).ParentId.ShouldBe(1);
// the folder parented on its own duplicate did not become its own parent
folders.Single(f => f.Id == 7).ParentId.ShouldBeNull();
// every media file follows the keeper — nothing orphaned, nothing deleted — while the
// trailing-space pair's dependents stay attached to their OWN folder
List<int> mediaFolderIds = (await context.Connection.QueryAsync<int>(
"SELECT LibraryFolderId FROM MediaFile ORDER BY Id")).ToList();
mediaFolderIds.ShouldBe([1, 1, 1, 9, 10]);
// the keeper had no ImageFolderDuration, so exactly one duplicate's setting is promoted to it
// (the lowest id) and the rest are dropped — the 1:1 unique index cannot hold both
List<int> durationFolderIds = (await context.Connection.QueryAsync<int>(
"SELECT LibraryFolderId FROM ImageFolderDuration ORDER BY Id")).ToList();
durationFolderIds.ShouldBe([1]);
await AssertHelperTablesDropped(context);
await AssertUniqueIndexExists(context);
}
}
[Test]
public async Task Migration_Leaves_A_Database_Without_Duplicates_Alone()
{
// positive control: the cleanup must not touch rows that were already unique
await using (TvContext context = SeedContext())
{
await context.Database.MigrateAsync(PreviousMigration);
await using SeedSession seed = await SeedSession.OpenAsync(context, provider);
await seed.ExecuteAsync(
"INSERT INTO LibraryPath (Id, LibraryId, Path) VALUES (1, 1, '/data/music')");
await seed.ExecuteAsync(
"""
INSERT INTO LibraryFolder (Id, LibraryPathId, Path, ParentId, Etag) VALUES
(1, 1, '/data/music/artist1', NULL, 'etag-1'),
(2, 1, '/data/music/artist2', 1, 'etag-2')
""");
await seed.ExecuteAsync(
"INSERT INTO ImageFolderDuration (Id, LibraryFolderId, DurationSeconds) VALUES (1, 2, 30.0)");
}
await using (TvContext context = MigrateContext())
{
await AssertForeignKeysEnforced(context);
await context.Database.MigrateAsync();
}
await using (TvContext context = MigrateContext())
{
List<FolderRow> folders = (await context.Connection.QueryAsync<FolderRow>(
"SELECT Id, Path, ParentId, Etag FROM LibraryFolder ORDER BY Id")).ToList();
folders.Select(f => f.Id).ToList().ShouldBe([1, 2]);
folders.Single(f => f.Id == 1).Etag.ShouldBe("etag-1");
folders.Single(f => f.Id == 2).Etag.ShouldBe("etag-2");
folders.Single(f => f.Id == 2).ParentId.ShouldBe(1);
(await context.Connection.ExecuteScalarAsync<int>(
"SELECT LibraryFolderId FROM ImageFolderDuration WHERE Id = 1")).ShouldBe(2);
// existing rows keep a null hash — the index applies because nulls are distinct
(await context.Connection.ExecuteScalarAsync<int>(
"SELECT COUNT(*) FROM LibraryFolder WHERE PathHash IS NULL")).ShouldBe(2);
}
}
// Foreign-key enforcement is the whole point of splitting the seed and migrate contexts, so prove it
// took effect rather than trusting a connection-string keyword: a typo, a pragma reset or a leaked
// SET FOREIGN_KEY_CHECKS=0 would silently revert this to its weaker form and still pass everything.
private async Task AssertForeignKeysEnforced(TvContext context)
{
string sql = provider is TestProvider.Sqlite
? "PRAGMA foreign_keys"
: "SELECT @@SESSION.foreign_key_checks";
(await context.Connection.ExecuteScalarAsync<long>(sql)).ShouldBe(
1,
"the migration must run with foreign keys ENFORCED — otherwise DELETE FROM LibraryFolder is "
+ "not actually tested against the Restrict constraints it must not violate");
}
private async Task AssertHelperTablesDropped(TvContext context)
{
string sql = provider is TestProvider.Sqlite
? "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name LIKE '__LibraryFolderDedupe%'"
: "SELECT COUNT(*) FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() "
+ "AND TABLE_NAME LIKE '\\_\\_LibraryFolderDedupe%'";
(await context.Connection.ExecuteScalarAsync<int>(sql)).ShouldBe(0, "helper tables were left behind");
}
private async Task AssertUniqueIndexExists(TvContext context)
{
string sql = provider is TestProvider.Sqlite
? "SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' "
+ "AND name = 'IX_LibraryFolder_LibraryPathId_PathHash'"
: "SELECT COUNT(DISTINCT INDEX_NAME) FROM information_schema.STATISTICS "
+ "WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'LibraryFolder' "
+ "AND INDEX_NAME = 'IX_LibraryFolder_LibraryPathId_PathHash'";
(await context.Connection.ExecuteScalarAsync<int>(sql)).ShouldBe(1, "the unique index was not created");
}
private static bool IsTrue(string? value) =>
value is "1" || string.Equals(value, "true", StringComparison.OrdinalIgnoreCase);
private DbContextOptions<TvContext> SqliteOptions(bool foreignKeys) =>
new DbContextOptionsBuilder<TvContext>()
.UseSqlite(
$"Data Source={_databasePath};Foreign Keys={foreignKeys}",
o => o.MigrationsAssembly("ErsatzTV.Infrastructure.Sqlite"))
.Options;
private DbContextOptions<TvContext> MySqlOptions(ServerVersion serverVersion) =>
new DbContextOptionsBuilder<TvContext>()
.UseMySql(
_mySqlConnectionString,
serverVersion,
o => o.MigrationsAssembly("ErsatzTV.Infrastructure.MySql"))
.Options;
private TvContext SeedContext() => Create(_seedOptions);
private TvContext MigrateContext() => Create(_migrateOptions);
private static TvContext Create(DbContextOptions<TvContext> options) =>
new(
options,
NullLoggerFactory.Instance,
new SlowQueryInterceptor(NullLogger<SlowQueryInterceptor>.Instance));
// Settable properties rather than a positional record: SQLite hands back INTEGER as Int64 while MySQL
// hands back INT as Int32, and Dapper only narrows for property setters, not constructor matching.
private sealed class FolderRow
{
public int Id { get; set; }
public string Path { get; set; } = null!;
public int? ParentId { get; set; }
public string Etag { get; set; } = null!;
}
/// <summary>
/// Holds one connection open for the whole seeding block. SQLite disables foreign keys through a
/// connection-string keyword, but MySQL's <c>foreign_key_checks</c> is a SESSION variable — and
/// Dapper closes a connection it had to open itself, which would reset it between statements.
/// Opening explicitly keeps the session, and therefore the setting, alive across every insert.
/// </summary>
private sealed class SeedSession : IAsyncDisposable
{
private readonly TvContext _context;
private SeedSession(TvContext context) => _context = context;
public static async Task<SeedSession> OpenAsync(TvContext context, TestProvider provider)
{
await context.Database.OpenConnectionAsync();
if (provider is TestProvider.MySql)
{
await context.Connection.ExecuteAsync("SET SESSION foreign_key_checks = 0");
}
return new SeedSession(context);
}
public Task ExecuteAsync(string sql) => _context.Connection.ExecuteAsync(sql);
public async ValueTask DisposeAsync() => await _context.Database.CloseConnectionAsync();
}
}
@@ -1,3 +1,4 @@
using ErsatzTV.Core;
using ErsatzTV.Core.Domain;
using ErsatzTV.Infrastructure.Data;
using ErsatzTV.Infrastructure.Data.Repositories;
@@ -105,6 +106,225 @@ public class LibraryRepositoryTests
count.ShouldBe(1);
}
// ersatztv#491: the unique index is on (LibraryPathId, PathHash) because Path is unbounded, so every
// new row must carry the hash or the constraint is unenforceable for it.
[Test]
public async Task GetOrAddFolder_Should_Populate_PathHash_On_New_Folder()
{
int libraryPathId = await SeedLibraryPath("/data/music");
var libraryPath = new LibraryPath { Id = libraryPathId, Path = "/data/music", LibraryFolders = null };
LibraryFolder result = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/artist1");
result.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/artist1"));
await using TvContext context = _db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.Id == result.Id);
persisted.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/artist1"));
}
// ersatztv#491: rows that predate the PathHash column are left null by the migration (nulls are
// distinct in a unique index, so the index applies cleanly); the first scan that touches one heals it.
[Test]
public async Task GetOrAddFolder_Should_Heal_A_Legacy_Null_PathHash()
{
int libraryPathId = await SeedLibraryPath("/data/music");
var libraryPath = new LibraryPath { Id = libraryPathId, Path = "/data/music", LibraryFolders = null };
int legacyId;
await using (TvContext seed = _db.CreateContext())
{
var legacy = new LibraryFolder
{
LibraryPathId = libraryPathId, Path = "/data/music/artist1", PathHash = null
};
await seed.LibraryFolders.AddAsync(legacy);
await seed.SaveChangesAsync();
legacyId = legacy.Id;
}
LibraryFolder result = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/artist1");
result.Id.ShouldBe(legacyId);
result.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/artist1"));
await using TvContext context = _db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.Id == legacyId);
persisted.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/artist1"));
(await context.LibraryFolders.CountAsync(f => f.LibraryPathId == libraryPathId)).ShouldBe(1);
}
// ersatztv#491 / H1: when the lookup matches more than one row (legacy duplicates that predate the
// unique index, which is exactly the state the migration cleans up), it must resolve deterministically
// to the lowest Id. An unordered FirstOrDefault can return a different row as the plan changes — and
// adding the composite index alone can flip it — which would make the PathHash heal non-idempotent:
// each scan would heal a different row and the second would collide on (LibraryPathId, PathHash).
[Test]
public async Task GetOrAddFolder_Should_Resolve_Legacy_Duplicates_Deterministically()
{
int libraryPathId = await SeedLibraryPath("/data/music");
var libraryPath = new LibraryPath { Id = libraryPathId, Path = "/data/music", LibraryFolders = null };
int firstId;
await using (TvContext seed = _db.CreateContext())
{
var a = new LibraryFolder { LibraryPathId = libraryPathId, Path = "/data/music/artist1" };
var b = new LibraryFolder { LibraryPathId = libraryPathId, Path = "/data/music/artist1" };
await seed.LibraryFolders.AddRangeAsync(a, b);
await seed.SaveChangesAsync();
firstId = Math.Min(a.Id, b.Id);
}
// repeated calls must agree, and must agree with MIN(Id) — the same row the migration keeps
LibraryFolder first = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/artist1");
LibraryFolder second = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/artist1");
first.Id.ShouldBe(firstId);
second.Id.ShouldBe(firstId);
await using TvContext context = _db.CreateContext();
// exactly one of the two got the hash — the heal is idempotent, not alternating
(await context.LibraryFolders.CountAsync(
f => f.LibraryPathId == libraryPathId && f.PathHash != null)).ShouldBe(1);
LibraryFolder healed = await context.LibraryFolders.SingleAsync(f => f.Id == firstId);
healed.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/artist1"));
}
// ersatztv#491 / H1: on MySQL the lookup's SQL equality is case-INsensitive (longtext under the
// default collation), so it hands back BOTH "/x/Foo" and "/x/foo" for a "/x/foo" scan — verified on a
// real MySQL 8.4 server, where an unordered LIMIT 1 returns "/x/Foo". PathHash is a case-SENSITIVE
// hash, so identity has to be settled ordinally or the heal stamps the wrong path's hash onto a row.
//
// This pins that decision with no database at all, because no SQLite-backed test can: SQLite's `=` on
// TEXT is binary, so the case-differing candidate never reaches the in-memory step. Feeding the
// candidate list directly is the only way to exercise it in CI.
[Test]
public void ResolveExact_Should_Pick_The_Ordinal_Match_Not_A_Case_Variant()
{
var candidates = new List<LibraryFolder>
{
new() { Id = 10, Path = "/x/Foo" },
new() { Id = 11, Path = "/x/foo" }
};
LibraryRepository.ResolveExact(candidates, "/x/foo").Id.ShouldBe(11);
LibraryRepository.ResolveExact(candidates, "/x/Foo").Id.ShouldBe(10);
// a spelling that matches nothing ordinally is absent, not "close enough"
LibraryRepository.ResolveExact(candidates, "/x/FOO").ShouldBeNull();
LibraryRepository.ResolveExact([], "/x/foo").ShouldBeNull();
}
// MySQL's comparison is always case-insensitive and, on a PAD SPACE collation, also ignores trailing
// spaces ('/x/foo' = '/x/foo ' is TRUE under utf8mb4_general_ci and utf8mb4_bin; 8.4's default
// utf8mb4_0900_ai_ci is NO PAD, so this axis is server-dependent). Where it applies, the SQL narrowing
// hands back trailing-space siblings too. Those are distinct directories on Linux and hash
// differently, so the ordinal settle has to keep them apart — same defect class as the case variant,
// second axis. Ordinal compares length first, so this holds; pin it.
[Test]
public void ResolveExact_Should_Distinguish_Paths_Differing_Only_In_Trailing_Space()
{
var candidates = new List<LibraryFolder>
{
new() { Id = 20, Path = "/x/foo" },
new() { Id = 21, Path = "/x/foo " }
};
LibraryRepository.ResolveExact(candidates, "/x/foo").Id.ShouldBe(20);
LibraryRepository.ResolveExact(candidates, "/x/foo ").Id.ShouldBe(21);
LibraryRepository.ResolveExact(candidates, "/x/foo ").ShouldBeNull();
// the two spellings must also hash differently, or the unique index would reject one of them and
// the migration's decision to keep both would be wrong
PathUtils.GetPathHash("/x/foo").ShouldNotBe(PathUtils.GetPathHash("/x/foo "));
}
// Candidates arrive ordered by Id, and the first ordinal match wins — so true duplicates resolve to
// MIN(Id), the same row the #491 migration keeps.
[Test]
public void ResolveExact_Should_Prefer_The_First_Candidate_On_A_True_Duplicate()
{
var candidates = new List<LibraryFolder>
{
new() { Id = 3, Path = "/x/Foo" },
new() { Id = 7, Path = "/x/foo" },
new() { Id = 9, Path = "/x/foo" }
};
LibraryRepository.ResolveExact(candidates, "/x/foo").Id.ShouldBe(7);
}
// End-to-end companion to the ResolveExact tests above. On SQLite this passes on pre-fix code too (the
// SQL equality already excludes the case variant); its value is guarding the in-memory step against
// later being relaxed to OrdinalIgnoreCase, and covering the heal/no-duplicate behaviour around it.
[Test]
public async Task GetOrAddFolder_Should_Not_Resolve_A_Folder_Differing_Only_In_Case()
{
int libraryPathId = await SeedLibraryPath("/data/music");
var libraryPath = new LibraryPath { Id = libraryPathId, Path = "/data/music", LibraryFolders = null };
int upperId;
await using (TvContext seed = _db.CreateContext())
{
var upper = new LibraryFolder { LibraryPathId = libraryPathId, Path = "/data/music/Foo" };
var lower = new LibraryFolder { LibraryPathId = libraryPathId, Path = "/data/music/foo" };
await seed.LibraryFolders.AddRangeAsync(upper, lower);
await seed.SaveChangesAsync();
upperId = upper.Id;
}
LibraryFolder result = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/foo");
result.Path.ShouldBe("/data/music/foo");
result.Id.ShouldNotBe(upperId);
result.PathHash.ShouldBe(PathUtils.GetPathHash("/data/music/foo"));
await using TvContext context = _db.CreateContext();
// the case-differing sibling must be untouched — no foreign hash stamped onto it
LibraryFolder upperPersisted = await context.LibraryFolders.SingleAsync(f => f.Id == upperId);
upperPersisted.Path.ShouldBe("/data/music/Foo");
upperPersisted.PathHash.ShouldBeNull();
// and no duplicate was inserted for either spelling
(await context.LibraryFolders.CountAsync(f => f.LibraryPathId == libraryPathId)).ShouldBe(2);
}
// The heal is opportunistic maintenance on a hot scan path: if some other row already owns
// (LibraryPathId, hash) — a legacy duplicate the migration's grouping could not see — it must leave
// the row unhealed rather than abort the scan.
[Test]
public async Task GetOrAddFolder_Should_Not_Fail_The_Scan_When_The_PathHash_Heal_Collides()
{
int libraryPathId = await SeedLibraryPath("/data/music");
var libraryPath = new LibraryPath { Id = libraryPathId, Path = "/data/music", LibraryFolders = null };
string hash = PathUtils.GetPathHash("/data/music/artist1");
int legacyId;
await using (TvContext seed = _db.CreateContext())
{
// a legacy row with a null hash, plus a squatter that already owns the hash it would heal to
var legacy = new LibraryFolder { LibraryPathId = libraryPathId, Path = "/data/music/artist1" };
var squatter = new LibraryFolder
{
LibraryPathId = libraryPathId, Path = "/data/music/squatter", PathHash = hash
};
await seed.LibraryFolders.AddRangeAsync(legacy, squatter);
await seed.SaveChangesAsync();
legacyId = legacy.Id;
}
// must not throw — the scan continues and simply returns the folder
LibraryFolder result = await _repository.GetOrAddFolder(libraryPath, Option<int>.None, "/data/music/artist1");
result.Id.ShouldBe(legacyId);
await using TvContext context = _db.CreateContext();
LibraryFolder persisted = await context.LibraryFolders.SingleAsync(f => f.Id == legacyId);
persisted.PathHash.ShouldBeNull(); // heal declined, not half-applied
}
private async Task<int> SeedLibraryPath(string path)
{
await using TvContext context = _db.CreateContext();
+1
View File
@@ -207,6 +207,7 @@ another doc or an old issue comment should land here and then follow the link.
- 2026-07-23 — Facet-value typeahead is a new endpoint, allow-listed to text fields, no caching (#434) — [`api.search-field-values`](decisions/records/api/search-field-values.md)
- 2026-07-23 — Relative-date rule builder operators are a frontend-only mapping onto existing Lucene macros (#435) — [`rulebuilder.relative-date-macros`](decisions/records/rulebuilder/relative-date-macros.md)
- 2026-07-25 — A media-server sweep also refuses when the api client silently dropped items whose projection threw; the ratio threshold is rejected (#484) — [`scan.projection-failure-sweep-guard`](decisions/records/scan/projection-failure-sweep-guard.md)
- 2026-07-25 — LibraryFolder identity is enforced by a unique index on `(LibraryPathId, PathHash)`, not an in-process lock (#491) — [`scan.libraryfolder-unique-identity`](decisions/records/scan/libraryfolder-unique-identity.md)
- 2026-07-25 — Music videos carry a per-library server identity; reconciliation is an itemId diff + soft trash (#496) — [`scan.musicvideo-server-identity`](decisions/records/scan/musicvideo-server-identity.md)
- 2026-07-25 — OKF evaluated and rejected as a replacement; two of its optional fields adopted (#603) — [`docs.decision-optional-provenance`](decisions/records/docs/decision-optional-provenance.md)
- 2026-07-25 — Rule-builder group nesting is bounded-arbitrary depth (`MAX_GROUP_DEPTH`), not one level (#436) — [`spa.rulebuilder-nesting`](decisions/records/spa/rulebuilder-nesting.md)
+1
View File
@@ -115,6 +115,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
| `scan.collections-scan-status` | `GET /api/v1/media-sources/collections-scan-status` reports a family-global (not per-source), boolean-only active-scan set read from `IEntityLocker`; the SPA reconciles authoritatively against it (with a grace-tick helper) instead of a fixed client-side timeout. | 2026-07-12 | [link](records/scan/collections-scan-status.md) |
| `scan.getoraddfolder-db-lookup` | `ILibraryRepository.GetOrAddFolder` resolves the existing folder via a DB query on `(LibraryPathId, Path)`, not the caller's `LibraryPath.LibraryFolders` in-memory navigation, since that navigation is only eager-loaded on the local scan path and is null on remote (Jellyfin) callers. | 2026-07-20 | [link](records/scan/getoraddfolder-db-lookup.md) |
| `scan.jellyfin-mixed-content-library` | A Jellyfin library whose collection type is `mixed` (or absent) maps to one ErsatzTV library of `LibraryMediaKind.Mixed`, scanned by running the movie/television/music-video scanners in sequence against that single library — a library is a place, not a media kind. | 2026-07-20 | [link](records/scan/jellyfin-mixed-content-library.md) |
| `scan.libraryfolder-unique-identity` | `LibraryFolder` uniqueness per `(LibraryPathId, Path)` is enforced by a database unique index over a SHA-256 `PathHash` (Path is unbounded and not portably indexable), and `LibraryRepository.GetOrAddFolder`/`SetEtag` tolerate the constraint violation by re-reading and adopting the winner's row. | 2026-07-25 | [link](records/scan/libraryfolder-unique-identity.md) |
| `scan.musicvideo-server-identity` | Jellyfin music videos carry a per-library server identity (`JellyfinMusicVideo : MusicVideo` with `ItemId`/`Etag`, TPT table + ItemId index), so `JellyfinMusicVideoLibraryScanner` folds onto a shared `MediaServerMusicVideoLibraryScanner` base that diffs the **server item id** and soft-trashes (`FlagFileNotFound`) instead of diffing local paths and hard-deleting. Rows predating the identity are **adopted in place** — the identity row is inserted against the same `MediaItem` id, scoped to the scanned library's own `LibraryPath` — never deleted and re-added. | 2026-07-25 | [link](records/scan/musicvideo-server-identity.md) |
| `scan.projection-failure-sweep-guard` | `MediaServerReconciliationGuard` takes a per-enumeration projection-failure count and refuses the file-not-found sweep (logged loudly) whenever it is non-zero against a non-empty existing set — at all six sweeps, including the nested per-show season and per-season episode ones #477 left unguarded (via `ShouldFlagMissingDescendants`, which applies the failure refusal but not #477's empty-fetch branch). Deliberate guard-clause skips (STRM, virtual, unsupported type) are explicitly **not** failures and never suppress a sweep. The missing-fraction / ratio threshold floated by #477 is **rejected**, not deferred. | 2026-07-25 | [link](records/scan/projection-failure-sweep-guard.md) |
| `scan.zero-item-fetch-guard` | A media-server library sweep refuses to flag missing items when a successful fetch returns zero incoming items against a non-empty existing set (`MediaServerReconciliationGuard.ShouldFlagMissing`), rather than treating an ambiguous empty result as a full-library deletion. | 2026-07-19 | [link](records/scan/zero-item-fetch-guard.md) |
@@ -0,0 +1,242 @@
---
key: scan.libraryfolder-unique-identity
title: '2026-07-25 — LibraryFolder identity is enforced by a unique index on `(LibraryPathId, PathHash)`, not an in-process lock (#491)'
status: active
since: '2026-07-25'
supersedes: none
superseded-by: none
rule: '`LibraryFolder` uniqueness per `(LibraryPathId, Path)` is enforced by a database unique index over a SHA-256 `PathHash` (Path is unbounded and not portably indexable), and `LibraryRepository.GetOrAddFolder`/`SetEtag` tolerate the constraint violation by re-reading and adopting the winner''s row.'
signals: 'GetOrAddFolder, SetEtag, LibraryFolder duplicate rows, check-then-insert race, unique index, PathHash, longtext index, dedupe migration, concurrent scan · paths: `LibraryRepository.GetOrAddFolder`, `LibraryFolderConfiguration`, `LibraryFolder.PathHash`, `Add_LibraryFolder_PathHash_UniqueIndex`, `LibraryFolderConcurrencyTests`, `LibraryFolderDedupeMigrationTests` · issues: #491, #488, #308'
mechanics: '`LibraryFolderConfiguration` (`HasIndex(f => new { f.LibraryPathId, f.PathHash }).IsUnique()`); dual-provider migration `Add_LibraryFolder_PathHash_UniqueIndex`'
---
**Storage constraint, not a lock.** `GetOrAddFolder` looks a folder up by `(LibraryPathId, Path)` and
inserts when absent (`scan.getoraddfolder-db-lookup`, #488). Lookup and insert are not atomic, so two
callers racing the same path both miss and both insert. The alternative on the table — an in-process
`SemaphoreSlim` keyed on the path — was rejected: it closes only the single-process window, leaves the
invariant undocumented in the schema, silently permits duplicates from any other writer, and cannot
clean the duplicates that already exist. The index makes the duplicate *impossible*; the repository's
catch turns the loser into an idempotent adopt.
**Reuse the #308 provider seam — and wire it in BOTH composition roots.** Detection goes through the
existing `TvContext.IsUniqueConstraintViolation` delegate (`concurrency.idempotent-concurrent-add`),
pointed at `SqliteErrorClassifier` / `MySqlErrorClassifier` and defaulting to a conservative "no", so an
unwired provider never silently swallows a save failure. No second unique-violation detector.
There are **two** composition roots, and #308 only needed one. `ErsatzTV/Startup.cs` is the host;
`ErsatzTV.Scanner/Program.cs` is a *separate executable* launched per scan by
`CallLibraryScannerHandler`, and it wires its own copy of the provider statics. #308's only consumer
(`ConcurrencyExtensions.TrySaveChangesForcingVersion`) runs solely in the host, so `Startup` sufficed
there. This record's consumer is the opposite: **every** production caller of `GetOrAddFolder`/`SetEtag`
lives in the scanner (the eight `*FolderScanner`s plus `JellyfinMusicVideoLibraryScanner`) and **none**
in the host. Wiring only `Startup` would leave the classifier at its `_ => false` default in the only
process that runs the code, so the catch would be inert and the `DbUpdateException` would escape and
fail the whole scan — strictly worse than the duplicate row it replaces. Both roots now wire it.
The failure mode is "a static nobody assigned", which no unit test can see, because every test harness
wires the classifier itself — the #491 negative control (`IsUniqueConstraintViolation = _ => false`)
was in fact an accidental reproduction of the real scanner configuration. `ProviderStaticsWiringTests`
(architecture tests) therefore asserts at the **source** level that the scanner assigns every
`TvContext.*` static the host assigns, with a single documented exemption (`IsSqlite`, read only by
`DbInitializer`/`DatabaseMigratorService`, both host-only). Removing either scanner assignment fails it
with the offending name.
**Index the hash, not the path — a portability trap.** The naive `HasIndex(f => new { f.LibraryPathId,
f.Path }).IsUnique()` does not work on both providers: `LibraryFolder.Path` maps to MySQL `longtext`,
which cannot be indexed without a prefix length (error 1170), and bounding it with `HasMaxLength`
would truncate existing long paths and cap folder depth. A prefix index would also inherit MySQL's
default *case-insensitive* collation and false-collide sibling folders differing only in case, which
are legal on a case-sensitive filesystem. So identity is carried by a `varchar(64)` SHA-256 hex
`PathHash` (`PathUtils.GetPathHash`) — byte-exact on both providers, fixed width, no length cap. This
is not a new pattern: `MediaFile` already carries exactly this `Path`/`PathHash` pair for the same
reason. `Path` keeps its unbounded column and stays the *query* key; the hash exists only to be
indexable.
**Legacy rows keep a null hash and heal lazily.** Backfilling the hash cannot be done inside the
migration — SQLite has no `sha2()` — and the only in-repo alternative is the `DatabaseMigratorService`
stop-migrate-run-SQL-continue hack that `Add_MediaFilePathHash` needed. It is not needed here: a unique
index treats nulls as distinct on both providers, so the index applies cleanly to any existing
database, and pre-#491 rows are still found by the `Path` lookup, so no insert can race them in the
meantime. `GetOrAddFolder` writes the hash the first time a scan touches such a row, so a database
converges after one scan without a startup migration phase.
**The lookup must be as case-sensitive as the hash.** `PathUtils.GetPathHash` hashes the exact bytes, but
`GetFolder`'s `Path == folder` is evaluated by the *server*, and on MySQL a `longtext` under the default
collation is case-**insensitive** — the same collation the codebase names in
`TvContext.CaseInsensitiveCollation`. Verified on MySQL 8.4: with rows `/x/Foo` (id 10) and `/x/foo`
(id 11), `WHERE Path = '/x/foo'` returns **both**, and an unordered `LIMIT 1` — what
`FirstOrDefaultAsync` emits — returns **id 10**. Constraint semantics and lookup semantics would
therefore disagree: a scan of `/x/foo` would resolve the `/x/Foo` row and the heal would stamp the wrong
path's hash onto it, corrupting the identity column with nothing able to detect or repair it. `GetFolder`
now treats the SQL equality as a *narrowing* filter only, orders by `Id` for determinism (an unordered
pick can flip as the plan changes — adding the composite index alone can do it, which would make the
heal non-idempotent and self-collide on the next scan), and settles identity in memory with an **ordinal**
comparison. The same function serves the post-violation re-read in `GetOrAddFolder` and `SetEtag`, so
neither can adopt a case-differing sibling as "the winner".
This is a **deliberate behaviour change on MySQL**, worth stating: `GetFolder` now returns null for a
case-differing spelling, so the scanner *inserts* a second `LibraryFolder` row where it previously reused
the case-differing one. That is the correct semantics for a case-sensitive filesystem, and it makes MySQL
match SQLite's long-standing behaviour (SQLite's `=` on TEXT has always been binary) rather than
diverging by provider. The visible consequence is on a **case-insensitive** filesystem backed by MySQL:
re-casing a library path now yields a second root `LibraryFolder` row instead of reuse. The pure
in-memory settle is factored out as `LibraryRepository.ResolveExact` so this decision is pinned by a
unit test rather than resting only on hand-run MySQL evidence.
**The heal must never be able to abort a scan.** It is opportunistic maintenance on a hot path, so it
goes through EF rather than the raw Dapper `UPDATE` it started as — a raw update throws a bare
`MySqlException` 1062 that no `DbUpdateException` filter can catch — and its unique-violation is caught
and dropped, leaving the row for the next scan. The filter also admits `DbUpdateConcurrencyException`:
it derives from `DbUpdateException` but carries no provider exception, so the classifier does **not**
recognize it, and a row deleted by a concurrent library edit between the heal's read and its save would
otherwise propagate and fail the scan — the exact invariant the comment above it claims. With the case-exact lookup above the collision is only
reachable via a legacy duplicate the migration's grouping cannot see (see the `NULL` caveat below), but
"declines to heal" is the only acceptable failure mode for a maintenance write.
**Ship the cleanup with the constraint.** Duplicates are not merely theoretical — before #488 the
lookup read a scan-start in-memory snapshot, so a folder created earlier in the *same* scan was
invisible and inserted again. The migration therefore audits and collapses them before creating the
index: keep the lowest `Id` per `(LibraryPathId, Path)`, repoint `MediaFile.LibraryFolderId` and
`LibraryFolder.ParentId` (both `Restrict`, so the delete would otherwise fail) at the keeper, and —
because `ImageFolderDuration` is 1:1 with a unique index of its own and cannot hold two rows for one
folder — keep the keeper's setting, or promote exactly one duplicate's when it has none, dropping the
rest. Two helper tables keep the statements readable; they are dropped at the end and created with
`DROP TABLE IF EXISTS` first so a retry after a partial failure is safe (DDL implicitly commits on
MySQL, so the migration is not atomic there). The MySQL copy collates every `Path` comparison
`CONVERT(Path USING binary)` — **byte-exact, not merely case-exact**. This is the second axis, and it
cost a round: the first attempt used `COLLATE utf8mb4_bin`, which does fix case, but `utf8mb4_bin` is a
**PAD SPACE** collation (only the `utf8mb4_0900_*` family is NO PAD). Verified on 8.4:
`'/media/Foo' = '/media/Foo '` is TRUE under it, so `/media/Foo` and `/media/Foo ` — two distinct legal
directories — grouped together and the second was **deleted irreversibly**, even though
`PathUtils.GetPathHash` hashes them differently and the unique index would have accepted both. A dedupe
that destroys data the constraint never required it to destroy is a strictly worse bug than the one it
fixes. Binary comparison is NO PAD and byte-exact, matching the hash exactly; `utf8mb4_0900_bin` would
also work but carries a server-version floor. The grouping, the join predicate and the keeper selection
all use it, and it is the *only* path comparison in either migration — every other predicate keys off an
integer id — so there is no mix of padded and unpadded comparisons across the steps.
The general lesson: **"case-exact" is not "byte-exact"**, and a same-provider fix for one collation axis
can silently leave another open. SQLite's `=` on TEXT is byte-exact with no padding behaviour, so the
SQLite migration was always correct and the divergence was invisible to a SQLite-only test — which is
structurally why a SQLite-only `LibraryFolderDedupeMigrationTests` could not catch it — now closed, see
"Both providers run the same fixture" below. The **runtime** path was never affected: `GetFolder`'s SQL
equality is only a superset narrowing (every collation quirk makes it more permissive, never less, so it
cannot miss a byte-exact match) and `ResolveExact` settles identity with `StringComparison.Ordinal`,
which compares length first and therefore separates trailing-space variants correctly.
Be precise about which axis is guaranteed, because the two are not alike. The schema pins only the
`utf8mb4` **charset** on `LibraryFolder.Path`, never a collation, so the effective comparison is the
server default. Case-insensitivity holds under every plausible default (8.4's `utf8mb4_0900_ai_ci` and
the older `utf8mb4_general_ci` are both `_ci`) — that axis is always live. PAD SPACE is
**server-dependent**: `utf8mb4_general_ci` has it, `utf8mb4_0900_ai_ci` does *not* (it is NO PAD).
Verified on 8.4, where the column really does come out `utf8mb4_0900_ai_ci` and `WHERE Path =
'/media/Foo'` matches the case variant but not the trailing-space one. The *migration* bug was
independent of that default because the old code applied an **explicit** `COLLATE utf8mb4_bin`, which is
PAD SPACE on every server; the runtime must simply tolerate both, which the superset + ordinal shape
does. The
keeper's `Etag` is cleared (which duplicate the scanner was actually writing to was arbitrary, so
`MIN(Id)`'s etag may describe a stale view and would suppress the repairing rescan — one rescan is
strictly safe), and a final `ParentId = NULL WHERE ParentId = Id` removes the self-cycle a folder
parented on its own duplicate would otherwise acquire.
**Limits of the cleanup, stated rather than implied.** (a) It is **not** an exhaustive sweep: rows with a
`NULL` `Path` are excluded by `NULL = NULL` and are left as-is — harmless, but it is why the heal's
collision catch is a real backstop rather than dead code. (b) It **deletes user data**, and `Down` only
restores the index — the deleted duplicate rows and the dropped `ImageFolderDuration` settings are
unrecoverable, so rollback is backup-restore, not `Down`. The `DROP TABLE IF EXISTS` guard makes the
migration safe to **re-run**, which is not the same as safe to **undo**. (c) Because the index is on a
column that is `NULL` for every pre-existing row, and `NULL`s are distinct, the deletion is not actually
*required* for the migration to apply — it is data hygiene, which is the safer posture for a destructive
statement. (d) CI's `migrations` job applies to a **fresh, empty** database, so it executes zero rows of
this logic; it proves the DDL orders correctly and nothing about the data mutation.
**Create the composite index before dropping the one it replaces.** Adding an explicit index whose
leftmost column is `LibraryPathId` makes EF drop the FK-convention index `IX_LibraryFolder_LibraryPathId`,
and EF scaffolds that drop *first*. On MySQL that fails outright — `Cannot drop index
'IX_LibraryFolder_LibraryPathId': needed in a foreign key constraint` — because InnoDB will not leave a
foreign key without a backing index. Both migrations are hand-reordered to create the composite (which
then backs the FK by leftmost prefix) before dropping the single-column index, and `Down` mirrors it.
This is invisible on SQLite and was only caught by replaying the migration against a real MySQL 8.4
server; the CI `migrations` job's fresh-DB apply would have caught it too, but only after the push.
**Tests.** `LibraryFolderConcurrencyTests` reproduces a real cross-connection race through the actual
repository via the shared-cache SQLite harness + a `SavingChanges` interceptor that inserts the
conflicting row on another connection mid-save, and asserts the interceptor fired (the vacuity guard —
a concurrency test that never raced proves nothing). A separate 8-thread × 10-round barrier test
asserts one row per path *and* that the number of insert attempts exceeded the number of rounds, so a
run where the threads never collided fails rather than passing hollow. Negative controls: removing the
index fails 6 of the 7 tests, and inverting the classifier (`IsUniqueConstraintViolation = _ => false`)
makes the same race throw, proving the catch is load-bearing.
`LibraryFolderDedupeMigrationTests` drives the *real* Sqlite migration against a database seeded at the
previous migration, so the cleanup SQL is exercised rather than restated — and it runs the migration
with **foreign keys ON** (matching production; seeding, which writes partial graphs, keeps them off), so
`DELETE FROM LibraryFolder` is executed against the two `Restrict` foreign keys it must not violate
rather than with enforcement disabled.
**Both providers run the same fixture, in CI.** `LibraryFolderDedupeMigrationTests` is parameterized
`[TestFixture(TestProvider.Sqlite)]` / `[TestFixture(TestProvider.MySql)]` over **one** fixture body: the
same seeded rows, the same expected survivor set (`1,4,5,6,7,9,10`), including both collation traps — a
case-differing sibling and a trailing-space sibling, each with its own dependent media file. A separate
MySQL-only copy was rejected: it would drift from the SQLite one and recreate the gap, whereas one
parameterized body makes "the two providers agree" a checked property. Assertions deliberately avoid
`WHERE Path = '…'`, which is itself collation-dependent and would silently mean something different per
provider; rows are read once and compared ordinally in memory.
The MySQL half of this migration has **no automated coverage**, and that is a known, tracked gap
(#627) rather than an oversight. CI's `migrations` job only ever applies migrations to a fresh
**empty** database, so it executes zero rows of dedupe logic — both MySQL-only collation defects
sailed through it and were caught only by hand-run containers that vanish with the session. A CI step
running the fixture against the job's existing `mysql:8.4` service was built and then **removed**: it
was non-deterministic across three attempts (a stale pooled session surviving `DROP DATABASE`, then
lost isolation from a shared database name, then a connect-before-create), and an intermittently-red
gate is worse than none — it trains everyone to re-run instead of read, which is how the original
defects escaped in the first place.
What survives is the fixture itself: `LibraryFolderDedupeMigrationTests` is parameterized over both
providers from **one shared body**, so the expectations cannot silently diverge, and the MySQL half is
opt-in via `ETV_TEST_MYSQL_CONNECTION` (a visible skip without it) with `ETV_REQUIRE_MYSQL_TESTS=1`
turning that skip into a hard failure for whoever re-arms it. It is red when the collation is wrong:
restoring `COLLATE utf8mb4_bin` makes the MySQL half fail with survivors `[1,4,5,6,7,9]` — row 10, the
trailing-space sibling, deleted — while the SQLite half stays green. Run it by hand against a real
server before touching this migration.
**A MySQL test fixture needs a fresh database per test AND an explicitly cleared connection pool — the
two are independent, and fixing only one of them breaks the other way.** This fixture failed in CI twice,
each time for a different half of that sentence, so both are recorded with their measurements (taken
against a real 8.4 server, not reasoned about):
- **A pooled session outlives `DROP DATABASE`.** Reopening the dropped database's connection string
*succeeds*, because MySqlConnector hands back the still-alive pooled session whose default schema no
longer exists. Whether a caller then sees success or `Unknown database` depends on whether the pool
reuses that session or has to open a fresh one — a fresh handshake names the dropped schema and fails
1049. That was failure #1: intermittent `Unknown database 'etv491_…'` at *connect* time, green on the
immediately preceding commit. It is invisible in any single green run.
- **An uncleared pool leaks a server connection per test.** MySqlConnector keys pools by connection
string, so a per-test database name means a per-test pool; left uncleared it leaked ~1 server thread
per iteration and hammering it exhausted `max_connections` (151) outright. A saturated server also
inflated this fixture's runtime from ~1m20s to 13 minutes — "slow" and "flaky" were the same defect.
The first fix collapsed to a single shared database to stop the leak. That was a **misdiagnosis worth
recording**: the leak came from never clearing pools, not from the names being unique, so the shared name
gave up per-test isolation to solve a problem pool-clearing already solved. It duly failed a different
way — failure #2, `Duplicate entry '1' for key 'LibraryPath.PRIMARY'` in 539 ms, the second test seeding
on top of the first's rows because the wipe silently did not happen and `MigrateAsync` no-opped against
an already-current `__EFMigrationsHistory`. A shared database makes correctness depend on a wipe
succeeding; a name that has never been used **cannot** contain another test's rows.
The shape that satisfies both: a **fresh `etv491_<guid>` database per test**, never created out of band
(the test's own `MigrateAsync(PreviousMigration)` creates it, keeping EF the single owner of the schema),
dropped in `TearDown` via `EnsureDeletedAsync` (guarded — a no-op when absent, unlike a raw `DROP
DATABASE`) followed by `ClearPoolAsync` **on that exact connection string**. Measured: 0 leaked threads
over 30 iterations, so isolation costs nothing. The stale-session hazard needs the connection string to
be reused after the drop, which a never-repeated name already makes impossible; clearing the pool is the
belt to that brace. No `[Retry]` anywhere — a gate that is re-run on failure trains people to re-run it,
which is worse than no gate.
Verified: 10 consecutive runs 10/10 green with server threads flat at 2 and zero leftover schemas; each
test run **alone** (twice each) and with the order **reversed** — order-independence is the specific
evidence failure #2 would have needed, since a contamination bug is invisible when a test runs first.
Still uncovered by automation, and stated rather than assumed: the lookup-collation premises behind the
ordinal fix were verified by hand on 8.4 (`WHERE Path = '/x/foo'` matching both `/x/Foo` and `/x/foo`
with `LIMIT 1` returning the wrong one; `'/media/Foo' = '/media/Foo '` TRUE under `utf8mb4_bin`).
`ResolveExact` unit tests pin the in-memory decision for both axes with no provider at all. Refs #491 #488 #308.