feat(484): suppress the media-server sweep on projection failures; reject the ratio threshold
Extends MediaServerReconciliationGuard (#477) with a second deterministic refusal: when the enumeration that produced the incoming set silently dropped items whose projection THREW, the file-not-found sweep is refused. A dropped item the server did return is indistinguishable from a deletion at the reconcile step, so a projection regression could otherwise mass-flag a healthy library FileNotFound (which EmptyTrash then deletes permanently). Deliberate guard-clause skips (STRM files, virtual items, unsupported types) are explicitly NOT failures and never suppress a sweep — counting them would permanently disable reconciliation for any library holding a single STRM file. The ratio / missing-fraction threshold is REJECTED, not deferred: it is a two-sided heuristic with no tunable default and no telemetry, and the failure it approximates is exactly observable via the projection-failure count (a genuine bulk deletion produces zero failures). Seam is deliberately narrow — the private ProjectTo* contract inside each api client changed from Option<T> to MediaServerProjectionResult<T> (projected/skipped/failed), the paged helper counts IsFailure in one place, and the scanner reads it through an optional trailing MediaServerProjectionFailureCounter on only the five library-level methods that feed a sweep. The counter is per-enumeration state created by the scanner, never a field on an api client. fixes #484
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Emby;
|
||||
using ErsatzTV.Core.Metadata;
|
||||
|
||||
namespace ErsatzTV.Core.Interfaces.Emby;
|
||||
|
||||
@@ -8,9 +9,20 @@ public interface IEmbyApiClient
|
||||
Task<Either<BaseError, EmbyServerInformation>> GetServerInformation(string address, string apiKey);
|
||||
Task<Either<BaseError, List<EmbyLibrary>>> GetLibraries(string address, string apiKey);
|
||||
|
||||
IAsyncEnumerable<Tuple<EmbyMovie, int>> GetMovieLibraryItems(string address, string apiKey, EmbyLibrary library);
|
||||
// #484: the two library-level enumerations that feed a reconciliation sweep accept an optional
|
||||
// per-enumeration counter. The caller creates one instance per enumeration and reads it only after
|
||||
// the enumeration completes; passing null (the default) opts out and costs existing callers nothing.
|
||||
IAsyncEnumerable<Tuple<EmbyMovie, int>> GetMovieLibraryItems(
|
||||
string address,
|
||||
string apiKey,
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null);
|
||||
|
||||
IAsyncEnumerable<Tuple<EmbyShow, int>> GetShowLibraryItems(string address, string apiKey, EmbyLibrary library);
|
||||
IAsyncEnumerable<Tuple<EmbyShow, int>> GetShowLibraryItems(
|
||||
string address,
|
||||
string apiKey,
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null);
|
||||
|
||||
IAsyncEnumerable<Tuple<EmbySeason, int>> GetSeasonLibraryItems(
|
||||
string address,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Jellyfin;
|
||||
using ErsatzTV.Core.Metadata;
|
||||
|
||||
namespace ErsatzTV.Core.Interfaces.Jellyfin;
|
||||
|
||||
@@ -8,20 +9,26 @@ public interface IJellyfinApiClient
|
||||
Task<Either<BaseError, JellyfinServerInformation>> GetServerInformation(string address, string authorizationHeader);
|
||||
Task<Either<BaseError, List<JellyfinLibrary>>> GetLibraries(string address, string authorizationHeader);
|
||||
|
||||
// #484: the three library-level enumerations that feed a reconciliation sweep accept an optional
|
||||
// per-enumeration counter. The caller creates one instance per enumeration and reads it only after
|
||||
// the enumeration completes; passing null (the default) opts out and costs existing callers nothing.
|
||||
IAsyncEnumerable<Tuple<JellyfinMovie, int>> GetMovieLibraryItems(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library);
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null);
|
||||
|
||||
IAsyncEnumerable<Tuple<MusicVideo, int>> GetMusicVideoLibraryItems(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library);
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null);
|
||||
|
||||
IAsyncEnumerable<Tuple<JellyfinShow, int>> GetShowLibraryItemsWithoutPeople(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library);
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null);
|
||||
|
||||
IAsyncEnumerable<Tuple<JellyfinSeason, int>> GetSeasonLibraryItems(
|
||||
string address,
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
namespace ErsatzTV.Core.Metadata;
|
||||
|
||||
// #484: the narrow seam that carries "how many items did the server return that we silently dropped
|
||||
// because their projection threw?" out of a media-server API client and back to the scanner that owns
|
||||
// the reconciliation sweep.
|
||||
//
|
||||
// Lifetime is deliberately PER ENUMERATION: the scanner that is about to run a sweep creates one
|
||||
// instance, hands it to the single library-items call whose result it will diff, and reads Count only
|
||||
// after that enumeration has completed. The API clients are long-lived singletons and scans for
|
||||
// different libraries run concurrently, so the counter must never be a field on a client or any
|
||||
// ambient/static state — that would leak one library's failures into another library's sweep decision.
|
||||
// Increments are interlocked anyway so a paginator that ever fans out stays correct.
|
||||
public sealed class MediaServerProjectionFailureCounter
|
||||
{
|
||||
private int _count;
|
||||
|
||||
/// <summary>
|
||||
/// Number of items the server returned whose projection threw and was swallowed.
|
||||
/// </summary>
|
||||
public int Count => Volatile.Read(ref _count);
|
||||
|
||||
public void RecordFailure() => Interlocked.Increment(ref _count);
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
namespace ErsatzTV.Core.Metadata;
|
||||
|
||||
// #484: a media-server API client maps each item the server returned through a private projection. That
|
||||
// projection has TWO reasons to produce nothing, and conflating them is dangerous:
|
||||
//
|
||||
// - Skipped — a deliberate guard clause (a virtual/non-FileSystem item, a STRM file, an unsupported
|
||||
// item type). The item is permanently and expectedly absent from the incoming set; a
|
||||
// library containing one STRM file produces a Skipped on every single scan, forever.
|
||||
// - Failed — the projection threw and was swallowed by a `catch { LogWarning; }`. The server DID
|
||||
// return the item; we simply could not build it. At the reconcile step that is
|
||||
// indistinguishable from a deletion, so a projection regression can mass-flag healthy
|
||||
// items FileNotFound (which EmptyTrash then deletes permanently).
|
||||
//
|
||||
// Only Failed may suppress the reconciliation sweep (see MediaServerReconciliationGuard). Treating
|
||||
// Skipped as a failure would permanently disable reconciliation for any library holding a single STRM
|
||||
// file, so stale rows would accumulate forever — a regression, not a safe default.
|
||||
public readonly struct MediaServerProjectionResult<T>
|
||||
{
|
||||
private MediaServerProjectionResult(Option<T> item, bool isFailure)
|
||||
{
|
||||
Item = item;
|
||||
IsFailure = isFailure;
|
||||
}
|
||||
|
||||
public Option<T> Item { get; }
|
||||
|
||||
/// <summary>
|
||||
/// True only when the projection threw. A deliberate guard-clause skip is NOT a failure.
|
||||
/// </summary>
|
||||
public bool IsFailure { get; }
|
||||
|
||||
public static MediaServerProjectionResult<T> Projected(T item) => new(item, false);
|
||||
|
||||
/// <summary>
|
||||
/// The server returned an item we deliberately and permanently do not import (virtual item, STRM
|
||||
/// file, unsupported type). Expected on every scan; never suppresses a sweep.
|
||||
/// </summary>
|
||||
public static MediaServerProjectionResult<T> Skipped() => new(Option<T>.None, false);
|
||||
|
||||
/// <summary>
|
||||
/// The projection threw and the exception was swallowed. The item exists upstream but is missing
|
||||
/// from the incoming set, so the sweep must not run.
|
||||
/// </summary>
|
||||
public static MediaServerProjectionResult<T> Failed() => new(Option<T>.None, true);
|
||||
|
||||
public static implicit operator MediaServerProjectionResult<T>(T item) => Projected(item);
|
||||
|
||||
public Option<T> ToOption() => Item;
|
||||
}
|
||||
@@ -3,6 +3,8 @@ using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Interfaces.Jellyfin;
|
||||
using ErsatzTV.Core.Interfaces.Metadata;
|
||||
using ErsatzTV.Core.Jellyfin;
|
||||
using ErsatzTV.Core.Metadata;
|
||||
using ErsatzTV.Infrastructure.Jellyfin;
|
||||
using LanguageExt;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
@@ -271,6 +273,219 @@ public class JellyfinApiClientTests
|
||||
}
|
||||
}
|
||||
|
||||
// #484: the projection-failure seam. The api client drops an item for two very different reasons and
|
||||
// only ONE of them may suppress the downstream reconciliation sweep, so these tests pin the split at
|
||||
// the layer that actually makes the distinction.
|
||||
[TestFixture]
|
||||
public class ProjectionFailureCounter
|
||||
{
|
||||
// A STRM file is a deliberate, permanent guard-clause skip. It must NOT be reported as a failure:
|
||||
// if it were, one STRM file would disable the library's reconciliation sweep on every scan
|
||||
// forever and stale rows would accumulate. This is the single most important assertion here.
|
||||
[Test]
|
||||
public async Task Deliberate_Skip_Is_Not_Counted_As_A_Failure()
|
||||
{
|
||||
const string response = """
|
||||
{
|
||||
"Items": [
|
||||
{
|
||||
"Name": "Streamed",
|
||||
"Id": "item-strm",
|
||||
"Path": "/data/music/streamed.strm",
|
||||
"Type": "MusicVideo",
|
||||
"LocationType": "FileSystem",
|
||||
"RunTimeTicks": 3120000000,
|
||||
"DateCreated": "2026-07-01T00:00:00Z",
|
||||
"ProviderIds": {}
|
||||
},
|
||||
{
|
||||
"Name": "Virtual",
|
||||
"Id": "item-virtual",
|
||||
"Path": "/data/music/virtual.mkv",
|
||||
"Type": "MusicVideo",
|
||||
"LocationType": "Virtual",
|
||||
"RunTimeTicks": 3120000000,
|
||||
"DateCreated": "2026-07-01T00:00:00Z",
|
||||
"ProviderIds": {}
|
||||
},
|
||||
{
|
||||
"Name": "Hells Bells",
|
||||
"Id": "item-1",
|
||||
"Path": "/data/music/hells-bells.mkv",
|
||||
"Type": "MusicVideo",
|
||||
"LocationType": "FileSystem",
|
||||
"RunTimeTicks": 3120000000,
|
||||
"DateCreated": "2026-07-01T00:00:00Z",
|
||||
"ProviderIds": {}
|
||||
}
|
||||
],
|
||||
"TotalRecordCount": 3
|
||||
}
|
||||
""";
|
||||
|
||||
var client = new JellyfinApiClient(
|
||||
new MemoryCache(new MemoryCacheOptions()),
|
||||
Substitute.For<IJellyfinPathReplacementService>(),
|
||||
Substitute.For<IFallbackMetadataProvider>(),
|
||||
new SingleResponseHttpClientFactory(response),
|
||||
Substitute.For<ILogger<JellyfinApiClient>>());
|
||||
|
||||
// no PathInfos => the network-path replacement loop is skipped and nothing can throw
|
||||
var library = new JellyfinLibrary { Id = 42, MediaSourceId = 1, ItemId = "library-1" };
|
||||
|
||||
var counter = new MediaServerProjectionFailureCounter();
|
||||
|
||||
List<MusicVideo> musicVideos = [];
|
||||
await foreach ((MusicVideo musicVideo, int _) in client.GetMusicVideoLibraryItems(
|
||||
"http://jellyfin.example",
|
||||
"MediaBrowser Token=abc",
|
||||
library,
|
||||
counter))
|
||||
{
|
||||
musicVideos.Add(musicVideo);
|
||||
}
|
||||
|
||||
musicVideos.Count.ShouldBe(1);
|
||||
counter.Count.ShouldBe(0);
|
||||
}
|
||||
|
||||
// A projection that THREW was swallowed by the catch and the item silently vanished — but the
|
||||
// server did return it. That must be reported so the sweep refuses to treat it as a deletion.
|
||||
[Test]
|
||||
public async Task Swallowed_Projection_Exception_Is_Counted_As_A_Failure()
|
||||
{
|
||||
const string response = """
|
||||
{
|
||||
"Items": [
|
||||
{
|
||||
"Name": "Hells Bells",
|
||||
"Id": "item-1",
|
||||
"Path": "/network/music/hells-bells.mkv",
|
||||
"Type": "MusicVideo",
|
||||
"LocationType": "FileSystem",
|
||||
"RunTimeTicks": 3120000000,
|
||||
"DateCreated": "2026-07-01T00:00:00Z",
|
||||
"ProviderIds": {}
|
||||
}
|
||||
],
|
||||
"TotalRecordCount": 1
|
||||
}
|
||||
""";
|
||||
|
||||
var pathReplacementService = Substitute.For<IJellyfinPathReplacementService>();
|
||||
pathReplacementService.ReplaceNetworkPath(
|
||||
Arg.Any<JellyfinMediaSource>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>())
|
||||
.Returns(_ => throw new InvalidOperationException("simulated projection regression"));
|
||||
|
||||
var client = new JellyfinApiClient(
|
||||
new MemoryCache(new MemoryCacheOptions()),
|
||||
pathReplacementService,
|
||||
Substitute.For<IFallbackMetadataProvider>(),
|
||||
new SingleResponseHttpClientFactory(response),
|
||||
Substitute.For<ILogger<JellyfinApiClient>>());
|
||||
|
||||
var library = new JellyfinLibrary
|
||||
{
|
||||
Id = 42,
|
||||
MediaSourceId = 1,
|
||||
ItemId = "library-1",
|
||||
PathInfos = [new JellyfinPathInfo { NetworkPath = "/network", Path = "/local" }]
|
||||
};
|
||||
|
||||
var counter = new MediaServerProjectionFailureCounter();
|
||||
|
||||
List<MusicVideo> musicVideos = [];
|
||||
await foreach ((MusicVideo musicVideo, int _) in client.GetMusicVideoLibraryItems(
|
||||
"http://jellyfin.example",
|
||||
"MediaBrowser Token=abc",
|
||||
library,
|
||||
counter))
|
||||
{
|
||||
musicVideos.Add(musicVideo);
|
||||
}
|
||||
|
||||
// the item is gone from the incoming set (exactly the dangerous case) and it is reported
|
||||
musicVideos.ShouldBeEmpty();
|
||||
counter.Count.ShouldBe(1);
|
||||
}
|
||||
|
||||
// The counter is created per enumeration by the caller, never held by the (singleton) client, so
|
||||
// two concurrent library scans cannot leak failures into each other's sweep decision.
|
||||
[Test]
|
||||
public async Task Counters_Are_Per_Enumeration_And_Do_Not_Leak_Across_Scans()
|
||||
{
|
||||
const string failingResponse = """
|
||||
{
|
||||
"Items": [
|
||||
{
|
||||
"Name": "Hells Bells",
|
||||
"Id": "item-1",
|
||||
"Path": "/network/music/hells-bells.mkv",
|
||||
"Type": "MusicVideo",
|
||||
"LocationType": "FileSystem",
|
||||
"RunTimeTicks": 3120000000,
|
||||
"DateCreated": "2026-07-01T00:00:00Z",
|
||||
"ProviderIds": {}
|
||||
}
|
||||
],
|
||||
"TotalRecordCount": 1
|
||||
}
|
||||
""";
|
||||
|
||||
var pathReplacementService = Substitute.For<IJellyfinPathReplacementService>();
|
||||
pathReplacementService.ReplaceNetworkPath(
|
||||
Arg.Any<JellyfinMediaSource>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>())
|
||||
.Returns(_ => throw new InvalidOperationException("simulated projection regression"));
|
||||
|
||||
var client = new JellyfinApiClient(
|
||||
new MemoryCache(new MemoryCacheOptions()),
|
||||
pathReplacementService,
|
||||
Substitute.For<IFallbackMetadataProvider>(),
|
||||
new SingleResponseHttpClientFactory(failingResponse),
|
||||
Substitute.For<ILogger<JellyfinApiClient>>());
|
||||
|
||||
var failingLibrary = new JellyfinLibrary
|
||||
{
|
||||
Id = 42,
|
||||
MediaSourceId = 1,
|
||||
ItemId = "library-1",
|
||||
PathInfos = [new JellyfinPathInfo { NetworkPath = "/network", Path = "/local" }]
|
||||
};
|
||||
|
||||
// same client instance, a second library whose items project cleanly
|
||||
var healthyLibrary = new JellyfinLibrary { Id = 43, MediaSourceId = 1, ItemId = "library-2" };
|
||||
|
||||
var failingCounter = new MediaServerProjectionFailureCounter();
|
||||
await foreach ((MusicVideo _, int _) in client.GetMusicVideoLibraryItems(
|
||||
"http://jellyfin.example",
|
||||
"MediaBrowser Token=abc",
|
||||
failingLibrary,
|
||||
failingCounter))
|
||||
{
|
||||
// no items survive the failing projection
|
||||
}
|
||||
|
||||
var healthyCounter = new MediaServerProjectionFailureCounter();
|
||||
await foreach ((MusicVideo _, int _) in client.GetMusicVideoLibraryItems(
|
||||
"http://jellyfin.example",
|
||||
"MediaBrowser Token=abc",
|
||||
healthyLibrary,
|
||||
healthyCounter))
|
||||
{
|
||||
// the healthy library has no PathInfos, so nothing throws
|
||||
}
|
||||
|
||||
failingCounter.Count.ShouldBe(1);
|
||||
healthyCounter.Count.ShouldBe(0);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class SingleResponseHttpClientFactory(string response) : IHttpClientFactory
|
||||
{
|
||||
public HttpClient CreateClient(string name) => new(new SingleResponseHttpMessageHandler(response));
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.Globalization;
|
||||
using System.Globalization;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Emby;
|
||||
@@ -72,7 +72,8 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
public IAsyncEnumerable<Tuple<EmbyMovie, int>> GetMovieLibraryItems(
|
||||
string address,
|
||||
string apiKey,
|
||||
EmbyLibrary library)
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null)
|
||||
=> GetPagedLibraryContents(
|
||||
address,
|
||||
library,
|
||||
@@ -82,12 +83,14 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
itemId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToMovie(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToMovie(lib, item)),
|
||||
projectionFailures);
|
||||
|
||||
public IAsyncEnumerable<Tuple<EmbyShow, int>> GetShowLibraryItems(
|
||||
string address,
|
||||
string apiKey,
|
||||
EmbyLibrary library)
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null)
|
||||
=> GetPagedLibraryContents(
|
||||
address,
|
||||
library,
|
||||
@@ -97,7 +100,8 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
itemId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(_, item) => ProjectToShow(item));
|
||||
(_, item) => ProjectToShow(item),
|
||||
projectionFailures);
|
||||
|
||||
public IAsyncEnumerable<Tuple<EmbySeason, int>> GetSeasonLibraryItems(
|
||||
string address,
|
||||
@@ -129,7 +133,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
seasonId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToEpisode(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToEpisode(lib, item)));
|
||||
|
||||
public IAsyncEnumerable<Tuple<EmbyCollection, int>> GetCollectionLibraryItems(string address, string apiKey)
|
||||
{
|
||||
@@ -208,7 +212,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
|
||||
foreach (EmbyLibraryItemResponse item in itemsResponse.Items)
|
||||
{
|
||||
return ProjectToShow(item);
|
||||
return ProjectToShow(item).ToOption();
|
||||
}
|
||||
|
||||
return BaseError.New($"Unable to locate show with id {showId}");
|
||||
@@ -251,7 +255,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
|
||||
foreach (EmbyLibraryItemResponse item in detailResponse.Items)
|
||||
{
|
||||
Option<EmbyShow> maybeShow = ProjectToShow(item);
|
||||
Option<EmbyShow> maybeShow = ProjectToShow(item).ToOption();
|
||||
foreach (EmbyShow show in maybeShow)
|
||||
{
|
||||
shows.Add(show);
|
||||
@@ -269,12 +273,20 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
}
|
||||
}
|
||||
|
||||
// #484: a library whose id could not be resolved is not a deliberate per-item skip, so fail closed —
|
||||
// it must never look like a clean "this item was deleted upstream" to a reconciliation sweep.
|
||||
private static MediaServerProjectionResult<TItem> WithLibrary<TItem>(
|
||||
Option<EmbyLibrary> maybeLibrary,
|
||||
Func<EmbyLibrary, MediaServerProjectionResult<TItem>> project) =>
|
||||
maybeLibrary.Match(project, MediaServerProjectionResult<TItem>.Failed);
|
||||
|
||||
private static async IAsyncEnumerable<Tuple<TItem, int>> GetPagedLibraryContents<TItem>(
|
||||
string address,
|
||||
Option<EmbyLibrary> maybeLibrary,
|
||||
string parentId,
|
||||
Func<IEmbyApi, string, int, int, Task<EmbyLibraryItemsResponse>> getItems,
|
||||
Func<Option<EmbyLibrary>, EmbyLibraryItemResponse, Option<TItem>> mapper)
|
||||
Func<Option<EmbyLibrary>, EmbyLibraryItemResponse, MediaServerProjectionResult<TItem>> mapper,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null)
|
||||
{
|
||||
IEmbyApi service = RestService.For<IEmbyApi>(address);
|
||||
const int PAGE_SIZE = 10;
|
||||
@@ -290,15 +302,29 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
pages = Math.Min(pages, (result.TotalRecordCount - 1) / PAGE_SIZE + 1);
|
||||
|
||||
#pragma warning disable VSTHRD003
|
||||
foreach (TItem item in result.Items.Map(item => mapper(maybeLibrary, item)).Somes())
|
||||
foreach (EmbyLibraryItemResponse response in result.Items)
|
||||
#pragma warning restore VSTHRD003
|
||||
{
|
||||
yield return new Tuple<TItem, int>(item, result.TotalRecordCount);
|
||||
MediaServerProjectionResult<TItem> projection = mapper(maybeLibrary, response);
|
||||
|
||||
// #484: a swallowed projection failure silently drops an item the server DID return, which
|
||||
// is indistinguishable from a deletion downstream. Report it so the sweep can refuse.
|
||||
// Deliberate skips (no media sources, virtual items, unsupported type) are NOT counted.
|
||||
if (projection.IsFailure)
|
||||
{
|
||||
projectionFailures?.RecordFailure();
|
||||
}
|
||||
|
||||
foreach (TItem item in projection.Item)
|
||||
{
|
||||
yield return new Tuple<TItem, int>(item, result.TotalRecordCount);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private Option<EmbyCollection> ProjectToCollection(EmbyLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<EmbyCollection> ProjectToCollection(EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -312,27 +338,27 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby collection");
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyCollection>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
private Option<MediaItem> ProjectToCollectionMediaItem(EmbyLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<MediaItem> ProjectToCollectionMediaItem(EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
return item.Type switch
|
||||
{
|
||||
"Movie" => new EmbyMovie { ItemId = item.Id },
|
||||
"Series" => new EmbyShow { ItemId = item.Id },
|
||||
"Season" => new EmbySeason { ItemId = item.Id },
|
||||
"Episode" => new EmbyEpisode { ItemId = item.Id },
|
||||
_ => Option<MediaItem>.None
|
||||
"Movie" => MediaServerProjectionResult<MediaItem>.Projected(new EmbyMovie { ItemId = item.Id }),
|
||||
"Series" => MediaServerProjectionResult<MediaItem>.Projected(new EmbyShow { ItemId = item.Id }),
|
||||
"Season" => MediaServerProjectionResult<MediaItem>.Projected(new EmbySeason { ItemId = item.Id }),
|
||||
"Episode" => MediaServerProjectionResult<MediaItem>.Projected(new EmbyEpisode { ItemId = item.Id }),
|
||||
_ => MediaServerProjectionResult<MediaItem>.Skipped()
|
||||
};
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby collection media item");
|
||||
return None;
|
||||
return MediaServerProjectionResult<MediaItem>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -377,13 +403,15 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
return None;
|
||||
}
|
||||
|
||||
private Option<EmbyMovie> ProjectToMovie(EmbyLibrary library, EmbyLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<EmbyMovie> ProjectToMovie(
|
||||
EmbyLibrary library,
|
||||
EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.MediaSources is null || item.MediaSources.Count == 0)
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyMovie>.Skipped();
|
||||
}
|
||||
|
||||
string path = item.Path ?? string.Empty;
|
||||
@@ -434,7 +462,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby movie");
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyMovie>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -564,7 +592,8 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
return new Writer { Name = person.Name };
|
||||
}
|
||||
|
||||
private Option<EmbyShow> ProjectToShow(EmbyLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<EmbyShow> ProjectToShow(EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -583,7 +612,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby show");
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyShow>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -657,7 +686,8 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
return metadata;
|
||||
}
|
||||
|
||||
private Option<EmbySeason> ProjectToSeason(EmbyLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<EmbySeason> ProjectToSeason(EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -716,17 +746,19 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby show");
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbySeason>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
private Option<EmbyEpisode> ProjectToEpisode(EmbyLibrary library, EmbyLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<EmbyEpisode> ProjectToEpisode(
|
||||
EmbyLibrary library,
|
||||
EmbyLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.LocationType == "Virtual")
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyEpisode>.Skipped();
|
||||
}
|
||||
|
||||
string path = item.Path ?? string.Empty;
|
||||
@@ -777,7 +809,7 @@ public class EmbyApiClient : IEmbyApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Emby movie");
|
||||
return None;
|
||||
return MediaServerProjectionResult<EmbyEpisode>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -79,7 +79,8 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
public IAsyncEnumerable<Tuple<JellyfinMovie, int>> GetMovieLibraryItems(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library) =>
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null) =>
|
||||
GetPagedLibraryItems(
|
||||
"JF Movies",
|
||||
address,
|
||||
@@ -91,12 +92,14 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
itemId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToMovie(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToMovie(lib, item)),
|
||||
projectionFailures);
|
||||
|
||||
public IAsyncEnumerable<Tuple<MusicVideo, int>> GetMusicVideoLibraryItems(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library) =>
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null) =>
|
||||
GetPagedLibraryItems(
|
||||
"JF Music Videos",
|
||||
address,
|
||||
@@ -108,12 +111,14 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
itemId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToMusicVideo(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToMusicVideo(lib, item)),
|
||||
projectionFailures);
|
||||
|
||||
public IAsyncEnumerable<Tuple<JellyfinShow, int>> GetShowLibraryItemsWithoutPeople(
|
||||
string address,
|
||||
string authorizationHeader,
|
||||
JellyfinLibrary library) =>
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null) =>
|
||||
GetPagedLibraryItems(
|
||||
"JF Shows",
|
||||
address,
|
||||
@@ -125,7 +130,8 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
itemId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(_, item) => ProjectToShow(item));
|
||||
(_, item) => ProjectToShow(item),
|
||||
projectionFailures);
|
||||
|
||||
public IAsyncEnumerable<Tuple<JellyfinSeason, int>> GetSeasonLibraryItems(
|
||||
string address,
|
||||
@@ -161,7 +167,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
seasonId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToEpisode(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToEpisode(lib, item)));
|
||||
|
||||
public IAsyncEnumerable<Tuple<JellyfinEpisode, int>> GetEpisodeLibraryItemsWithoutPeople(
|
||||
string address,
|
||||
@@ -179,7 +185,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
seasonId,
|
||||
startIndex: skip,
|
||||
limit: pageSize),
|
||||
(maybeLibrary, item) => maybeLibrary.Map(lib => ProjectToEpisode(lib, item)).Flatten());
|
||||
(maybeLibrary, item) => WithLibrary(maybeLibrary, lib => ProjectToEpisode(lib, item)));
|
||||
|
||||
public IAsyncEnumerable<Tuple<JellyfinCollection, int>> GetCollectionLibraryItems(
|
||||
string address,
|
||||
@@ -269,7 +275,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
|
||||
foreach (JellyfinLibraryItemResponse item in itemsResponse.Items)
|
||||
{
|
||||
return ProjectToShow(item);
|
||||
return ProjectToShow(item).ToOption();
|
||||
}
|
||||
|
||||
return BaseError.New($"Unable to locate show with id {showId}");
|
||||
@@ -313,7 +319,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
|
||||
foreach (JellyfinLibraryItemResponse item in detailResponse.Items)
|
||||
{
|
||||
Option<JellyfinShow> maybeShow = ProjectToShow(item);
|
||||
Option<JellyfinShow> maybeShow = ProjectToShow(item).ToOption();
|
||||
foreach (JellyfinShow show in maybeShow)
|
||||
{
|
||||
shows.Add(show);
|
||||
@@ -353,7 +359,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
|
||||
foreach (JellyfinLibraryItemResponse item in itemsResponse.Items)
|
||||
{
|
||||
return ProjectToEpisode(library, item);
|
||||
return ProjectToEpisode(library, item).ToOption();
|
||||
}
|
||||
|
||||
return BaseError.New($"Unable to locate episode with id {episodeId}");
|
||||
@@ -366,6 +372,13 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
}
|
||||
}
|
||||
|
||||
// #484: a library whose id could not be resolved is not a deliberate per-item skip, so fail closed —
|
||||
// it must never look like a clean "this item was deleted upstream" to a reconciliation sweep.
|
||||
private static MediaServerProjectionResult<TItem> WithLibrary<TItem>(
|
||||
Option<JellyfinLibrary> maybeLibrary,
|
||||
Func<JellyfinLibrary, MediaServerProjectionResult<TItem>> project) =>
|
||||
maybeLibrary.Match(project, MediaServerProjectionResult<TItem>.Failed);
|
||||
|
||||
private async IAsyncEnumerable<Tuple<TItem, int>> GetPagedLibraryItems<TItem>(
|
||||
string pageDescription,
|
||||
string address,
|
||||
@@ -373,7 +386,8 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
int mediaSourceId,
|
||||
string parentId,
|
||||
Func<IJellyfinApi, string, int, int, Task<JellyfinLibraryItemsResponse>> getItems,
|
||||
Func<Option<JellyfinLibrary>, JellyfinLibraryItemResponse, Option<TItem>> mapper)
|
||||
Func<Option<JellyfinLibrary>, JellyfinLibraryItemResponse, MediaServerProjectionResult<TItem>> mapper,
|
||||
MediaServerProjectionFailureCounter projectionFailures = null)
|
||||
{
|
||||
IJellyfinApi service = ServiceForAddress(address);
|
||||
|
||||
@@ -396,41 +410,55 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
// update page count
|
||||
pages = Math.Min(pages, (result.TotalRecordCount - 1) / SystemEnvironment.JellyfinPageSize + 1);
|
||||
|
||||
foreach (TItem item in result.Items.Map(item => mapper(maybeLibrary, item)).Somes())
|
||||
foreach (JellyfinLibraryItemResponse response in result.Items)
|
||||
{
|
||||
yield return new Tuple<TItem, int>(item, result.TotalRecordCount);
|
||||
MediaServerProjectionResult<TItem> projection = mapper(maybeLibrary, response);
|
||||
|
||||
// #484: a swallowed projection failure silently drops an item the server DID return, which
|
||||
// is indistinguishable from a deletion downstream. Report it so the sweep can refuse.
|
||||
// Deliberate skips (STRM, virtual, unsupported type) are NOT counted here.
|
||||
if (projection.IsFailure)
|
||||
{
|
||||
projectionFailures?.RecordFailure();
|
||||
}
|
||||
|
||||
foreach (TItem item in projection.Item)
|
||||
{
|
||||
yield return new Tuple<TItem, int>(item, result.TotalRecordCount);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private Option<MediaItem> ProjectToCollectionMediaItem(JellyfinLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<MediaItem> ProjectToCollectionMediaItem(JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.LocationType != "FileSystem")
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<MediaItem>.Skipped();
|
||||
}
|
||||
|
||||
if (Path.GetExtension(item.Path)?.ToLowerInvariant() == ".strm")
|
||||
{
|
||||
_logger.LogWarning("STRM files are not supported; skipping {Path}", item.Path);
|
||||
return None;
|
||||
return MediaServerProjectionResult<MediaItem>.Skipped();
|
||||
}
|
||||
|
||||
return item.Type switch
|
||||
{
|
||||
"Movie" => new JellyfinMovie { ItemId = item.Id },
|
||||
"Series" => new JellyfinShow { ItemId = item.Id },
|
||||
"Season" => new JellyfinSeason { ItemId = item.Id },
|
||||
"Episode" => new JellyfinEpisode { ItemId = item.Id },
|
||||
_ => None
|
||||
"Movie" => MediaServerProjectionResult<MediaItem>.Projected(new JellyfinMovie { ItemId = item.Id }),
|
||||
"Series" => MediaServerProjectionResult<MediaItem>.Projected(new JellyfinShow { ItemId = item.Id }),
|
||||
"Season" => MediaServerProjectionResult<MediaItem>.Projected(new JellyfinSeason { ItemId = item.Id }),
|
||||
"Episode" => MediaServerProjectionResult<MediaItem>.Projected(
|
||||
new JellyfinEpisode { ItemId = item.Id }),
|
||||
_ => MediaServerProjectionResult<MediaItem>.Skipped()
|
||||
};
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin collection media item");
|
||||
return None;
|
||||
return MediaServerProjectionResult<MediaItem>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -511,19 +539,21 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
return None;
|
||||
}
|
||||
|
||||
private Option<JellyfinMovie> ProjectToMovie(JellyfinLibrary library, JellyfinLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<JellyfinMovie> ProjectToMovie(
|
||||
JellyfinLibrary library,
|
||||
JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.LocationType != "FileSystem")
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinMovie>.Skipped();
|
||||
}
|
||||
|
||||
if (Path.GetExtension(item.Path)?.ToLowerInvariant() == ".strm")
|
||||
{
|
||||
_logger.LogInformation("STRM files are not supported; skipping {Path}", item.Path);
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinMovie>.Skipped();
|
||||
}
|
||||
|
||||
string path = item.Path ?? string.Empty;
|
||||
@@ -574,7 +604,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin movie");
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinMovie>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -663,19 +693,21 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
return metadata;
|
||||
}
|
||||
|
||||
private Option<MusicVideo> ProjectToMusicVideo(JellyfinLibrary library, JellyfinLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<MusicVideo> ProjectToMusicVideo(
|
||||
JellyfinLibrary library,
|
||||
JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.LocationType != "FileSystem")
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<MusicVideo>.Skipped();
|
||||
}
|
||||
|
||||
if (Path.GetExtension(item.Path)?.ToLowerInvariant() == ".strm")
|
||||
{
|
||||
_logger.LogInformation("STRM files are not supported; skipping {Path}", item.Path);
|
||||
return None;
|
||||
return MediaServerProjectionResult<MusicVideo>.Skipped();
|
||||
}
|
||||
|
||||
string path = item.Path ?? string.Empty;
|
||||
@@ -724,7 +756,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin music video");
|
||||
return None;
|
||||
return MediaServerProjectionResult<MusicVideo>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -843,7 +875,8 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
return new Writer { Name = person.Name };
|
||||
}
|
||||
|
||||
private Option<JellyfinShow> ProjectToShow(JellyfinLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<JellyfinShow> ProjectToShow(JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -862,7 +895,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin show");
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinShow>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -936,7 +969,8 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
return metadata;
|
||||
}
|
||||
|
||||
private Option<JellyfinSeason> ProjectToSeason(JellyfinLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<JellyfinSeason> ProjectToSeason(JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -1005,11 +1039,12 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin season");
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinSeason>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
private Option<JellyfinCollection> ProjectToCollection(JellyfinLibraryItemResponse item)
|
||||
// NOTE (#484): no deliberate-skip guard clause here — every drop from this projection is a failure.
|
||||
private MediaServerProjectionResult<JellyfinCollection> ProjectToCollection(JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -1023,23 +1058,25 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin collection");
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinCollection>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
private Option<JellyfinEpisode> ProjectToEpisode(JellyfinLibrary library, JellyfinLibraryItemResponse item)
|
||||
private MediaServerProjectionResult<JellyfinEpisode> ProjectToEpisode(
|
||||
JellyfinLibrary library,
|
||||
JellyfinLibraryItemResponse item)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (item.LocationType != "FileSystem")
|
||||
{
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinEpisode>.Skipped();
|
||||
}
|
||||
|
||||
if (Path.GetExtension(item.Path)?.ToLowerInvariant() == ".strm")
|
||||
{
|
||||
_logger.LogWarning("STRM files are not supported; skipping {Path}", item.Path);
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinEpisode>.Skipped();
|
||||
}
|
||||
|
||||
string path = item.Path ?? string.Empty;
|
||||
@@ -1090,7 +1127,7 @@ public class JellyfinApiClient : IJellyfinApiClient
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Error projecting Jellyfin episode");
|
||||
return None;
|
||||
return MediaServerProjectionResult<JellyfinEpisode>.Failed();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -81,7 +81,9 @@ public class MediaServerMovieLibraryScannerTests
|
||||
CancellationToken.None);
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<JellyfinMovie, int>> GetMovieLibraryItems(
|
||||
JellyfinConnectionParameters connectionParameters, JellyfinLibrary library) => EmptyMovies();
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) => EmptyMovies();
|
||||
|
||||
protected override string MediaServerItemId(JellyfinMovie movie) => movie.ItemId;
|
||||
protected override string MediaServerEtag(JellyfinMovie movie) => movie.Etag;
|
||||
|
||||
@@ -79,7 +79,9 @@ public class MediaServerOtherVideoLibraryScannerTests
|
||||
CancellationToken.None);
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<PlexOtherVideo, int>> GetOtherVideoLibraryItems(
|
||||
PlexConnectionParameters connectionParameters, PlexLibrary library) => EmptyOtherVideos();
|
||||
PlexConnectionParameters connectionParameters,
|
||||
PlexLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) => EmptyOtherVideos();
|
||||
|
||||
protected override string MediaServerItemId(PlexOtherVideo otherVideo) => otherVideo.Key;
|
||||
protected override string MediaServerEtag(PlexOtherVideo otherVideo) => otherVideo.Etag;
|
||||
|
||||
@@ -6,9 +6,10 @@ using Shouldly;
|
||||
|
||||
namespace ErsatzTV.Scanner.Tests.Core.Metadata;
|
||||
|
||||
// #477: the deterministic policy behind the media-server anti-nuke guard. An empty incoming set with
|
||||
// existing items present is the only case that skips the sweep (and logs); every other combination
|
||||
// reconciles normally.
|
||||
// #477/#484: the deterministic policy behind the media-server anti-nuke guard. Two cases skip the sweep
|
||||
// (and log), both requiring existing items to be at risk: an EMPTY incoming set (#477) and a non-zero
|
||||
// PROJECTION FAILURE count (#484). Every other combination reconciles normally — in particular a short
|
||||
// incoming set caused by deliberate skips, and a genuine bulk deletion, both still sweep.
|
||||
public class MediaServerReconciliationGuardTests
|
||||
{
|
||||
[Test]
|
||||
@@ -66,4 +67,84 @@ public class MediaServerReconciliationGuardTests
|
||||
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 3, 0).ShouldBeTrue();
|
||||
}
|
||||
|
||||
// #484: an item whose projection threw was silently dropped, so it is missing from the incoming set
|
||||
// even though the server DID return it. Sweeping here would flag a healthy item FileNotFound.
|
||||
[Test]
|
||||
public void Projection_Failure_Skips_And_Warns()
|
||||
{
|
||||
var logger = Substitute.For<ILogger>();
|
||||
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 4, 5, 1).ShouldBeFalse();
|
||||
|
||||
logger.Received(1).Log(
|
||||
LogLevel.Warning,
|
||||
Arg.Any<EventId>(),
|
||||
Arg.Any<object>(),
|
||||
Arg.Any<Exception>(),
|
||||
Arg.Any<Func<object, Exception?, string>>());
|
||||
}
|
||||
|
||||
// #484 THE regression guard. Deliberate guard-clause skips (a STRM file, a virtual item, an
|
||||
// unsupported type) are NOT projection failures: they never reach the counter, so a short incoming
|
||||
// set caused only by them must still sweep. If skips were counted, one STRM file in a library would
|
||||
// permanently disable that library's reconciliation and stale rows would accumulate forever.
|
||||
[Test]
|
||||
public void Deliberate_Skips_Do_Not_Suppress_The_Sweep()
|
||||
{
|
||||
var logger = Substitute.For<ILogger>();
|
||||
|
||||
// 5 exist locally, the server returned 5, 2 were deliberately skipped => 3 incoming, 0 failures
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 3, 5, 0).ShouldBeTrue();
|
||||
|
||||
logger.DidNotReceive().Log(
|
||||
LogLevel.Warning,
|
||||
Arg.Any<EventId>(),
|
||||
Arg.Any<object>(),
|
||||
Arg.Any<Exception>(),
|
||||
Arg.Any<Func<object, Exception?, string>>());
|
||||
}
|
||||
|
||||
// a genuine bulk deletion produces zero projection failures, so it is unaffected by #484 — this is
|
||||
// why the rejected ratio threshold was not needed to tell the two apart.
|
||||
[Test]
|
||||
public void Bulk_Deletion_With_No_Failures_Still_Flags()
|
||||
{
|
||||
var logger = Substitute.For<ILogger>();
|
||||
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 1, 500, 0).ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void Projection_Failure_With_No_Existing_Items_Is_A_Noop_Sweep()
|
||||
{
|
||||
var logger = Substitute.For<ILogger>();
|
||||
|
||||
// nothing exists locally, so the sweep flags nothing either way — don't emit the scary warning
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 2, 0, 3).ShouldBeTrue();
|
||||
|
||||
logger.DidNotReceive().Log(
|
||||
LogLevel.Warning,
|
||||
Arg.Any<EventId>(),
|
||||
Arg.Any<object>(),
|
||||
Arg.Any<Exception>(),
|
||||
Arg.Any<Func<object, Exception?, string>>());
|
||||
}
|
||||
|
||||
// both refusals can be true at once (a fetch that dropped everything it returned); either is
|
||||
// sufficient, and the failure branch is reported because it names the actual cause.
|
||||
[Test]
|
||||
public void Projection_Failure_Wins_Over_The_Zero_Incoming_Message()
|
||||
{
|
||||
var logger = Substitute.For<ILogger>();
|
||||
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(logger, "Movies", 0, 5, 2).ShouldBeFalse();
|
||||
|
||||
logger.Received(1).Log(
|
||||
LogLevel.Warning,
|
||||
Arg.Any<EventId>(),
|
||||
Arg.Is<object>(o => o.ToString()!.Contains("silently dropped")),
|
||||
Arg.Any<Exception>(),
|
||||
Arg.Any<Func<object, Exception?, string>>());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,6 +134,57 @@ public class MediaServerTelevisionLibraryScannerTests
|
||||
Arg.Any<int[]>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
// #484: the same partial deletion as Removed_Show_Cascades_..., but the api client silently dropped
|
||||
// an item whose projection threw. "show-6366" is missing from the incoming list only because we
|
||||
// failed to build it, not because the server stopped reporting it — flagging it would trash a
|
||||
// healthy show (and, via the #476 cascade, all of its seasons and episodes). Paired with the
|
||||
// cascade test above as a positive control: identical arrangement, only the counter differs.
|
||||
[Test]
|
||||
public async Task Projection_Failure_Suppresses_The_Partial_Deletion_Sweep()
|
||||
{
|
||||
var televisionRepository = Substitute.For<IJellyfinTelevisionRepository>();
|
||||
var scannerProxy = Substitute.For<IScannerProxy>();
|
||||
|
||||
var library = new JellyfinLibrary { Id = 42, Name = "TV Shows" };
|
||||
|
||||
televisionRepository.GetExistingShows(library, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<JellyfinItemEtag>
|
||||
{
|
||||
new() { ItemId = "show-keep", State = MediaItemState.Normal },
|
||||
new() { ItemId = "show-6366", State = MediaItemState.Normal }
|
||||
});
|
||||
|
||||
televisionRepository.GetOrAdd(library, Arg.Any<JellyfinShow>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Left<BaseError, MediaItemScanResult<JellyfinShow>>(BaseError.New("skip metadata in test")));
|
||||
|
||||
scannerProxy.UpdateProgress(Arg.Any<decimal>(), Arg.Any<CancellationToken>()).Returns(true);
|
||||
|
||||
var projectionFailures = new MediaServerProjectionFailureCounter();
|
||||
projectionFailures.RecordFailure();
|
||||
|
||||
var scanner = new TestTelevisionLibraryScanner(scannerProxy);
|
||||
|
||||
Either<BaseError, Unit> result = await scanner.Scan(
|
||||
televisionRepository,
|
||||
new JellyfinConnectionParameters("http://jellyfin.example", "abc", 7),
|
||||
library,
|
||||
Shows(new JellyfinShow
|
||||
{
|
||||
ItemId = "show-keep",
|
||||
ShowMetadata = new List<ShowMetadata> { new() { Title = "Keeper" } }
|
||||
}),
|
||||
projectionFailures);
|
||||
|
||||
result.IsRight.ShouldBeTrue();
|
||||
|
||||
await televisionRepository.DidNotReceive().FlagFileNotFoundShows(
|
||||
Arg.Any<JellyfinLibrary>(), Arg.Any<List<string>>(), Arg.Any<CancellationToken>());
|
||||
await televisionRepository.DidNotReceive().FlagFileNotFoundSeasonsForShows(
|
||||
Arg.Any<List<int>>(), Arg.Any<CancellationToken>());
|
||||
await televisionRepository.DidNotReceive().FlagFileNotFoundEpisodesForSeasons(
|
||||
Arg.Any<List<int>>(), Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
private static async IAsyncEnumerable<Tuple<JellyfinShow, int>> EmptyShows()
|
||||
{
|
||||
await Task.CompletedTask;
|
||||
@@ -172,18 +223,22 @@ public class MediaServerTelevisionLibraryScannerTests
|
||||
JellyfinItemEtag> televisionRepository,
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
JellyfinLibrary library,
|
||||
IAsyncEnumerable<Tuple<JellyfinShow, int>> showEntries) =>
|
||||
IAsyncEnumerable<Tuple<JellyfinShow, int>> showEntries,
|
||||
MediaServerProjectionFailureCounter? projectionFailures = null) =>
|
||||
ScanLibrary(
|
||||
televisionRepository,
|
||||
connectionParameters,
|
||||
library,
|
||||
_ => string.Empty,
|
||||
showEntries,
|
||||
projectionFailures ?? new MediaServerProjectionFailureCounter(),
|
||||
false,
|
||||
CancellationToken.None);
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<JellyfinShow, int>> GetShowLibraryItems(
|
||||
JellyfinConnectionParameters connectionParameters, JellyfinLibrary library) =>
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
protected override string MediaServerItemId(JellyfinShow show) => show.ItemId;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.IO.Abstractions;
|
||||
using System.IO.Abstractions;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Emby;
|
||||
@@ -81,11 +81,13 @@ public class EmbyMovieLibraryScanner :
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<EmbyMovie, int>> GetMovieLibraryItems(
|
||||
EmbyConnectionParameters connectionParameters,
|
||||
EmbyLibrary library) =>
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_embyApiClient.GetMovieLibraryItems(
|
||||
connectionParameters.Address,
|
||||
connectionParameters.ApiKey,
|
||||
library);
|
||||
library,
|
||||
projectionFailures);
|
||||
|
||||
protected override Task<Option<MovieMetadata>> GetFullMetadata(
|
||||
EmbyConnectionParameters connectionParameters,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.IO.Abstractions;
|
||||
using System.IO.Abstractions;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Emby;
|
||||
@@ -133,8 +133,13 @@ public class EmbyTelevisionLibraryScanner : MediaServerTelevisionLibraryScanner<
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<EmbyShow, int>> GetShowLibraryItems(
|
||||
EmbyConnectionParameters connectionParameters,
|
||||
EmbyLibrary library) =>
|
||||
_embyApiClient.GetShowLibraryItems(connectionParameters.Address, connectionParameters.ApiKey, library);
|
||||
EmbyLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_embyApiClient.GetShowLibraryItems(
|
||||
connectionParameters.Address,
|
||||
connectionParameters.ApiKey,
|
||||
library,
|
||||
projectionFailures);
|
||||
|
||||
protected override string MediaServerItemId(EmbyShow show) => show.ItemId;
|
||||
protected override string MediaServerItemId(EmbySeason season) => season.ItemId;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.IO.Abstractions;
|
||||
using System.IO.Abstractions;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Extensions;
|
||||
@@ -81,11 +81,13 @@ public class JellyfinMovieLibraryScanner :
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<JellyfinMovie, int>> GetMovieLibraryItems(
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
JellyfinLibrary library) =>
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_jellyfinApiClient.GetMovieLibraryItems(
|
||||
connectionParameters.Address,
|
||||
connectionParameters.AuthorizationHeader,
|
||||
library);
|
||||
library,
|
||||
projectionFailures);
|
||||
|
||||
protected override Task<Option<MovieMetadata>> GetFullMetadata(
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
|
||||
@@ -79,11 +79,18 @@ public class JellyfinMusicVideoLibraryScanner : IJellyfinMusicVideoLibraryScanne
|
||||
|
||||
var processed = 0;
|
||||
var incomingPaths = new List<string>();
|
||||
|
||||
// #484: one counter per enumeration, created here and read only after the enumeration completes.
|
||||
// It is never a field on the (singleton) api client, so concurrent scans of different libraries
|
||||
// cannot leak failures into each other's sweep decision.
|
||||
var projectionFailures = new MediaServerProjectionFailureCounter();
|
||||
|
||||
await foreach ((MusicVideo incoming, int totalCount) in _jellyfinApiClient
|
||||
.GetMusicVideoLibraryItems(
|
||||
connectionParameters.Address,
|
||||
connectionParameters.AuthorizationHeader,
|
||||
library)
|
||||
library,
|
||||
projectionFailures)
|
||||
.WithCancellation(cancellationToken))
|
||||
{
|
||||
if (cancellationToken.IsCancellationRequested)
|
||||
@@ -118,7 +125,12 @@ public class JellyfinMusicVideoLibraryScanner : IJellyfinMusicVideoLibraryScanne
|
||||
}
|
||||
}
|
||||
|
||||
await TrashMissingMusicVideos(library, libraryPath, incomingPaths, cancellationToken);
|
||||
await TrashMissingMusicVideos(
|
||||
library,
|
||||
libraryPath,
|
||||
incomingPaths,
|
||||
projectionFailures,
|
||||
cancellationToken);
|
||||
|
||||
return Unit.Default;
|
||||
}
|
||||
@@ -141,17 +153,21 @@ public class JellyfinMusicVideoLibraryScanner : IJellyfinMusicVideoLibraryScanne
|
||||
JellyfinLibrary library,
|
||||
LibraryPath libraryPath,
|
||||
List<string> incomingPaths,
|
||||
MediaServerProjectionFailureCounter projectionFailures,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
var existingPaths = (await _musicVideoRepository.FindMusicVideoPaths(libraryPath)).ToList();
|
||||
|
||||
// #477: refuse the sweep when a successful fetch returned zero items but rows exist locally — an empty
|
||||
// incoming set is indistinguishable from a transient error and would otherwise wipe the whole library.
|
||||
// #484: also refuse when the api client silently dropped items whose projection threw — those are
|
||||
// items Jellyfin DID return, so treating them as deletions would hard-delete healthy rows.
|
||||
if (!MediaServerReconciliationGuard.ShouldFlagMissing(
|
||||
_logger,
|
||||
library.Name,
|
||||
incomingPaths.Count,
|
||||
existingPaths.Count))
|
||||
existingPaths.Count,
|
||||
projectionFailures.Count))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.IO.Abstractions;
|
||||
using System.IO.Abstractions;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
using ErsatzTV.Core.Errors;
|
||||
@@ -137,11 +137,13 @@ public class JellyfinTelevisionLibraryScanner : MediaServerTelevisionLibraryScan
|
||||
|
||||
protected override IAsyncEnumerable<Tuple<JellyfinShow, int>> GetShowLibraryItems(
|
||||
JellyfinConnectionParameters connectionParameters,
|
||||
JellyfinLibrary library) =>
|
||||
JellyfinLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_jellyfinApiClient.GetShowLibraryItemsWithoutPeople(
|
||||
connectionParameters.Address,
|
||||
connectionParameters.AuthorizationHeader,
|
||||
library);
|
||||
library,
|
||||
projectionFailures);
|
||||
|
||||
protected override string MediaServerItemId(JellyfinShow show) => show.ItemId;
|
||||
protected override string MediaServerItemId(JellyfinSeason season) => season.ItemId;
|
||||
|
||||
@@ -52,12 +52,18 @@ public abstract class MediaServerMovieLibraryScanner<TConnectionParameters, TLib
|
||||
{
|
||||
try
|
||||
{
|
||||
// #484: one counter per enumeration, created here and read only after the enumeration
|
||||
// completes. It is never a field on the (singleton) api client, so concurrent scans of
|
||||
// different libraries cannot leak failures into each other's sweep decision.
|
||||
var projectionFailures = new MediaServerProjectionFailureCounter();
|
||||
|
||||
return await ScanLibrary(
|
||||
movieRepository,
|
||||
connectionParameters,
|
||||
library,
|
||||
getLocalPath,
|
||||
GetMovieLibraryItems(connectionParameters, library),
|
||||
GetMovieLibraryItems(connectionParameters, library, projectionFailures),
|
||||
projectionFailures,
|
||||
deepScan,
|
||||
cancellationToken);
|
||||
}
|
||||
@@ -73,6 +79,7 @@ public abstract class MediaServerMovieLibraryScanner<TConnectionParameters, TLib
|
||||
TLibrary library,
|
||||
Func<TMovie, string> getLocalPath,
|
||||
IAsyncEnumerable<Tuple<TMovie, int>> movieEntries,
|
||||
MediaServerProjectionFailureCounter projectionFailures,
|
||||
bool deepScan,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
@@ -204,7 +211,11 @@ public abstract class MediaServerMovieLibraryScanner<TConnectionParameters, TLib
|
||||
|
||||
// trash movies that are no longer present on the media server
|
||||
if (MediaServerReconciliationGuard.ShouldFlagMissing(
|
||||
_logger, library.Name, incomingItemIds.Count, existingMovies.Count))
|
||||
_logger,
|
||||
library.Name,
|
||||
incomingItemIds.Count,
|
||||
existingMovies.Count,
|
||||
projectionFailures.Count))
|
||||
{
|
||||
var fileNotFoundItemIds = existingMovies.Keys.Except(incomingItemIds).ToList();
|
||||
List<int> ids = await movieRepository.FlagFileNotFound(library, fileNotFoundItemIds);
|
||||
@@ -220,9 +231,13 @@ public abstract class MediaServerMovieLibraryScanner<TConnectionParameters, TLib
|
||||
protected abstract string MediaServerItemId(TMovie movie);
|
||||
protected abstract string MediaServerEtag(TMovie movie);
|
||||
|
||||
// #484: projectionFailures is the per-enumeration sink the api client reports swallowed projection
|
||||
// failures into. Implementations that cannot silently drop an item (Plex projects without an
|
||||
// Option/catch, so a bad item throws and unwinds the scan) simply ignore it.
|
||||
protected abstract IAsyncEnumerable<Tuple<TMovie, int>> GetMovieLibraryItems(
|
||||
TConnectionParameters connectionParameters,
|
||||
TLibrary library);
|
||||
TLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures);
|
||||
|
||||
protected abstract Task<Option<MovieMetadata>> GetFullMetadata(
|
||||
TConnectionParameters connectionParameters,
|
||||
|
||||
@@ -52,12 +52,17 @@ public abstract class MediaServerOtherVideoLibraryScanner<TConnectionParameters,
|
||||
{
|
||||
try
|
||||
{
|
||||
// #484: one counter per enumeration — see MediaServerMovieLibraryScanner for why it must
|
||||
// not be shared or ambient.
|
||||
var projectionFailures = new MediaServerProjectionFailureCounter();
|
||||
|
||||
return await ScanLibrary(
|
||||
otherVideoRepository,
|
||||
connectionParameters,
|
||||
library,
|
||||
getLocalPath,
|
||||
GetOtherVideoLibraryItems(connectionParameters, library),
|
||||
GetOtherVideoLibraryItems(connectionParameters, library, projectionFailures),
|
||||
projectionFailures,
|
||||
deepScan,
|
||||
cancellationToken);
|
||||
}
|
||||
@@ -73,6 +78,7 @@ public abstract class MediaServerOtherVideoLibraryScanner<TConnectionParameters,
|
||||
TLibrary library,
|
||||
Func<TOtherVideo, string> getLocalPath,
|
||||
IAsyncEnumerable<Tuple<TOtherVideo, int>> otherVideoEntries,
|
||||
MediaServerProjectionFailureCounter projectionFailures,
|
||||
bool deepScan,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
@@ -211,7 +217,11 @@ public abstract class MediaServerOtherVideoLibraryScanner<TConnectionParameters,
|
||||
|
||||
// trash OtherVideo that are no longer present on the media server
|
||||
if (MediaServerReconciliationGuard.ShouldFlagMissing(
|
||||
_logger, library.Name, incomingItemIds.Count, existingOtherVideos.Count))
|
||||
_logger,
|
||||
library.Name,
|
||||
incomingItemIds.Count,
|
||||
existingOtherVideos.Count,
|
||||
projectionFailures.Count))
|
||||
{
|
||||
var fileNotFoundItemIds = existingOtherVideos.Keys.Except(incomingItemIds).ToList();
|
||||
List<int> ids = await otherVideoRepository.FlagFileNotFound(library, fileNotFoundItemIds);
|
||||
@@ -227,9 +237,11 @@ public abstract class MediaServerOtherVideoLibraryScanner<TConnectionParameters,
|
||||
protected abstract string MediaServerItemId(TOtherVideo otherVideo);
|
||||
protected abstract string MediaServerEtag(TOtherVideo otherVideo);
|
||||
|
||||
// #484: see MediaServerMovieLibraryScanner.GetMovieLibraryItems for the counter's contract.
|
||||
protected abstract IAsyncEnumerable<Tuple<TOtherVideo, int>> GetOtherVideoLibraryItems(
|
||||
TConnectionParameters connectionParameters,
|
||||
TLibrary library);
|
||||
TLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures);
|
||||
|
||||
protected abstract Task<Option<OtherVideoMetadata>> GetFullMetadata(
|
||||
TConnectionParameters connectionParameters,
|
||||
|
||||
@@ -14,16 +14,48 @@ namespace ErsatzTV.Scanner.Core.Metadata;
|
||||
// partial-deletion sweep would otherwise handle (see #476's cascade, which still fires for the common
|
||||
// case where survivors are present and only some items are gone). The cost of not flagging a genuinely
|
||||
// emptied library (stale rows persist until an item returns or the library is removed) is far smaller
|
||||
// than a one-scan permanent wipe. Ratio-thresholds and projection-failure detection are deferred — see
|
||||
// docs/decisions.md and the #477 follow-up.
|
||||
// than a one-scan permanent wipe.
|
||||
//
|
||||
// #484 extended the policy with a SECOND deterministic refusal and rejected the ratio threshold:
|
||||
//
|
||||
// - projection failures (implemented). A media-server API client maps every item the server returned
|
||||
// through a private projection whose `catch` swallows the exception and drops the item. A dropped
|
||||
// item the server DID return is indistinguishable from a deletion here, so one projection
|
||||
// regression could mass-flag a healthy library. When the enumeration that produced `incomingCount`
|
||||
// reports any such failure, the sweep is refused. Deliberate guard-clause skips (STRM files,
|
||||
// virtual items, unsupported types) are NOT failures and never suppress the sweep — see
|
||||
// MediaServerProjectionResult; counting them would permanently disable reconciliation for any
|
||||
// library holding a single STRM file.
|
||||
// - ratio / missing-fraction threshold (REJECTED). It is a two-sided heuristic: set low it silently
|
||||
// suppresses legitimate bulk deletions, set high it misses the partial fetch it exists for, and
|
||||
// there is no telemetry to tune it with. The failure it approximates is exactly observable via the
|
||||
// projection-failure count above, and a genuine bulk deletion produces zero projection failures, so
|
||||
// the deterministic signal has no false positives where the heuristic has unbounded ones.
|
||||
//
|
||||
// See docs/decisions.md `scan.projection-failure-sweep-guard`.
|
||||
internal static class MediaServerReconciliationGuard
|
||||
{
|
||||
public static bool ShouldFlagMissing(
|
||||
ILogger logger,
|
||||
string libraryName,
|
||||
int incomingCount,
|
||||
int existingCount)
|
||||
int existingCount,
|
||||
int projectionFailureCount = 0)
|
||||
{
|
||||
if (projectionFailureCount > 0 && existingCount > 0)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"Media server library {Library} silently dropped {FailureCount} item(s) that failed to "
|
||||
+ "project during this scan ({IncomingCount} usable, {ExistingCount} exist locally); "
|
||||
+ "skipping the file-not-found sweep because a dropped item is indistinguishable from a "
|
||||
+ "deletion and would be flagged as missing",
|
||||
libraryName,
|
||||
projectionFailureCount,
|
||||
incomingCount,
|
||||
existingCount);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (incomingCount == 0 && existingCount > 0)
|
||||
{
|
||||
logger.LogWarning(
|
||||
|
||||
@@ -55,12 +55,18 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
{
|
||||
try
|
||||
{
|
||||
// #484: one counter per enumeration, created here and read only after the enumeration
|
||||
// completes. It is never a field on the (singleton) api client, so concurrent scans of
|
||||
// different libraries cannot leak failures into each other's sweep decision.
|
||||
var projectionFailures = new MediaServerProjectionFailureCounter();
|
||||
|
||||
return await ScanLibrary(
|
||||
televisionRepository,
|
||||
connectionParameters,
|
||||
library,
|
||||
getLocalPath,
|
||||
GetShowLibraryItems(connectionParameters, library),
|
||||
GetShowLibraryItems(connectionParameters, library, projectionFailures),
|
||||
projectionFailures,
|
||||
deepScan,
|
||||
cancellationToken);
|
||||
}
|
||||
@@ -70,9 +76,13 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
}
|
||||
}
|
||||
|
||||
// #484: projectionFailures is the per-enumeration sink the api client reports swallowed projection
|
||||
// failures into. Implementations that cannot silently drop an item (Plex projects without an
|
||||
// Option/catch, so a bad item throws and unwinds the scan) simply ignore it.
|
||||
protected abstract IAsyncEnumerable<Tuple<TShow, int>> GetShowLibraryItems(
|
||||
TConnectionParameters connectionParameters,
|
||||
TLibrary library);
|
||||
TLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures);
|
||||
|
||||
protected abstract string MediaServerItemId(TShow show);
|
||||
protected abstract string MediaServerItemId(TSeason season);
|
||||
@@ -87,6 +97,7 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
TLibrary library,
|
||||
Func<TEpisode, string> getLocalPath,
|
||||
IAsyncEnumerable<Tuple<TShow, int>> showEntries,
|
||||
MediaServerProjectionFailureCounter? projectionFailures,
|
||||
bool deepScan,
|
||||
bool cleanupFileNotFoundItems,
|
||||
CancellationToken cancellationToken)
|
||||
@@ -172,7 +183,11 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
|
||||
if (cleanupFileNotFoundItems &&
|
||||
MediaServerReconciliationGuard.ShouldFlagMissing(
|
||||
_logger, library.Name, incomingItemIds.Count, existingShows.Count))
|
||||
_logger,
|
||||
library.Name,
|
||||
incomingItemIds.Count,
|
||||
existingShows.Count,
|
||||
projectionFailures?.Count ?? 0))
|
||||
{
|
||||
// trash shows that are no longer present on the media server
|
||||
var fileNotFoundItemIds = existingShows.Map(s => s.MediaServerItemId).Except(incomingItemIds).ToList();
|
||||
@@ -201,6 +216,7 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
TLibrary library,
|
||||
Func<TEpisode, string> getLocalPath,
|
||||
IAsyncEnumerable<Tuple<TShow, int>> showEntries,
|
||||
MediaServerProjectionFailureCounter projectionFailures,
|
||||
bool deepScan,
|
||||
CancellationToken cancellationToken) =>
|
||||
await InternalScanLibrary(
|
||||
@@ -209,10 +225,13 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
library,
|
||||
getLocalPath,
|
||||
showEntries,
|
||||
projectionFailures,
|
||||
deepScan,
|
||||
true,
|
||||
cancellationToken);
|
||||
|
||||
// no sweep runs on this path (cleanupFileNotFoundItems: false), so there is nothing for a
|
||||
// projection-failure counter to guard — #484.
|
||||
protected async Task<Either<BaseError, Unit>> ScanLibraryWithoutCleanup(
|
||||
IMediaServerTelevisionRepository<TLibrary, TShow, TSeason, TEpisode, TEtag> televisionRepository,
|
||||
TConnectionParameters connectionParameters,
|
||||
@@ -227,6 +246,7 @@ public abstract class MediaServerTelevisionLibraryScanner<TConnectionParameters,
|
||||
library,
|
||||
getLocalPath,
|
||||
showEntries,
|
||||
null,
|
||||
deepScan,
|
||||
false,
|
||||
cancellationToken);
|
||||
|
||||
@@ -89,9 +89,12 @@ public class PlexMovieLibraryScanner :
|
||||
|
||||
protected override string MediaServerEtag(PlexMovie movie) => movie.Etag;
|
||||
|
||||
// #484: Plex's movie projection returns a bare PlexMovie with no catch, so a bad item throws and
|
||||
// unwinds the whole scan instead of being silently dropped — there is no failure to count here.
|
||||
protected override IAsyncEnumerable<Tuple<PlexMovie, int>> GetMovieLibraryItems(
|
||||
PlexConnectionParameters connectionParameters,
|
||||
PlexLibrary library) =>
|
||||
PlexLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_plexServerApiClient.GetMovieLibraryContents(
|
||||
library,
|
||||
connectionParameters.Connection,
|
||||
|
||||
@@ -89,9 +89,12 @@ public class PlexOtherVideoLibraryScanner :
|
||||
|
||||
protected override string MediaServerEtag(PlexOtherVideo otherVideo) => otherVideo.Etag;
|
||||
|
||||
// #484: Plex's other-video projection returns a bare PlexOtherVideo with no catch, so a bad item
|
||||
// throws and unwinds the whole scan instead of being silently dropped — nothing to count here.
|
||||
protected override IAsyncEnumerable<Tuple<PlexOtherVideo, int>> GetOtherVideoLibraryItems(
|
||||
PlexConnectionParameters connectionParameters,
|
||||
PlexLibrary library) =>
|
||||
PlexLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_plexServerApiClient.GetOtherVideoLibraryContents(
|
||||
library,
|
||||
connectionParameters.Connection,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
using System.IO.Abstractions;
|
||||
using System.IO.Abstractions;
|
||||
using System.Text.RegularExpressions;
|
||||
using ErsatzTV.Core;
|
||||
using ErsatzTV.Core.Domain;
|
||||
@@ -233,9 +233,12 @@ public partial class PlexTelevisionLibraryScanner :
|
||||
// }
|
||||
// }
|
||||
|
||||
// #484: Plex's show projection returns a bare PlexShow with no catch, so a bad item throws and
|
||||
// unwinds the whole scan instead of being silently dropped — there is no failure to count here.
|
||||
protected override IAsyncEnumerable<Tuple<PlexShow, int>> GetShowLibraryItems(
|
||||
PlexConnectionParameters connectionParameters,
|
||||
PlexLibrary library) =>
|
||||
PlexLibrary library,
|
||||
MediaServerProjectionFailureCounter projectionFailures) =>
|
||||
_plexServerApiClient.GetShowLibraryContents(
|
||||
library,
|
||||
connectionParameters.Connection,
|
||||
|
||||
@@ -58,7 +58,11 @@ public class JellyfinMusicVideoLibraryScannerTests
|
||||
MusicVideo incoming = BuildIncoming(VideoPath, artistName: "Artist 1", title: "Song 1");
|
||||
|
||||
var apiClient = Substitute.For<IJellyfinApiClient>();
|
||||
apiClient.GetMusicVideoLibraryItems(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<JellyfinLibrary>())
|
||||
apiClient.GetMusicVideoLibraryItems(
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<JellyfinLibrary>(),
|
||||
Arg.Any<MediaServerProjectionFailureCounter>())
|
||||
.Returns(OneItem(incoming));
|
||||
|
||||
var pathReplacement = Substitute.For<IJellyfinPathReplacementService>();
|
||||
@@ -672,7 +676,11 @@ public class JellyfinMusicVideoLibraryScannerTests
|
||||
private static IJellyfinApiClient FakeApi(params Func<MusicVideo>[] items)
|
||||
{
|
||||
var apiClient = Substitute.For<IJellyfinApiClient>();
|
||||
apiClient.GetMusicVideoLibraryItems(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<JellyfinLibrary>())
|
||||
apiClient.GetMusicVideoLibraryItems(
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<string>(),
|
||||
Arg.Any<JellyfinLibrary>(),
|
||||
Arg.Any<MediaServerProjectionFailureCounter>())
|
||||
.Returns(_ => Items(items));
|
||||
return apiClient;
|
||||
}
|
||||
|
||||
@@ -3777,3 +3777,75 @@ section renders the date itself, never a stale/fresh verdict.
|
||||
much of this corpus is agent-authored, distinguishing agent-inferred from human-confirmed-after-
|
||||
measuring is real signal; it just needs its own actor convention and wasn't worth coupling to this
|
||||
change.
|
||||
## 2026-07-25 — A media-server sweep also refuses when the api client silently dropped items whose projection threw; the ratio threshold is rejected (#484)
|
||||
`key: scan.projection-failure-sweep-guard` · `status: active` · `since: 2026-07-25` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** `MediaServerReconciliationGuard.ShouldFlagMissing` takes a per-enumeration projection-failure count and refuses the file-not-found sweep (logged loudly) whenever it is non-zero against a non-empty existing set; deliberate guard-clause skips (STRM, virtual, unsupported type) are explicitly **not** failures and never suppress a sweep. The missing-fraction / ratio threshold floated by #477 is **rejected**, not deferred.
|
||||
**Signals:** projection failure, silently dropped item, deferred ratio threshold rejected, STRM skip vs failure, library sweep anti-nuke follow-up · paths: `MediaServerProjectionResult`, `MediaServerProjectionFailureCounter`, `MediaServerReconciliationGuard`, `JellyfinApiClient.GetPagedLibraryItems`, `EmbyApiClient.GetPagedLibraryContents` · issues: #484, #477, #476
|
||||
**Mechanics:** `MediaServerReconciliationGuardTests` policy table · `JellyfinApiClientTests.ProjectionFailureCounter` · `MediaServerTelevisionLibraryScannerTests.Projection_Failure_Suppresses_The_Partial_Deletion_Sweep`
|
||||
|
||||
This resolves both items the `scan.zero-item-fetch-guard` record (#477) left deferred. That record's rule
|
||||
is unchanged and still in force: a zero-item fetch against a non-empty library still refuses the sweep.
|
||||
This is a **second, independent** refusal on the same guard, plus a decision not to build the first.
|
||||
|
||||
- **The two ways a projection produces nothing are NOT the same thing, and conflating them is the main
|
||||
way to get this wrong.** Each media-server api client maps every item the server returned through a
|
||||
private `ProjectTo*` and drops the ones that yield nothing. A *deliberate skip* is a guard clause at
|
||||
the top of the projection — Jellyfin `ProjectToMovie`/`ProjectToMusicVideo` (`LocationType !=
|
||||
"FileSystem"`, `.strm`), `ProjectToEpisode` (same two), `ProjectToCollectionMediaItem` (both plus an
|
||||
unmatched `item.Type`); Emby `ProjectToMovie` (no `MediaSources`), `ProjectToEpisode` (`LocationType
|
||||
== "Virtual"`), `ProjectToCollectionMediaItem` (unmatched `item.Type`). These are permanent and
|
||||
expected: a library holding one STRM file emits one on **every** scan, forever. A *failure drop* is
|
||||
the `catch (Exception ex) { LogWarning(ex, "Error projecting …"); }` that every `ProjectTo*` in all
|
||||
three clients ends with — the server DID return that item, we just could not build it, and at the
|
||||
reconcile step that is indistinguishable from a deletion. Only failure drops suppress the sweep.
|
||||
Counting deliberate skips would permanently disable reconciliation for any library containing a single
|
||||
STRM file, so stale rows would accumulate forever — a regression, not a safe default.
|
||||
`Jellyfin`/`Emby` `ProjectToShow`, `ProjectToSeason` and `ProjectToCollection` have **no** guard
|
||||
clause at all, so for them every drop is a failure.
|
||||
- **Rejected — the missing-fraction / ratio threshold.** It is a two-sided heuristic. Set low it
|
||||
silently suppresses legitimate bulk deletions (stale rows persist invisibly, and the user's only
|
||||
signal is a warning nobody reads); set high it misses the partial-fetch case it exists for. Choosing
|
||||
the number needs per-install telemetry we do not collect, and no default is defensible for both a
|
||||
20-item library and a 20,000-item one. Decisively: the failure it approximates is **exactly
|
||||
observable** by the mechanism above, so accepting an unbounded false-suppression risk to approximate
|
||||
it is a bad trade. A genuine bulk deletion produces **zero** projection failures (and a
|
||||
correspondingly smaller server-reported total), so the deterministic signal has no false positives on
|
||||
the very case the ratio threshold would have broken.
|
||||
- **A three-state projection result, not a wider tuple.** The seam is deliberately narrow.
|
||||
`IAsyncEnumerable<Tuple<TItem, int>>` appears in ~90 signatures across `ErsatzTV.Core/Interfaces/
|
||||
{Jellyfin,Emby,Plex}`, the three api clients and ~15 scanners; widening it for this would be a
|
||||
disproportionate, risky refactor. Instead the **private** mapper contract inside each client changed
|
||||
from `Option<TItem>` to `MediaServerProjectionResult<TItem>` (projected / skipped / failed) — private,
|
||||
so zero public churn — and the paged helper counts `IsFailure` in exactly one place per client. The
|
||||
scanner reads the count through an **optional trailing parameter** on only the library-level methods
|
||||
that actually feed a sweep (`IJellyfinApiClient.GetMovieLibraryItems` /
|
||||
`GetMusicVideoLibraryItems` / `GetShowLibraryItemsWithoutPeople`, `IEmbyApiClient.GetMovieLibraryItems`
|
||||
/ `GetShowLibraryItems`), so every other call site is untouched. A library that cannot be resolved
|
||||
(`Option<TLibrary>.None` inside a mapper) is treated as a **failure**, not a skip — fail closed.
|
||||
- **The counter is per-enumeration state, never ambient.** `MediaServerProjectionFailureCounter` is
|
||||
created by the scanner that owns the sweep, handed to the single enumeration whose result it will
|
||||
diff, and read only after that enumeration completes. It is never a field on an api client (those are
|
||||
long-lived and shared) and never static, so concurrent scans of different libraries cannot leak
|
||||
failures into each other's sweep decision; increments are interlocked so a paginator that ever fans
|
||||
out stays correct.
|
||||
- **Wired into the four real sweep call sites**, matching #477's scope exactly: the three library-level
|
||||
base scanners (`MediaServerMovieLibraryScanner`, `MediaServerTelevisionLibraryScanner`,
|
||||
`MediaServerOtherVideoLibraryScanner`) and `JellyfinMusicVideoLibraryScanner`. The nested TV
|
||||
season/episode sweeps stay unguarded for the reasons in `scan.zero-item-fetch-guard`, and
|
||||
`ScanLibraryWithoutCleanup` (single-show rescan) passes no counter because it runs no sweep.
|
||||
- **Plex reports zero failures because it has none.** `PlexServerApiClient`'s movie/show/other-video
|
||||
projections return a bare entity with no `Option` and no `catch`, so a bad item throws and unwinds the
|
||||
whole scan — the pre-existing "protection by accident of control flow" that #477 named. The Plex
|
||||
scanner overrides therefore accept the counter and ignore it; that is honest, not a gap. If those
|
||||
projections ever grow a swallowing `catch`, they must report into the counter at the same time.
|
||||
- **Tests.** `MediaServerReconciliationGuardTests` pins the extended policy table: a failure count skips
|
||||
and warns; a short incoming set with **zero** failures (the deliberate-skip case) still sweeps; a
|
||||
1-of-500 bulk deletion with zero failures still sweeps; failures against an empty local set stay a
|
||||
silent no-op; the failure branch is reported when both refusals apply. `JellyfinApiClientTests
|
||||
.ProjectionFailureCounter` drives the real client over real JSON and proves the split at the layer
|
||||
that makes it — a STRM plus a virtual item leave the counter at 0 while the healthy item still flows,
|
||||
a projection made to throw yields no items and a count of 1, and two enumerations on one client
|
||||
instance keep separate counts. `MediaServerTelevisionLibraryScannerTests
|
||||
.Projection_Failure_Suppresses_The_Partial_Deletion_Sweep` is the integration pair to #476's cascade
|
||||
test: identical arrangement, only the counter differs, so the cascade test is the positive control
|
||||
that keeps the new test from passing vacuously.
|
||||
|
||||
@@ -114,6 +114,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `scan.getoraddfolder-db-lookup` | `ILibraryRepository.GetOrAddFolder` resolves the existing folder via a DB query on `(LibraryPathId, Path)`, not the caller's `LibraryPath.LibraryFolders` in-memory navigation, since that navigation is only eager-loaded on the local scan path and is null on remote (Jellyfin) callers. | 2026-07-20 | [link](../decisions.md#2026-07-20--ilibraryrepositorygetoraddfolder-resolves-the-folder-from-the-db-not-the-callers-librarypathlibraryfolders-navigation-488) |
|
||||
| `scan.jellyfin-mixed-content-library` | A Jellyfin library whose collection type is `mixed` (or absent) maps to one ErsatzTV library of `LibraryMediaKind.Mixed`, scanned by running the movie/television/music-video scanners in sequence against that single library — a library is a place, not a media kind. | 2026-07-20 | [link](../decisions.md#2026-07-20-489--jellyfin-mixed-content-libraries-map-to-one-library-holding-many-kinds) |
|
||||
| `scan.musicvideo-reconciliation` | `JellyfinMusicVideoLibraryScanner` reconciles removed music videos by a library-scoped local-path diff plus hard delete (`TrashMissingMusicVideos`), not the server-itemId soft-trash pattern the other media-server scanners use, because music videos carry no server identity. | 2026-07-20 | [link](../decisions.md#2026-07-20--jellyfinmusicvideolibraryscanner-reconciles-by-library-scoped-path-diff--hard-delete-not-server-itemid-soft-trash-494) |
|
||||
| `scan.projection-failure-sweep-guard` | `MediaServerReconciliationGuard.ShouldFlagMissing` takes a per-enumeration projection-failure count and refuses the file-not-found sweep (logged loudly) whenever it is non-zero against a non-empty existing set; deliberate guard-clause skips (STRM, virtual, unsupported type) are explicitly **not** failures and never suppress a sweep. The missing-fraction / ratio threshold floated by #477 is **rejected**, not deferred. | 2026-07-25 | [link](../decisions.md#2026-07-25--a-media-server-sweep-also-refuses-when-the-api-client-silently-dropped-items-whose-projection-threw-the-ratio-threshold-is-rejected-484) |
|
||||
| `scan.zero-item-fetch-guard` | A media-server library sweep refuses to flag missing items when a successful fetch returns zero incoming items against a non-empty existing set (`MediaServerReconciliationGuard.ShouldFlagMissing`), rather than treating an ambiguous empty result as a full-library deletion. | 2026-07-19 | [link](../decisions.md#2026-07-19--a-media-server-library-sweep-refuses-to-flag-when-a-successful-fetch-returns-zero-items-rather-than-nuking-the-whole-library-477) |
|
||||
| `sched.auto-tune-foundation` | Auto-tune preview enumeration uses EF distinct+count queries for exact counts, while each created channel is persisted as a live SmartCollection; coexistence with existing channels/numbers is additive-only, never mutating. | 2026-07-16 | [link](../decisions.md#2026-07-16--auto-tuning-enumerates-via-ef-persists-via-smartcollection-additive-coexistence-69) |
|
||||
| `sched.autotune-detailpanel-members` | The Auto-Tune DetailPanel's per-channel content-source list is a live `ISearchIndex.Search` roll-up through the server-owned `AutoTuneAxisMap.GenerateQuery`, not an EF distinct+count query, so the preview matches exactly what the built channel's SmartCollection will contain. | 2026-07-17 | [link](../decisions.md#2026-07-17--auto-tune-detailpanel-member-list--live-search-index-roll-up-not-ef-enumeration-384) |
|
||||
|
||||
Reference in New Issue
Block a user