ersatztv/docs/ci-cd.md

CI/CD for the ErsatzTV Fork

The fork builds its own Docker image via Gitea Actions on the homelab and pushes to the Gitea container registry. Runner + registry were provisioned in server-management#172; the build pipeline is ersatztv#4; test/prod containers are server-management#481.

Versioning & releases

The fork inherits upstream ErsatzTV's scheme: vYY.<release-seq>.<patch> (lightweight, v-prefixed git tags).

• YY — two-digit year.
• <release-seq> — a sequential release counter within the year, reset at each year boundary. It is not the calendar month. (Evidence: v25.2.0 shipped in June 2025, v25.5.0 in Sep, v26.3.0 in Feb 2026 — minors don't track months; and v25.9.0 → v26.1.0 shows the year-reset.)
• <patch> — a small follow-up/hotfix on the same release line (e.g. v26.1.0 → v26.1.1, days later).

Upstream's final release was v26.3.0 (archived). Our line continues from there:

Tag	Meaning
v26.3.1	Upstream 26.3.0 rebuilt on our infra (Gitea CI/registry, fork ffmpeg base) — no application changes. A patch bump, because nothing functional changed.
v26.4.0	First fork release carrying application changes. Later 2026 releases continue 26.5.0, 26.6.0, …; a new year resets to 27.1.0.
v26.7.0	Blazor-removal release: ChicoryTV became the only UI.
v26.8.0	Secured/versioned ChicoryTV SPA + REST API go-live release (#335).

Before cutting a release — consolidate docs/decisions.md. The log is append-only between releases (ersatztv#303 H9), so a release boundary is where superseded entries get pruned/merged and the Index refreshed. Fold and drop any entry marked > **Superseded …**, then commit with [decisions-edit] in the message (the append-only guard blocks history edits otherwise). Mark-and-keep during the arc, consolidate here — or sooner if the decisions-guard job's 1800-line consolidation-floor warning fires.

Cutting a release: keep build and promotion as two explicit phases (#335):

1. Confirm main CI is green; run the full local gate plus dotnet list package --vulnerable --include-transitive; then push a vYY.N.P tag on that exact main commit.
2. Wait for tag CI to build :prod + the immutable :<version> + :<sha> images. Run scripts/security-scan.sh on bumblebee against the immutable :<version> image, not a moving tag, and triage every ZAP/semgrep finding.
3. Only after the candidate passes, manually DeployStack media-servers and observe its pre-deploy output. Prod's compose deliberately follows floating :prod (Timothy's 2026-07-11 decision), so no CI push or pin bump is needed. Global Auto Update (auto_update: true on media-servers) is the daily fallback, not the pre-scan promotion mechanism; do not cut a tag close enough to its 03:00 run that an unscanned digest could be promoted first.

server-management#585 source-confirmed that Global Auto Update invokes the same DeployStack execution as a manual promotion, and extended the #553 pre-deploy hook to detect a floating-tag digest change. Either path now takes the fail-closed prod backup; server-management#589 then wired migration-smoke.sh into that hook, against the exact candidate and the backup it just made. A backup, fetch, or migration-smoke failure aborts before the live container is recreated. See homelab-docs/Docker/ErsatzTV.md for the operational evidence and rollback procedure.

Gotcha: never put a [skip ci] token in a commit you intend to tag — Gitea reads skip-ci from the tagged commit and will suppress the release build. (Also, workflow_dispatch on a tag ref isn't supported on this Gitea version, so the tag push must do the triggering.) Release commits, and anything you'll tag, must not contain skip-ci.

Also avoid firing several pushes back-to-back (e.g. a [skip ci] commit, then main, then a tag, all within ~1s). Observed once on this Gitea instance: the later events were silently dropped — no ActionRun records created at all, even though the runner was online and the workflow active. Pushing again, spaced out, created the runs normally. If a push/tag doesn't produce a run, re-push (or push an empty commit) rather than assuming the runner is broken.

The workflow: .gitea/workflows/docker-build.yml

Single workflow. Gating jobs test + migrations run in parallel and gate build; a non-blocking docs-reminder job runs on PRs only (see below). Prod deploy is not a CI job — it's Komodo Global Auto Update off the :prod tag (see "Cutting a release").

Triggers & tags

Trigger	test job	build job	Image tags pushed
pull_request	✅	— (skipped)	none
push to main	✅	✅	:latest + :<short-sha>
push tag v*	✅	✅	:prod + :<version> + :<short-sha>
workflow_dispatch	✅	✅	only if ref is main/v*, else build-only (no push)

:latest is the test/dev channel (every main commit). Prod's compose follows the floating :prod tag (reverted from the 2026-07-07 version pin on 2026-07-11) — never :latest. Both :prod and :<version> are produced by pushing a v* tag; prod tracks :prod and is redeployed by Komodo Global Auto Update (see "Cutting a release"). The immutable :<version> tags remain for reproducible rollback (docker run …:26.6.0).

Concurrency is scoped per event+ref (group: ersatztv-build-${{ github.event_name }}-${{ github.ref }}, cancel-in-progress for PRs): PR runs parallelize across PRs, a new sync auto-cancels its superseded run, and image builds still serialize within their own ref. Do NOT push main and a v* tag simultaneously — those are separate groups but share the :buildcache tag and the smoke container name; tag only after the main build is green. (History: originally one global group serializing ALL runs for the single runner — with three runners that starved the queue; changed 2026-07-11, server-management#574.)

Three runners serve the fork (server-management#570/#574): ci-runner (VM 127 pve4, ubuntu-latest, 2 slots), bumblebee-runner (bumblebee, ubuntu-latest, 2 slots, jobs capped --cpus=4 --memory=10g so CI can't starve prod media playback), and small-runner (bumblebee, label small, 4 slots) — the small-jobs lane. The build and docs-reminder jobs use runs-on: small: Gitea dispatches a job as a runner task even when its if skips it, and those skip-tasks used to wait behind long builds (observed 31 min) stalling every PR run.

test job

dotnet restore → strip the Scanner project ref (sed -i '/Scanner/d', matching the Docker build) → dotnet build -c Release → dotnet test -c Release --no-build. Gates the image build.

• Code coverage (ersatztv#15): dotnet test runs with --collect:"XPlat Code Coverage" --settings coverlet.runsettings --results-directory ./coverage, so coverlet.collector (referenced by every *.Tests project) emits a Cobertura report per project. A follow-up Coverage summary step merges them with ReportGenerator (TextSummary to the log, MarkdownSummaryGithub to the job step summary). No floor is enforced yet ("decide on a floor later" — #15); the step is continue-on-error: true, so a missing report or a transient tool install never blocks a build.
  • coverlet.runsettings excludes generated EF migration code (**/Migrations/*.cs, ~2.59M generated lines vs ~200k authored). Instrumenting it OOM-killed the shared test job (exit 137); excluding it cuts the instrumented surface ~126× (2.5M→20k coverable lines in the whole-solution Architecture.Tests process) and makes the percentage reflect authored code.
• Shallow checkout: fetch-depth: 1 (ersatztv#190) — this job never runs git describe/git log, only build needs full history/tags for version computation, so test and migrations both check out shallow. build's checkout stays fetch-depth: 0.
• NuGet package cache: both test and migrations cache ~/.nuget/packages via actions/cache@v4, keyed on hashFiles('Directory.Packages.props', 'global.json') with a restore-keys OS-level fallback (ersatztv#190). Avoids a from-scratch dotnet restore on every run; the key only changes when the central package manifest or SDK pin changes.

build job

1. Compute INFO_VERSION (git describe + short sha on main; tag version on v*).
2. docker/setup-buildx-action with buildkitd-config-inline setting http = true for 192.168.1.95:3000 — BuildKit does not inherit the host daemon's insecure-registries, so without this, cache/base-image/push over the HTTP registry fails (http: server gave HTTP response to HTTPS client).
3. docker/login-action with repo secrets REGISTRY_USER / REGISTRY_PASSWORD.
4. docker/build-push-action@v6: amd64-only, docker/Dockerfile, INFO_VERSION build-arg, registry layer cache (type=registry,ref=…:buildcache, cache-to … ignore-error=true).
5. Smoke + IPTV E2E test: pull the just-pushed :<sha>, run it, poll for HTTP readiness (docker exec … python3 → http://localhost:8409/), then assert the real Jellyfin-facing surfaces on the freshly built image (ersatztv#16): /iptv/channels.m3u returns 2xx containing #EXTM3U, and /iptv/xmltv.xml returns 2xx containing a <tv root. xmltv.xml needs channels.xml (written by the scheduler a few seconds after boot), so each endpoint is polled with a deadline. Unique container name + trap … EXIT cleanup; dumps container logs on failure. Catches routing / base-URL (#1) / migration regressions that leave the app "up" but serving broken output.

functional-e2e job (advisory; PR + main)

Boots the app from source and drives the manual live-E2E curl flows sessions have historically re-run by hand, turning them into a CI regression net (ersatztv#299). It is the automatable half of docs/e2e-local.md; the two scripts it chains run identically locally and in CI:

1. npm ci + npm run build (SPA), dotnet build ErsatzTV.sln -c Release, ensure ffmpeg is on PATH.
2. ETV_BUILD_CONFIG=Release scripts/e2e-local.sh <fresh-config> — copies wwwroot, launches dotnet ErsatzTV.dll in the background (logging to a file so the launch step returns once the app is ready), prints PID/CONFIG_DIR.
3. scripts/e2e-functional.sh http://localhost:8409 <config> — asserts, all curl-only and deterministic (no seeded media, no browser): the legacy→SPA redirect sweep (+ the /api, /artwork never-redirect exemption), the auth/CSRF/security-stamp flow (setup-claim → read-gate 401/200 → re-claim 409 → CSRF 403 → login 401/200 → logout 403/204 → post-logout stamp-revocation 401), the library-scan status contract (404 unknown / 202 queued / scan-status 200), and the If-Match/412 round-trip on rerun-collections. A trap kills the instance on step exit.

Advisory, by design (the issue's "keep it a separate job so a functional-E2E flake can't block the unit-test gate"): it is not a needs: of build and not (yet) a required check, so a flake blocks nothing. Promote it to a required check / build dependency once it's proven reliable — the same staged rollout the migrations job used. SQLite is the default provider, so unlike migrations it needs no DB service container. Runs on PRs and on main (regression net); skipped for v* tag builds. Out of scope for this first cut (need the scanner subprocess + seeded media, or a browser, to be deterministic — tracked as ersatztv#299 follow-ups): the racy 409 "already-scanning" re-trigger, the playout-build lock 409, and the genuinely UI-interactive Playwright flows.

docs-reminder job (non-blocking, PR-only)

A lightweight nudge that enforces the CLAUDE.md "docs-update is part of done" rule for the one case that's easy to forget and easy to detect: a PR that touches a SPA screen (web/src/screens/*.tsx) or ErsatzTV/LegacyUiRedirects.cs but does not update docs/blazor-route-parity.md. It diffs the PR against its base branch and emits a ::warning:: annotation (never fails the build — it's a reminder, not a gate; prose-doc gates get gamed with token edits). Deliberately has no setup-dotnet/setup-node (and thus no actions/cache), so it can't hit the cache-save hangs seen on the VM-127 runner (server-management#570). It does not cover the remaining doc obligations in the CLAUDE.md table (domain-model, spa-conventions) — those stay on the author. (The API contract is mechanized by the blocking api-docs job, and docs/decisions.md by the blocking decisions-guard job below.)

decisions-guard job (blocking, PR-only)

Enforces the docs/decisions.md append-only convention (ersatztv#303 H9): fails a PR whose merge-base diff deletes or modifies any existing line of that file, unless a commit in the range carries the [decisions-edit] token (for a factual fix or a documented supersession — see the decisions.md header). Pure insertions (a normal new entry: TOC line + appended block) pass. The job also emits a non-blocking consolidation nudge once the file exceeds 1800 lines (the read-cost floor — one default agent Read caps at 2000 lines), so append-only can't outgrow what agents read. It runs the same .claude/hooks/decisions-guard.sh the Husky commit-msg hook uses, so the detection logic is shared and can't drift. Granularity differs, deliberately: Husky checks each commit (staged mode, that commit's own message must carry the token); CI checks the PR-wide net diff (range mode, accepts the token in any commit of the range). The local hook is therefore the stricter, primary gate; CI is the backstop for direct pushes or bypassed hooks. Like docs-reminder, it's a seconds-long git diff with no dotnet/node setup (runs-on: small).

Dockerfile notes (docker/Dockerfile)

• Base image: 192.168.1.95:3000/timothy/ersatztv-ffmpeg:8.1.2 (our Gitea fork of the archived ghcr.io/ersatztv/ersatztv-ffmpeg). FFmpeg 8 base image work landed in ersatztv-ffmpeg#4; app-side compatibility work landed in ersatztv#9.
• Copies Directory.Build.props, Directory.Build.targets, Directory.Packages.props, global.json, .editorconfig before dotnet restore so the image build uses the same MSBuild config, central package versions, SDK pin, and analyzer severities as local/CI builds (it previously copied only *.sln). Directory.Packages.props is required here: under Central Package Management the csproj carry no inline versions, so the image's restore fails (NU1015) without the central manifest.
• amd64-only (the runner/build host is x86_64). No arm32/arm64, no DMG/exe artifacts, no GHCR/DockerHub.
• openapi-generator jar layer ordering (ersatztv#190): the wget for the openapi-generator-cli jar runs before the COPY of ErsatzTV/wwwroot/openapi/, so the ~30MB download layer is cached independently of the openapi spec. Previously the jar was downloaded after that COPY, so any PR touching the spec (e.g. v1.json) busted the download layer too and re-fetched the jar on every such change. Codegen itself still runs after the spec COPY, since it needs both the jar and the spec files.

Dependency management (Central Package Management + scans)

Central Package Management (CPM) — package versions live in a single repo-root Directory.Packages.props (ManagePackageVersionsCentrally=true); the per-project csproj reference packages by name only (no Version=). One source of truth, atomic one-line bumps, and cross-project version drift is structurally impossible. To add or change a dependency, edit the <PackageVersion> entry centrally — never put a Version= back on a <PackageReference> (that trips NU1008). The Docker build must copy this file before restore (see Dockerfile notes). The .mcp/ vendored tool (gitignored, not in the solution) keeps inline versions via a local-only .mcp/Directory.Packages.props opt-out (ManagePackageVersionsCentrally=false). (ersatztv#14)

NuGet audit — .NET 10 runs NuGet audit on restore. Several projects set TreatWarningsAsErrors=true, so vulnerable transitive packages failed the build. Directory.Build.props demotes low/moderate/high advisories (NU1901-1903) to warnings and promotes NU1904 (critical) to an error in every project via WarningsAsErrors. The advisories that prompted this were resolved in ersatztv#8 (NCalcSync→6.x; SQLitePCLRaw bundle 3.x) and ersatztv#314 (Microsoft.OpenApi 2.0.0→2.7.5, GHSA-v5pm-xwqc-g5wc High — direct-pinned in ErsatzTV.csproj over the 2.0.0 that Microsoft.AspNetCore.OpenApi + Scalar.AspNetCore pull transitively; the SQLitePCLRaw override pattern; regenerates the OpenAPI doc byte-identically). The NU1901-1903 demotion is kept by design: criticals (NU1904) still hard-block, while low/moderate/high advisories surface as warnings + via the weekly scan and Renovate security PRs, rather than breaking unrelated PRs the moment a new transitive advisory drops.

Scheduled vulnerability scan — .gitea/workflows/dependency-scan.yml runs weekly (cron 0 6 * * 1) + on workflow_dispatch: dotnet list package --vulnerable --include-transitive over the full solution (incl. Scanner, which the image build strips). dotnet list exits 0 even with findings, so the step (bash -euo pipefail) greps for the "has the following vulnerable packages" marker and fails the run if present. Detection only — it surfaces advisories on a schedule, a Gitea-native stand-in for Dependabot; it does not open update PRs (that's Renovate — server-management#484). Gitea registers schedule triggers only from the default branch, so the cron starts after merge to main; use workflow_dispatch to run on demand. It went green once ersatztv#8 cleared the NCalcSync/SQLitePCLRaw advisories — a red run now means a new advisory has appeared. (ersatztv#14, ersatztv#8)

Renovate (automated update PRs) — .gitea/workflows/renovate.yml runs self-hosted Renovate weekly (cron 0 3 * * 1) + on workflow_dispatch, as a renovate/renovate:43 container job on the shared act_runner. This is the proposing layer the scan above deliberately omits: it opens grouped dependency-update PRs and OSV-driven vulnerability-fix PRs against main, and maintains a Dependency Dashboard issue listing the full backlog. Config is the repo-root renovate.json — managers nuget (via CPM), github-actions, and dockerfile (scoped to the built docker/Dockerfile; it reads the HTTP-only Gitea registry for the ersatztv-ffmpeg base via a RENOVATE_HOST_RULES host rule — insecureRegistry + registry read creds, set in the workflow env, not the committed config). The docker-compose manager is unused (repo compose files are build:-only). Auth: a dedicated renovate Gitea bot (Write collaborator) via repo Actions secrets RENOVATE_TOKEN (bot PAT) + GH_COM_TOKEN (no-scope github.com PAT for changelogs — named GH_, not GITHUB_, a prefix Gitea reserves). Patch bumps to test/dev-only packages (NUnit*, NSubstitute, Shouldly, coverlet, Microsoft.NET.Test.Sdk, Testably.Abstractions*, threading analyzer) auto-merge once the Build & test (.NET) check passes — branch protection on main requires that context; everything else is manual review (ersatztv is prod-bearing). Range-pinned packages (e.g. EF Core [9.0.x,10)) are respected — no v10 jump. PR volume is throttled (prConcurrentLimit 5 + config:recommended's prHourlyLimit 2); tick a dashboard checkbox or raise the limits to drain faster. workflow_dispatch defaults to a safe dry run. Cross-repo rollout tracked in server-management#484. (server-management#484)

Security scanning — black-box DAST + SAST (scripts/security-scan.sh, ersatztv#314)

Every other security check we run is in-ecosystem / white-box — SonarAnalyzer, NetArchTest, the adversarial fork + Codex review passes, the api-docs/format/decisions CI gates, dotnet list package --vulnerable — so they share our blind spots. scripts/security-scan.sh is the out-of-ecosystem, black-box complement and a #197 exit criterion (HARD GATE before remote exposure): it drives the running product from outside our C#/review stack.

• What it does. Boots a throwaway container from the image under test (fresh empty config volume; never the deployed prod/test container — the authenticated active scan sends attack payloads to write endpoints), reads the generated machine key, and runs an authenticated OWASP ZAP API scan (zap-api-scan.py) that imports the static /openapi/v1.json so it exercises every declared /api/v1 operation, injecting X-Api-Key on every request via a ZAP replacer rule so it reaches the [RequiresAuthentication] + RequireKeyForReads surface (not just the /app shell an unauthenticated spider sees). Then a semgrep SAST cross-check (p/security-audit + p/secrets + p/csharp). The container is torn down on exit.
• Where/when. Runs on the docker host (bumblebee — the Mac has no docker), like migration-smoke.sh: scripts/security-scan.sh [IMAGE] [PORT] (defaults …:latest / 8411). It is a manual release-gate, deliberately not a per-PR CI job — it needs docker + a booted image, takes several minutes, and is noisy (expect to tune, not take raw). The continuous layer is the per-PR white-box gates + the weekly dependency-scan; this is the per-release black-box pass. Re-run it each release and before any change to the exposure posture.
• Triage. ZAP exits non-zero on any FAIL-level alert; the tooling is noisy, so triage each finding false-positive vs real. Real, in-scope, go-live-blocking findings get fixed (e.g. the security headers from the #319 baseline; the Microsoft.OpenApi pin above); LAN-expected noise (Private-IP disclosure) is revisited only for genuine remote exposure. nuclei (template-based CVE fingerprinting) is an optional third pass — deferred while its template fetch is blocked in the runner env (pre-seed a template volume to add it); ZAP covers the DAST baseline and semgrep the SAST, so it is not on the critical path.

Static analysis & formatting

Analyzers — Directory.Build.props enables the SDK analyzers at latest-All and turns on Microsoft.VisualStudio.Threading.Analyzers for every project. Directory.Build.targets also references Roslynator, SonarAnalyzer.CSharp, Meziantou.Analyzer, and AsyncFixer repo-wide (versions central via CPM; guarded on ManagePackageVersionsCentrally so the gitignored .mcp tool isn't pulled in). They are introduced incrementally (ersatztv#15). eng/analyzers/sdk-all-suggestion.globalconfig enumerates the .NET 10 SDK All inventory at suggestion; this exact-ID baseline is necessary because the SDK's generated latest-All severities outrank .editorconfig bulk settings. .editorconfig keeps the threading and curated-pack baselines at suggestion. Diagnostics remain visible to IDEs and dotnet format analyzers, but do not create a wall of failures (a direct latest-All trial activated 455 existing errors in the TWAE projects).

Promotion is the enforcement — set an reviewed rule to warning in .editorconfig and append its ID to the central WarningsAsErrors list in Directory.Build.props. The explicit list makes the rule block in every project, including test projects that do not otherwise use TWAE. On a major SDK upgrade, regenerate the checked-in SDK baseline from analysislevel_<major>_all.globalconfig, preserve SDK none entries, and review newly introduced rules before accepting the snapshot.

Promoted rules are recorded here so the blocking subset stays intentional and reviewable:

• Sonar S3981 — warning + WarningsAsErrors (ersatztv#15): rejects collection-count comparisons that are constant regardless of collection size. Its first finding exposed Workers.Count >= 0, which permanently classified scheduled memory releases as busy and skipped the intended aggressive idle collection.

• StyleCop.Analyzers is intentionally excluded: its latest stable (1.1.118) crashes (AD0001) on C# record declarations, and its rules overlap the existing .editorconfig/Roslynator. Revisit via the record-compatible 1.2.0-beta only if specifically wanted.

• The former Blazor .razor caveat is retired: Blazor removal deleted the Razor sources and their temporary Sonar NoWarn list. The .razor/.cshtml suggestion scopes remain in .editorconfig only as a defensive default if server-rendered view code is ever reintroduced.

Formatting — the inherited tree still contains legacy UTF-8 BOM/whitespace debt, so the standing policy is format as you touch, not a mass rewrite (ersatztv#311). The Husky pre-commit hook runs dotnet format --verify-no-changes for staged C# files, and the blocking format CI job repeats that check for C# files changed by the PR. Untouched legacy files remain outside the gate; .gitattributes pins line endings. A one-time full-tree normalization remains a separate, unmade decision.

Migration integrity (EF Core, both providers)

TvContext (ErsatzTV.Infrastructure/Data/TvContext.cs) has two migration sets — one per provider project: ErsatzTV.Infrastructure.Sqlite/Migrations and ErsatzTV.Infrastructure.MySql/Migrations, each with its own TvContextModelSnapshot. A model change needs a migration in BOTH. Add them with scripts/add-migration.sh <Name> (runs the EF CLI for each provider). The EF CLI pattern (provider selected by the post--- arg, which Startup reads as the provider config key):

dotnet ef <cmd> --context TvContext --startup-project ErsatzTV \
  --project ErsatzTV.Infrastructure.{Sqlite|MySql} -- --provider {Sqlite|MySql}

The migrations job in docker-build.yml runs on every push/PR and, for each provider:

1. dotnet ef migrations has-pending-model-changes — fails if an entity changed without a matching migration (model drift), so a forgotten migration can't merge.
2. dotnet ef database update against a fresh empty DB — applies all migrations in order and fails on any broken/un-orderable one.

• SQLite (the prod provider) uses a throwaway file (ETV_CONFIG_FOLDER=$(mktemp -d)); no service needed. Validated: 787 migrations → 139 tables.
• MySql uses ServerVersion.AutoDetect, which connects at config time, so the job needs a reachable server — provided by a services: mysql:8.4 container (the act_runner uses Docker execution with an auto-created per-job network — service reachable as mysql:3306 (the old bumblebee runner pinned network downloadswarm; relocated in server-management#570)). Connection string via MySql__ConnectionString (→ config key MySql:ConnectionString). Validated: 305 migrations → 137 tables. It's an independent gate (not yet a needs: of the image build) so the new MySql-service dependency can't block image builds until it's proven; promote it to a required check once stable.

Caveat — non-transactional operations: some migrations (e.g. SQLite PRAGMA foreign_keys) run outside a transaction and warn at startup; they can't be rolled back mid-migration, so review such migrations carefully (this is part of what motivated the apply-to-fresh check before the prod cutover, server-management#481).

Resilience — the MySql apply is retried (concurrent-runner contention, not a model bug): both runners (ci-runner VM 127 + bumblebee-runner) serve ubuntu-latest, and when two migration jobs land on the same host at once (common when several PRs push together), each spins its own mysql:8.4 service container and they starve each other — producing intermittent Command Timeout expired or mid-replay MySqlEndOfStreamException (dropped connection) on the MySql apply-to-fresh-DB step. This is pure infra flakiness — has-pending-model-changes (the actual model check) still passes, and the same commit passes on a quieter host. The job hardens against it two ways: the connection string sets DefaultCommandTimeout=300 (up from MySqlConnector's 30s default), and the apply is wrapped in a 3× retry that resumes from __EFMigrationsHistory (EF commits each migration in its own transaction, so an interrupted one rolls back and the retry continues). A real migration failure fails deterministically on every attempt, so the retry never masks it. If a run still flakes past the retry, re-trigger (Gitea has no rerun API on this version — push, or the run drains); don't treat a lone MySql-apply red as a code problem without checking the failure mode.

Migration-on-prod-copy smoke — release path (scripts/migration-smoke.sh, ersatztv#315)

The migrations job above only proves a migration is well-formed against a fresh, empty DB. It can't prove it applies cleanly to the accumulated prod SQLite — real row volume, historical values, and the post-migration data steps ErsatzTV runs on startup: DatabaseMigratorService (a BackgroundService) applies pending migrations, then DbInitializer.Initialize + PopulatePathHashes (an UPDATE over the real MediaFile table). A migration green on a fresh DB can still fail or corrupt on prod, and today you'd only find out mid-deploy after the container recreates.

scripts/migration-smoke.sh rehearses it on a throwaway copy of the latest prod backup — it never touches the live DB:

scripts/migration-smoke.sh --image <ref-about-to-be-promoted> [--db <backup.sqlite3>] [--timeout 180]

It copies the backup into a temp config dir, boots the new image against it (ETV_CONFIG_FOLDER), and gates PASS on the Done applying database migrations log line — not merely on HTTP readiness, since the migrator runs concurrently with Kestrel, so the web server can serve before/while migrations run. FAIL = the container exits before finishing, a migration exception appears in the logs, migrations don't finish within --timeout, or the app won't serve /iptv/channels.m3u afterwards. The smoke container, the DB copy, and the temp dir are always torn down on exit (the ErsatzTV image runs as root, so cleanup deletes its root-owned config files from inside a throwaway root container — otherwise each run would leak the multi-hundred-MB copy). Exit 0 = clean, 1 = migration/boot failure, 2 = usage error.

• --db default: the newest ersatztv.sqlite3 under $ETV_BACKUP_DIR (default ~/downloadswarm/ersatztv-backups — where the host-side pre-deploy backup hook writes timestamped snapshots). Pass --image = the version tag about to be promoted.
• Where it runs: it's meant to run on the docker host as a Komodo pre-deploy step (which already produces the backup — see "Cutting a release" and the #553 pre-deploy backup caveat), so a bad migration aborts the promote before the live container recreates. Wiring it into that hook is a server-management concern (cross-repo — this repo owns the script + docs, server-management owns the Komodo hook). Until wired, run it by hand before cutting a migration-bearing release.
• Validated live 2026-07-12: :latest against a copy of the 283 MB prod backup → migrations applied cleanly, app booted and served, temp dir removed.

Pre-commit hooks (web/)

The repo uses husky git hooks (installed via web/'s lint-staged + npm) to catch lint/format/type/API-drift errors locally, before they reach CI. Because the git root and the npm project dir differ (monorepo: no root package.json, the JS/TS project lives entirely in web/), the wiring is:

• husky + lint-staged are devDependencies of web/package.json (not a root package — there isn't one).
• The committed hook scripts live at the repo root: .husky/pre-commit, .husky/pre-push, .husky/commit-msg.
• web/package.json's prepare script (cd .. && husky) runs on npm install inside web/ and points git at the repo-root .husky dir (git config core.hooksPath .husky/_ — the _ subdir is husky's generated internal dir, gitignored via its own .husky/_/.gitignore; only the hook scripts themselves are committed). This works because npm keeps web/node_modules/.bin on PATH for the prepare script even after it cd ..s to the repo root (which husky's init requires — it hard-checks for .git in the current directory).

The four hooks:

1. pre-commit — (a) cd web && npx lint-staged: runs eslint --fix on staged web/src/**/*.{ts,tsx} files, then a project-wide npm run typecheck (tsc -b isn't file-scoped, so it runs the full check, but only when a .ts/.tsx file is staged); (b) back at the repo root, if any *.cs files are staged, dotnet format ErsatzTV.sln --verify-no-changes --include <staged .cs> — a formatting violation blocks the commit. The .cs step is skipped entirely when no .cs is staged, so web-only commits don't pay the sln-load cost; when it does run it's scoped to the staged files (~6-7s wall in practice, dominated by the workspace load); (c) H3 (ersatztv#303) — refuses a staged root-level *.png (git diff --cached --name-only | grep -E '^[^/]+\.png$'), belt-and-suspenders with the .gitignore screenshot rule so a forced git add -f still can't land a review/debug screenshot at the repo root. Nested *.png (real assets) pass.
2. pre-push — CI-parity gate: cd web && npm run check:api && npm run lint && npm run typecheck && npm run build. check:api guards generated-OpenAPI drift (ErsatzTV/wwwroot/openapi/v1.json → web/src/api/generated/v1.d.ts); the full lint/typecheck/build catch a staged change that breaks an unstaged file (lint-staged only sees staged files). Any failure blocks the push.
3. commit-msg — (a) enforces the CLAUDE.md protocol: the message must carry a Co-Authored-By: trailer, else the commit is rejected (merge commits are exempt, detected via git rev-parse --verify MERGE_HEAD); (b) H9 (ersatztv#303) — runs .claude/hooks/decisions-guard.sh staged "$1", which blocks the commit if it deletes/modifies an existing line of docs/decisions.md unless the message carries the [decisions-edit] token. Append-only enforcement; see docs/decisions.md header for the supersession/consolidation rules. The same script backs the blocking decisions-guard CI job (range mode over the PR's merge-base diff) so local and CI enforcement can't drift.

• Worktree/subdir gotcha: git exports GIT_DIR (and friends) while running hooks. In a worktree or any subdir, an explicit GIT_DIR makes nested git commands mislocate the working tree — pre-push's check:api (git diff --exit-code, run from web/) then silently reports "no diff" and lets drift through. pre-push therefore unsets GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE first. (pre-commit's .cs collection uses git diff --cached, index-vs-HEAD, which needs only GIT_DIR and is unaffected.)
• Practical effect: a fresh web/ npm install (after cloning or pulling this change) installs all four hooks automatically — no separate setup step. Commits that touch only non-web/, non-.cs files skip linting/formatting (lint-staged no-ops with nothing to run, the .cs step is skipped).

Registry

Gitea Packages, HTTP-only at 192.168.1.95:3000. the ci-runner VM's Docker daemon (192.168.1.127) has it as an insecure-registry (server-management#172; runner relocated off jazz in #570). Images: 192.168.1.95:3000/timothy/ersatztv:<tag>.

Test / prod environments

Container/compose wiring lives in server-management (project boundary): test ersatztv-test on 8410 (:latest), prod ersatztv on 8409 (:prod). See server-management#481 for the full spec (registry pull on jazz, volumes, Jellyfin isolation for test, Watchtower/manual promotion).

Retired upstream workflows

The upstream .github/workflows/ (ci.yml, docker.yml, artifacts.yml, release.yml, pr.yml, issue-stale.yml) were removed — they targeted GHCR/DockerHub + Azure/Apple signing and called reusable workflows at dead ersatztv/ersatztv@main paths, and ran as noise (incl. a daily stale-issue cron) on the Gitea runner. Upstream is archived, so there are no future merges to preserve them for. The dead .github/dependabot.yml and FUNDING.yml (upstream-pointed) were also removed.

Known follow-ups

• Pin third-party actions to commit SHAs (currently floating major tags cloned from github.com at runtime) — low priority for a homelab; tracked informally.