ersatztv/docs/ci-cd.md

CI/CD for the ErsatzTV Fork

The fork builds its own Docker image via Gitea Actions on the homelab and pushes to the Gitea container registry. Runner + registry were provisioned in server-management#172; the build pipeline is ersatztv#4; test/prod containers are server-management#481.

Versioning & releases

The fork inherits upstream ErsatzTV's scheme: vYY.<release-seq>.<patch> (lightweight, v-prefixed git tags).

• YY — two-digit year.
• <release-seq> — a sequential release counter within the year, reset at each year boundary. It is not the calendar month. (Evidence: v25.2.0 shipped in June 2025, v25.5.0 in Sep, v26.3.0 in Feb 2026 — minors don't track months; and v25.9.0 → v26.1.0 shows the year-reset.)
• <patch> — a small follow-up/hotfix on the same release line (e.g. v26.1.0 → v26.1.1, days later).

Upstream's final release was v26.3.0 (archived). Our line continues from there:

Tag	Meaning
v26.3.1	Upstream 26.3.0 rebuilt on our infra (Gitea CI/registry, fork ffmpeg base) — no application changes. A patch bump, because nothing functional changed.
v26.4.0	Reserved for our first release that carries actual app changes (e.g. the #1 M3U fix). Later 2026 releases: 26.5.0, 26.6.0, …; a new year resets to 27.1.0.

Cutting a release: push a vYY.N.P tag on main → CI builds :prod + :<version> + :<sha>; server-management does the prod switch (server-management#481).

Gotcha: never put a [skip ci] token in a commit you intend to tag — Gitea reads skip-ci from the tagged commit and will suppress the release build. (Also, workflow_dispatch on a tag ref isn't supported on this Gitea version, so the tag push must do the triggering.) Release commits, and anything you'll tag, must not contain skip-ci.

Also avoid firing several pushes back-to-back (e.g. a [skip ci] commit, then main, then a tag, all within ~1s). Observed once on this Gitea instance: the later events were silently dropped — no ActionRun records created at all, even though the runner was online and the workflow active. Pushing again, spaced out, created the runs normally. If a push/tag doesn't produce a run, re-push (or push an empty commit) rather than assuming the runner is broken.

The workflow: .gitea/workflows/docker-build.yml

Single workflow. Gating jobs test + migrations run in parallel and gate build; a non-blocking docs-reminder job runs on PRs only (see below).

Triggers & tags

Trigger	test job	build job	Image tags pushed
pull_request	✅	— (skipped)	none
push to main	✅	✅	:latest + :<short-sha>
push tag v*	✅	✅	:prod + :<version> + :<short-sha>
workflow_dispatch	✅	✅	only if ref is main/v*, else build-only (no push)

:latest is the test/dev channel (every main commit). Prod pins :prod, never :latest — enforced in the prod compose (server-management#481). :prod is only produced by pushing a v* tag.

Concurrency is scoped per event+ref (group: ersatztv-build-${{ github.event_name }}-${{ github.ref }}, cancel-in-progress for PRs): PR runs parallelize across PRs, a new sync auto-cancels its superseded run, and image builds still serialize within their own ref. Do NOT push main and a v* tag simultaneously — those are separate groups but share the :buildcache tag and the smoke container name; tag only after the main build is green. (History: originally one global group serializing ALL runs for the single runner — with three runners that starved the queue; changed 2026-07-11, server-management#574.)

Three runners serve the fork (server-management#570/#574): ci-runner (VM 127 pve4, ubuntu-latest, 2 slots), bumblebee-runner (bumblebee, ubuntu-latest, 2 slots, jobs capped --cpus=4 --memory=10g so CI can't starve prod media playback), and small-runner (bumblebee, label small, 4 slots) — the small-jobs lane. The build and docs-reminder jobs use runs-on: small: Gitea dispatches a job as a runner task even when its if skips it, and those skip-tasks used to wait behind long builds (observed 31 min) stalling every PR run.

test job

dotnet restore → strip the Scanner project ref (sed -i '/Scanner/d', matching the Docker build) → dotnet build -c Release → dotnet test -c Release --no-build. Gates the image build.

• Shallow checkout: fetch-depth: 1 (ersatztv#190) — this job never runs git describe/git log, only build needs full history/tags for version computation, so test and migrations both check out shallow. build's checkout stays fetch-depth: 0.
• NuGet package cache: both test and migrations cache ~/.nuget/packages via actions/cache@v4, keyed on hashFiles('Directory.Packages.props', 'global.json') with a restore-keys OS-level fallback (ersatztv#190). Avoids a from-scratch dotnet restore on every run; the key only changes when the central package manifest or SDK pin changes.

build job

1. Compute INFO_VERSION (git describe + short sha on main; tag version on v*).
2. docker/setup-buildx-action with buildkitd-config-inline setting http = true for 192.168.1.95:3000 — BuildKit does not inherit the host daemon's insecure-registries, so without this, cache/base-image/push over the HTTP registry fails (http: server gave HTTP response to HTTPS client).
3. docker/login-action with repo secrets REGISTRY_USER / REGISTRY_PASSWORD.
4. docker/build-push-action@v6: amd64-only, docker/Dockerfile, INFO_VERSION build-arg, registry layer cache (type=registry,ref=…:buildcache, cache-to … ignore-error=true).
5. Smoke + IPTV E2E test: pull the just-pushed :<sha>, run it, poll for HTTP readiness (docker exec … python3 → http://localhost:8409/), then assert the real Jellyfin-facing surfaces on the freshly built image (ersatztv#16): /iptv/channels.m3u returns 2xx containing #EXTM3U, and /iptv/xmltv.xml returns 2xx containing a <tv root. xmltv.xml needs channels.xml (written by the scheduler a few seconds after boot), so each endpoint is polled with a deadline. Unique container name + trap … EXIT cleanup; dumps container logs on failure. Catches routing / base-URL (#1) / migration regressions that leave the app "up" but serving broken output.

docs-reminder job (non-blocking, PR-only)

A lightweight nudge that enforces the CLAUDE.md "docs-update is part of done" rule for the one case that's easy to forget and easy to detect: a PR that touches a SPA screen (web/src/screens/*.tsx) or ErsatzTV/LegacyUiRedirects.cs but does not update docs/blazor-route-parity.md. It diffs the PR against its base branch and emits a ::warning:: annotation (never fails the build — it's a reminder, not a gate; prose-doc gates get gamed with token edits). Deliberately has no setup-dotnet/setup-node (and thus no actions/cache), so it can't hit the cache-save hangs seen on the VM-127 runner (server-management#570). It does not cover the other doc obligations in the CLAUDE.md table (api-conventions, domain-model, decisions, spa-conventions) — those stay on the author.

Dockerfile notes (docker/Dockerfile)

• Base image: 192.168.1.95:3000/timothy/ersatztv-ffmpeg:8.1.2 (our Gitea fork of the archived ghcr.io/ersatztv/ersatztv-ffmpeg). FFmpeg 8 base image work landed in ersatztv-ffmpeg#4; app-side compatibility work landed in ersatztv#9.
• Copies Directory.Build.props, Directory.Build.targets, Directory.Packages.props, global.json, .editorconfig before dotnet restore so the image build uses the same MSBuild config, central package versions, SDK pin, and analyzer severities as local/CI builds (it previously copied only *.sln). Directory.Packages.props is required here: under Central Package Management the csproj carry no inline versions, so the image's restore fails (NU1015) without the central manifest.
• amd64-only (the runner/build host is x86_64). No arm32/arm64, no DMG/exe artifacts, no GHCR/DockerHub.
• openapi-generator jar layer ordering (ersatztv#190): the wget for the openapi-generator-cli jar runs before the COPY of ErsatzTV/wwwroot/openapi/, so the ~30MB download layer is cached independently of the openapi spec. Previously the jar was downloaded after that COPY, so any PR touching the spec (e.g. v1.json) busted the download layer too and re-fetched the jar on every such change. Codegen itself still runs after the spec COPY, since it needs both the jar and the spec files.

Dependency management (Central Package Management + scans)

Central Package Management (CPM) — package versions live in a single repo-root Directory.Packages.props (ManagePackageVersionsCentrally=true); the per-project csproj reference packages by name only (no Version=). One source of truth, atomic one-line bumps, and cross-project version drift is structurally impossible. To add or change a dependency, edit the <PackageVersion> entry centrally — never put a Version= back on a <PackageReference> (that trips NU1008). The Docker build must copy this file before restore (see Dockerfile notes). The .mcp/ vendored tool (gitignored, not in the solution) keeps inline versions via a local-only .mcp/Directory.Packages.props opt-out (ManagePackageVersionsCentrally=false). (ersatztv#14)

NuGet audit — .NET 10 runs NuGet audit on restore. Several projects set TreatWarningsAsErrors=true, so vulnerable transitive packages failed the build. Directory.Build.props demotes low/moderate/high advisories (NU1901-1903) to warnings and promotes NU1904 (critical) to an error in every project via WarningsAsErrors. The advisories that prompted this were resolved in ersatztv#8 (NCalcSync→6.x; SQLitePCLRaw bundle 3.x). The NU1901-1903 demotion is kept by design: criticals (NU1904) still hard- block, while low/moderate/high advisories surface as warnings + via the weekly scan and Renovate security PRs, rather than breaking unrelated PRs the moment a new transitive advisory drops.

Scheduled vulnerability scan — .gitea/workflows/dependency-scan.yml runs weekly (cron 0 6 * * 1) + on workflow_dispatch: dotnet list package --vulnerable --include-transitive over the full solution (incl. Scanner, which the image build strips). dotnet list exits 0 even with findings, so the step (bash -euo pipefail) greps for the "has the following vulnerable packages" marker and fails the run if present. Detection only — it surfaces advisories on a schedule, a Gitea-native stand-in for Dependabot; it does not open update PRs (that's Renovate — server-management#484). Gitea registers schedule triggers only from the default branch, so the cron starts after merge to main; use workflow_dispatch to run on demand. It went green once ersatztv#8 cleared the NCalcSync/SQLitePCLRaw advisories — a red run now means a new advisory has appeared. (ersatztv#14, ersatztv#8)

Renovate (automated update PRs) — .gitea/workflows/renovate.yml runs self-hosted Renovate weekly (cron 0 3 * * 1) + on workflow_dispatch, as a renovate/renovate:43 container job on the shared act_runner. This is the proposing layer the scan above deliberately omits: it opens grouped dependency-update PRs and OSV-driven vulnerability-fix PRs against main, and maintains a Dependency Dashboard issue listing the full backlog. Config is the repo-root renovate.json — managers nuget (via CPM), github-actions, and dockerfile (scoped to the built docker/Dockerfile; it reads the HTTP-only Gitea registry for the ersatztv-ffmpeg base via a RENOVATE_HOST_RULES host rule — insecureRegistry + registry read creds, set in the workflow env, not the committed config). The docker-compose manager is unused (repo compose files are build:-only). Auth: a dedicated renovate Gitea bot (Write collaborator) via repo Actions secrets RENOVATE_TOKEN (bot PAT) + GH_COM_TOKEN (no-scope github.com PAT for changelogs — named GH_, not GITHUB_, a prefix Gitea reserves). Patch bumps to test/dev-only packages (NUnit*, NSubstitute, Shouldly, coverlet, Microsoft.NET.Test.Sdk, Testably.Abstractions*, threading analyzer) auto-merge once the Build & test (.NET) check passes — branch protection on main requires that context; everything else is manual review (ersatztv is prod-bearing). Range-pinned packages (e.g. EF Core [9.0.x,10)) are respected — no v10 jump. PR volume is throttled (prConcurrentLimit 5 + config:recommended's prHourlyLimit 2); tick a dashboard checkbox or raise the limits to drain faster. workflow_dispatch defaults to a safe dry run. Cross-repo rollout tracked in server-management#484. (server-management#484)

Static analysis & formatting

Analyzer packs — Directory.Build.targets references Roslynator, SonarAnalyzer.CSharp, Meziantou.Analyzer, and AsyncFixer for every project (versions central via CPM; guarded on ManagePackageVersionsCentrally so the gitignored .mcp tool isn't pulled in). They are introduced incrementally (ersatztv#15): .editorconfig sets dotnet_analyzer_diagnostic.severity = suggestion so the packs surface findings without failing the TreatWarningsAsErrors (TWAE) build. Promotion is the enforcement — raising a rule to warning makes it a CI-blocking error via the existing TWAE build, so no separate lint step is needed.

• StyleCop.Analyzers is intentionally excluded: its latest stable (1.1.118) crashes (AD0001) on C# record declarations, and its rules overlap the existing .editorconfig/Roslynator. Revisit via the record-compatible 1.2.0-beta only if specifically wanted.
• Blazor .razor caveat: editorconfig severity overrides don't reach analyzer diagnostics in Razor @code (source-generator limitation — dotnet format can't fix them either), so the currently-firing SonarAnalyzer rules are temporarily NoWarn-ed in ErsatzTV.csproj and burned down rule-by-rule in ersatztv#25. The same rules run at suggestion on .cs.

Formatting — the tree isn't yet dotnet format-clean (mixed UTF-8 BOM + whitespace inherited from upstream: ~1,500 BOM files + ~480 whitespace). A one-time normalization lands as its own dedicated PR (kept out of the analyzer work to stay reviewable); afterwards dotnet format whitespace --verify-no-changes (+ style) joins the test job so drift can't return. .gitattributes already pins line endings.

Migration integrity (EF Core, both providers)

TvContext (ErsatzTV.Infrastructure/Data/TvContext.cs) has two migration sets — one per provider project: ErsatzTV.Infrastructure.Sqlite/Migrations and ErsatzTV.Infrastructure.MySql/Migrations, each with its own TvContextModelSnapshot. A model change needs a migration in BOTH. Add them with scripts/add-migration.sh <Name> (runs the EF CLI for each provider). The EF CLI pattern (provider selected by the post--- arg, which Startup reads as the provider config key):

dotnet ef <cmd> --context TvContext --startup-project ErsatzTV \
  --project ErsatzTV.Infrastructure.{Sqlite|MySql} -- --provider {Sqlite|MySql}

The migrations job in docker-build.yml runs on every push/PR and, for each provider:

1. dotnet ef migrations has-pending-model-changes — fails if an entity changed without a matching migration (model drift), so a forgotten migration can't merge.
2. dotnet ef database update against a fresh empty DB — applies all migrations in order and fails on any broken/un-orderable one.

• SQLite (the prod provider) uses a throwaway file (ETV_CONFIG_FOLDER=$(mktemp -d)); no service needed. Validated: 787 migrations → 139 tables.
• MySql uses ServerVersion.AutoDetect, which connects at config time, so the job needs a reachable server — provided by a services: mysql:8.4 container (the act_runner uses Docker execution with an auto-created per-job network — service reachable as mysql:3306 (the old bumblebee runner pinned network downloadswarm; relocated in server-management#570)). Connection string via MySql__ConnectionString (→ config key MySql:ConnectionString). Validated: 305 migrations → 137 tables. It's an independent gate (not yet a needs: of the image build) so the new MySql-service dependency can't block image builds until it's proven; promote it to a required check once stable.

Caveat — non-transactional operations: some migrations (e.g. SQLite PRAGMA foreign_keys) run outside a transaction and warn at startup; they can't be rolled back mid-migration, so review such migrations carefully (this is part of what motivated the apply-to-fresh check before the prod cutover, server-management#481).

Pre-commit hooks (web/)

The repo uses husky git hooks (installed via web/'s lint-staged + npm) to catch lint/format/type/API-drift errors locally, before they reach CI. Because the git root and the npm project dir differ (monorepo: no root package.json, the JS/TS project lives entirely in web/), the wiring is:

• husky + lint-staged are devDependencies of web/package.json (not a root package — there isn't one).
• The committed hook scripts live at the repo root: .husky/pre-commit, .husky/pre-push, .husky/commit-msg.
• web/package.json's prepare script (cd .. && husky) runs on npm install inside web/ and points git at the repo-root .husky dir (git config core.hooksPath .husky/_ — the _ subdir is husky's generated internal dir, gitignored via its own .husky/_/.gitignore; only the hook scripts themselves are committed). This works because npm keeps web/node_modules/.bin on PATH for the prepare script even after it cd ..s to the repo root (which husky's init requires — it hard-checks for .git in the current directory).

The four hooks:

1. pre-commit — (a) cd web && npx lint-staged: runs eslint --fix on staged web/src/**/*.{ts,tsx} files, then a project-wide npm run typecheck (tsc -b isn't file-scoped, so it runs the full check, but only when a .ts/.tsx file is staged); (b) back at the repo root, if any *.cs files are staged, dotnet format ErsatzTV.sln --verify-no-changes --include <staged .cs> — a formatting violation blocks the commit. The .cs step is skipped entirely when no .cs is staged, so web-only commits don't pay the sln-load cost; when it does run it's scoped to the staged files (~6-7s wall in practice, dominated by the workspace load).
2. pre-push — CI-parity gate: cd web && npm run check:api && npm run lint && npm run typecheck && npm run build. check:api guards generated-OpenAPI drift (ErsatzTV/wwwroot/openapi/v1.json → web/src/api/generated/v1.d.ts); the full lint/typecheck/build catch a staged change that breaks an unstaged file (lint-staged only sees staged files). Any failure blocks the push.
3. commit-msg — enforces the CLAUDE.md protocol: the message must carry a Co-Authored-By: trailer, else the commit is rejected. Merge commits are exempt (detected via git rev-parse --verify MERGE_HEAD).

• Worktree/subdir gotcha: git exports GIT_DIR (and friends) while running hooks. In a worktree or any subdir, an explicit GIT_DIR makes nested git commands mislocate the working tree — pre-push's check:api (git diff --exit-code, run from web/) then silently reports "no diff" and lets drift through. pre-push therefore unsets GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE first. (pre-commit's .cs collection uses git diff --cached, index-vs-HEAD, which needs only GIT_DIR and is unaffected.)
• Practical effect: a fresh web/ npm install (after cloning or pulling this change) installs all four hooks automatically — no separate setup step. Commits that touch only non-web/, non-.cs files skip linting/formatting (lint-staged no-ops with nothing to run, the .cs step is skipped).

Registry

Gitea Packages, HTTP-only at 192.168.1.95:3000. the ci-runner VM's Docker daemon (192.168.1.127) has it as an insecure-registry (server-management#172; runner relocated off jazz in #570). Images: 192.168.1.95:3000/timothy/ersatztv:<tag>.

Test / prod environments

Container/compose wiring lives in server-management (project boundary): test ersatztv-test on 8410 (:latest), prod ersatztv on 8409 (:prod). See server-management#481 for the full spec (registry pull on jazz, volumes, Jellyfin isolation for test, Watchtower/manual promotion).

Retired upstream workflows

The upstream .github/workflows/ (ci.yml, docker.yml, artifacts.yml, release.yml, pr.yml, issue-stale.yml) were removed — they targeted GHCR/DockerHub + Azure/Apple signing and called reusable workflows at dead ersatztv/ersatztv@main paths, and ran as noise (incl. a daily stale-issue cron) on the Gitea runner. Upstream is archived, so there are no future merges to preserve them for. The dead .github/dependabot.yml and FUNDING.yml (upstream-pointed) were also removed.

Known follow-ups

• Pin third-party actions to commit SHAs (currently floating major tags cloned from github.com at runtime) — low priority for a homelab; tracked informally.