ersatztv/docs/ci-cd.md

# CI/CD for the ErsatzTV Fork

The fork builds its own Docker image via **Gitea Actions** on the homelab and pushes
to the **Gitea container registry**. Runner + registry were provisioned in
server-management#172; the build pipeline is ersatztv#4; test/prod containers are
server-management#481.

## Versioning & releases

The fork inherits upstream ErsatzTV's scheme: **`vYY.<release-seq>.<patch>`** (lightweight, `v`-prefixed git tags).

- **`YY`** — two-digit year.
- **`<release-seq>`** — a sequential release counter **within the year**, reset at each year boundary. It is **not** the calendar month. (Evidence: `v25.2.0` shipped in June 2025, `v25.5.0` in Sep, `v26.3.0` in Feb 2026 — minors don't track months; and `v25.9.0` → `v26.1.0` shows the year-reset.)
- **`<patch>`** — a small follow-up/hotfix on the *same* release line (e.g. `v26.1.0` → `v26.1.1`, days later).

Upstream's final release was **`v26.3.0`** (archived). Our line continues from there:

| Tag | Meaning |
|-----|---------|
| `v26.3.1` | Upstream 26.3.0 **rebuilt on our infra** (Gitea CI/registry, fork ffmpeg base) — **no application changes**. A patch bump, because nothing functional changed. |
| `v26.4.0` | Reserved for our **first release that carries actual app changes** (e.g. the #1 M3U fix). Later 2026 releases: `26.5.0`, `26.6.0`, …; a new year resets to `27.1.0`. |

**Cutting a release:** push a `vYY.N.P` tag on `main` → CI builds `:prod` + `:<version>` + `:<sha>`, then the **`bump-prod-compose`** job rewrites the pinned `ersatztv:<version>` tag in the server-management media-servers compose (via the `ersatztv-ci-deploy` write deploy key, id 5 on server-management + the `SERVERMGMT_DEPLOY_KEY` secret here) and pushes to its `master` branch → the Gitea→Komodo webhook redeploys prod with a pre-deploy backup. **No manual pin bump needed** (the job is idempotent: a no-op if the compose already pins that version).

**Gotcha:** never put a `[skip ci]` token in a commit you intend to tag — Gitea reads skip-ci from the *tagged* commit and will **suppress the release build**. (Also, `workflow_dispatch` on a tag ref isn't supported on this Gitea version, so the tag *push* must do the triggering.) Release commits, and anything you'll tag, must not contain skip-ci.

**Also avoid firing several pushes back-to-back** (e.g. a `[skip ci]` commit, then `main`, then a tag, all within ~1s). Observed once on this Gitea instance: the later events were silently dropped — no `ActionRun` records created at all, even though the runner was online and the workflow `active`. Pushing again, spaced out, created the runs normally. If a push/tag doesn't produce a run, re-push (or push an empty commit) rather than assuming the runner is broken.

## The workflow: `.gitea/workflows/docker-build.yml`

Single workflow. Gating jobs `test` + `migrations` run in parallel and gate `build`; a
non-blocking `docs-reminder` job runs on PRs only (see below); a `bump-prod-compose` job
runs only on `v*` tags (after `build`) to auto-deploy prod (see "Cutting a release").

### Triggers & tags

| Trigger | `test` job | `build` job | Image tags pushed |
|---------|:----------:|:-----------:|-------------------|
| `pull_request` | ✅ | — (skipped) | none |
| push to `main` | ✅ | ✅ | `:latest` + `:<short-sha>` |
| push tag `v*` | ✅ | ✅ | `:prod` + `:<version>` + `:<short-sha>` |
| `workflow_dispatch` | ✅ | ✅ | only if ref is `main`/`v*`, else build-only (no push) |

`:latest` is the **test/dev** channel (every `main` commit). Prod pins the immutable
**`:<version>`** tag (e.g. `ersatztv:26.7.0`) in the compose — never `:latest`, and no
longer a floating `:prod` (pinned since 2026-07-07) — so a redeploy is reproducible and
gated by the `bump-prod-compose` commit. Both `:prod` and `:<version>` are produced by
pushing a `v*` tag; the `bump-prod-compose` job pins the `:<version>` form.

Concurrency is scoped **per event+ref** (`group: ersatztv-build-${{ github.event_name }}-${{ github.ref }}`,
`cancel-in-progress` for PRs): PR runs parallelize across PRs, a new sync auto-cancels
its superseded run, and image builds still serialize within their own ref. Do NOT push
`main` and a `v*` tag simultaneously — those are separate groups but share the
`:buildcache` tag and the smoke container name; tag only after the main build is green.
(History: originally one global group serializing ALL runs for the single runner —
with three runners that starved the queue; changed 2026-07-11, server-management#574.)

Three runners serve the fork (server-management#570/#574): `ci-runner` (VM 127 pve4,
`ubuntu-latest`, 2 slots), `bumblebee-runner` (bumblebee, `ubuntu-latest`, 2 slots,
jobs capped `--cpus=4 --memory=10g` so CI can't starve prod media playback), and
`small-runner` (bumblebee, label **`small`**, 4 slots) — the small-jobs lane. The
`build` and `docs-reminder` jobs use `runs-on: small`: Gitea dispatches a job as a
runner task even when its `if` skips it, and those skip-tasks used to wait behind
long builds (observed 31 min) stalling every PR run.

### `test` job

`dotnet restore` → strip the Scanner project ref (`sed -i '/Scanner/d'`, matching the
Docker build) → `dotnet build -c Release` → `dotnet test -c Release --no-build`. Gates
the image build.

- **Shallow checkout**: `fetch-depth: 1` (ersatztv#190) — this job never runs `git
  describe`/`git log`, only `build` needs full history/tags for version computation, so
  `test` and `migrations` both check out shallow. `build`'s checkout stays `fetch-depth: 0`.
- **NuGet package cache**: both `test` and `migrations` cache `~/.nuget/packages` via
  `actions/cache@v4`, keyed on `hashFiles('Directory.Packages.props', 'global.json')` with
  a `restore-keys` OS-level fallback (ersatztv#190). Avoids a from-scratch `dotnet restore`
  on every run; the key only changes when the central package manifest or SDK pin changes.

### `build` job

1. Compute `INFO_VERSION` (`git describe` + short sha on `main`; tag version on `v*`).
2. `docker/setup-buildx-action` with `buildkitd-config-inline` setting `http = true`
   for `192.168.1.95:3000` — **BuildKit does not inherit the host daemon's
   `insecure-registries`**, so without this, cache/base-image/push over the HTTP
   registry fails (`http: server gave HTTP response to HTTPS client`).
3. `docker/login-action` with repo secrets `REGISTRY_USER` / `REGISTRY_PASSWORD`.
4. `docker/build-push-action@v6`: amd64-only, `docker/Dockerfile`, `INFO_VERSION`
   build-arg, registry layer cache (`type=registry,ref=…:buildcache`,
   `cache-to … ignore-error=true`).
5. **Smoke + IPTV E2E test**: pull the just-pushed `:<sha>`, run it, poll for HTTP
   readiness (`docker exec … python3` → `http://localhost:8409/`), then assert the real
   Jellyfin-facing surfaces on the freshly built image (ersatztv#16): `/iptv/channels.m3u`
   returns 2xx containing `#EXTM3U`, and `/iptv/xmltv.xml` returns 2xx containing a `<tv`
   root. `xmltv.xml` needs `channels.xml` (written by the scheduler a few seconds after
   boot), so each endpoint is polled with a deadline. Unique container name + `trap … EXIT`
   cleanup; dumps container logs on failure. Catches routing / base-URL (#1) / migration
   regressions that leave the app "up" but serving broken output.

### `docs-reminder` job (non-blocking, PR-only)

A lightweight nudge that enforces the CLAUDE.md "docs-update is part of done" rule for the
one case that's easy to forget and easy to detect: a PR that touches a SPA screen
(`web/src/screens/*.tsx`) or `ErsatzTV/LegacyUiRedirects.cs` but **does not** update
`docs/blazor-route-parity.md`. It diffs the PR against its base branch and emits a
`::warning::` annotation (never fails the build — it's a reminder, not a gate; prose-doc
gates get gamed with token edits). Deliberately has **no** `setup-dotnet`/`setup-node` (and
thus no `actions/cache`), so it can't hit the cache-save hangs seen on the VM-127 runner
(server-management#570). It does not cover the other doc obligations in the CLAUDE.md table
(api-conventions, domain-model, decisions, spa-conventions) — those stay on the author.

## Dockerfile notes (`docker/Dockerfile`)

- Base image: **`192.168.1.95:3000/timothy/ersatztv-ffmpeg:8.1.2`** (our Gitea fork of
  the archived `ghcr.io/ersatztv/ersatztv-ffmpeg`). FFmpeg 8 base image work landed in
  ersatztv-ffmpeg#4; app-side compatibility work landed in ersatztv#9.
- Copies `Directory.Build.props`, `Directory.Build.targets`, `Directory.Packages.props`,
  `global.json`, `.editorconfig` before `dotnet restore` so the image build uses the same
  MSBuild config, central package versions, SDK pin, and analyzer severities as local/CI
  builds (it previously copied only `*.sln`). `Directory.Packages.props` is **required**
  here: under Central Package Management the csproj carry no inline versions, so the
  image's restore fails (`NU1015`) without the central manifest.
- amd64-only (the runner/build host is x86_64). No arm32/arm64, no DMG/exe artifacts, no GHCR/DockerHub.
- **openapi-generator jar layer ordering** (ersatztv#190): the `wget` for the
  openapi-generator-cli jar runs *before* the `COPY` of `ErsatzTV/wwwroot/openapi/`, so the
  ~30MB download layer is cached independently of the openapi spec. Previously the jar was
  downloaded after that `COPY`, so any PR touching the spec (e.g. `v1.json`) busted the
  download layer too and re-fetched the jar on every such change. Codegen itself still runs
  after the spec `COPY`, since it needs both the jar and the spec files.

## Dependency management (Central Package Management + scans)

**Central Package Management (CPM)** — package versions live in a single repo-root
`Directory.Packages.props` (`ManagePackageVersionsCentrally=true`); the per-project
csproj reference packages by name only (no `Version=`). One source of truth, atomic
one-line bumps, and cross-project version drift is structurally impossible. To add or
change a dependency, edit the `<PackageVersion>` entry centrally — never put a `Version=`
back on a `<PackageReference>` (that trips `NU1008`). The Docker build must copy this file
before restore (see Dockerfile notes). The `.mcp/` vendored tool (gitignored, not in the
solution) keeps inline versions via a **local-only** `.mcp/Directory.Packages.props`
opt-out (`ManagePackageVersionsCentrally=false`). (ersatztv#14)

**NuGet audit** — .NET 10 runs NuGet audit on restore. Several projects set
`TreatWarningsAsErrors=true`, so vulnerable transitive packages failed the build.
`Directory.Build.props` demotes low/moderate/high advisories (NU1901-1903) to warnings
and promotes NU1904 (critical) to an error in **every** project via `WarningsAsErrors`.
The advisories that prompted this were resolved in ersatztv#8 (NCalcSync→6.x; SQLitePCLRaw
bundle 3.x). The NU1901-1903 demotion is **kept by design**: criticals (NU1904) still hard-
block, while low/moderate/high advisories surface as warnings + via the weekly scan and
Renovate security PRs, rather than breaking unrelated PRs the moment a new transitive
advisory drops.

**Scheduled vulnerability scan** — `.gitea/workflows/dependency-scan.yml` runs weekly
(cron `0 6 * * 1`) + on `workflow_dispatch`: `dotnet list package --vulnerable
--include-transitive` over the **full** solution (incl. Scanner, which the image build
strips). `dotnet list` exits 0 even with findings, so the step (`bash -euo pipefail`)
greps for the "has the following vulnerable packages" marker and fails the run if present.
Detection only — it surfaces advisories on a schedule, a Gitea-native stand-in for
Dependabot; it does **not** open update PRs (that's Renovate — server-management#484).
Gitea registers `schedule` triggers only from the default branch, so the cron starts
after merge to `main`; use `workflow_dispatch` to run on demand. It went **green** once
ersatztv#8 cleared the NCalcSync/SQLitePCLRaw advisories — a red run now means a **new**
advisory has appeared. (ersatztv#14, ersatztv#8)

**Renovate (automated update PRs)** — `.gitea/workflows/renovate.yml` runs self-hosted
[Renovate](https://docs.renovatebot.com) weekly (cron `0 3 * * 1`) + on `workflow_dispatch`,
as a `renovate/renovate:43` container job on the shared act_runner. This is the *proposing*
layer the scan above deliberately omits: it opens grouped dependency-update PRs and
OSV-driven vulnerability-fix PRs against `main`, and maintains a **Dependency Dashboard**
issue listing the full backlog. Config is the repo-root `renovate.json` — managers `nuget`
(via CPM), `github-actions`, and `dockerfile` (scoped to the built `docker/Dockerfile`; it reads the
HTTP-only Gitea registry for the `ersatztv-ffmpeg` base via a `RENOVATE_HOST_RULES` host rule —
`insecureRegistry` + registry read creds, set in the workflow env, not the committed config). The
docker-compose manager is unused (repo compose files are `build:`-only). Auth: a dedicated
**`renovate` Gitea bot** (Write
collaborator) via repo Actions secrets `RENOVATE_TOKEN` (bot PAT) + `GH_COM_TOKEN` (no-scope
github.com PAT for changelogs — named `GH_`, **not** `GITHUB_`, a prefix Gitea reserves).
**Patch** bumps to test/dev-only packages (NUnit\*, NSubstitute, Shouldly, coverlet,
`Microsoft.NET.Test.Sdk`, `Testably.Abstractions*`, threading analyzer) **auto-merge** once
the `Build & test (.NET)` check passes — branch protection on `main` requires that context;
everything else is manual review (ersatztv is prod-bearing). Range-pinned packages (e.g. EF
Core `[9.0.x,10)`) are respected — no v10 jump. PR volume is throttled (`prConcurrentLimit`
5 + `config:recommended`'s `prHourlyLimit` 2); tick a dashboard checkbox or raise the limits
to drain faster. `workflow_dispatch` defaults to a safe **dry run**. Cross-repo rollout
tracked in server-management#484. (server-management#484)

## Static analysis & formatting

**Analyzer packs** — `Directory.Build.targets` references **Roslynator**, **SonarAnalyzer.CSharp**,
**Meziantou.Analyzer**, and **AsyncFixer** for every project (versions central via CPM; guarded on
`ManagePackageVersionsCentrally` so the gitignored `.mcp` tool isn't pulled in). They are introduced
**incrementally** (ersatztv#15): `.editorconfig` sets `dotnet_analyzer_diagnostic.severity = suggestion`
so the packs surface findings without failing the `TreatWarningsAsErrors` (TWAE) build. **Promotion is
the enforcement** — raising a rule to `warning` makes it a CI-blocking error via the existing TWAE
build, so no separate lint step is needed.

- **StyleCop.Analyzers is intentionally excluded**: its latest stable (1.1.118) crashes (`AD0001`) on
  C# `record` declarations, and its rules overlap the existing `.editorconfig`/Roslynator. Revisit via
  the record-compatible `1.2.0-beta` only if specifically wanted.
- **Blazor `.razor` caveat**: editorconfig severity overrides don't reach analyzer diagnostics in Razor
  `@code` (source-generator limitation — `dotnet format` can't fix them either), so the currently-firing
  SonarAnalyzer rules are temporarily `NoWarn`-ed in `ErsatzTV.csproj` and burned down rule-by-rule in
  **ersatztv#25**. The same rules run at `suggestion` on `.cs`.

**Formatting** — the tree isn't yet `dotnet format`-clean (mixed UTF-8 BOM + whitespace inherited from
upstream: ~1,500 BOM files + ~480 whitespace). A one-time normalization lands as its **own dedicated
PR** (kept out of the analyzer work to stay reviewable); afterwards `dotnet format whitespace
--verify-no-changes` (+ `style`) joins the `test` job so drift can't return. `.gitattributes` already
pins line endings.

## Migration integrity (EF Core, both providers)

`TvContext` (`ErsatzTV.Infrastructure/Data/TvContext.cs`) has **two** migration sets — one per
provider project: `ErsatzTV.Infrastructure.Sqlite/Migrations` and
`ErsatzTV.Infrastructure.MySql/Migrations`, each with its own `TvContextModelSnapshot`. **A model
change needs a migration in BOTH.** Add them with `scripts/add-migration.sh <Name>` (runs the EF CLI
for each provider). The EF CLI pattern (provider selected by the post-`--` arg, which `Startup`
reads as the `provider` config key):

```
dotnet ef <cmd> --context TvContext --startup-project ErsatzTV \
  --project ErsatzTV.Infrastructure.{Sqlite|MySql} -- --provider {Sqlite|MySql}
```

The **`migrations` job** in `docker-build.yml` runs on every push/PR and, for **each** provider:

1. `dotnet ef migrations has-pending-model-changes` — fails if an entity changed without a matching
   migration (**model drift**), so a forgotten migration can't merge.
2. `dotnet ef database update` against a **fresh empty DB** — applies *all* migrations in order and
   fails on any broken/un-orderable one.

- **SQLite** (the prod provider) uses a throwaway file (`ETV_CONFIG_FOLDER=$(mktemp -d)`); no service
  needed. Validated: 787 migrations → 139 tables.
- **MySql** uses `ServerVersion.AutoDetect`, which **connects at config time**, so the job needs a
  reachable server — provided by a `services: mysql:8.4` container (the act_runner uses Docker
  execution with an auto-created per-job network — service reachable as `mysql:3306` (the old
  bumblebee runner pinned network `downloadswarm`; relocated in server-management#570)). Connection
  string via `MySql__ConnectionString` (→ config key `MySql:ConnectionString`). Validated: 305
  migrations → 137 tables. It's an **independent gate** (not yet a `needs:` of the image build) so
  the new MySql-service dependency can't block image builds until it's proven; promote it to a
  required check once stable.

**Caveat — non-transactional operations**: some migrations (e.g. SQLite `PRAGMA foreign_keys`) run
outside a transaction and warn at startup; they can't be rolled back mid-migration, so review such
migrations carefully (this is part of what motivated the apply-to-fresh check before the prod
cutover, server-management#481).

## Pre-commit hooks (web/)

The repo uses **husky** git hooks (installed via `web/`'s **lint-staged** + npm) to catch
lint/format/type/API-drift errors locally, before they reach CI. Because the git root and
the npm project dir differ (monorepo: no root `package.json`, the JS/TS project lives
entirely in `web/`), the wiring is:

- `husky` + `lint-staged` are devDependencies of `web/package.json` (not a root package —
  there isn't one).
- The committed hook scripts live at the repo root: `.husky/pre-commit`, `.husky/pre-push`,
  `.husky/commit-msg`.
- `web/package.json`'s `prepare` script (`cd .. && husky`) runs on `npm install` inside
  `web/` and points git at the repo-root `.husky` dir (`git config core.hooksPath
  .husky/_` — the `_` subdir is husky's generated internal dir, gitignored via its own
  `.husky/_/.gitignore`; only the hook scripts themselves are committed). This works
  because npm keeps `web/node_modules/.bin` on `PATH` for the `prepare` script even after
  it `cd ..`s to the repo root (which husky's init requires — it hard-checks for `.git`
  in the *current* directory).

**The four hooks:**

1. **`pre-commit`** — (a) `cd web && npx lint-staged`: runs `eslint --fix` on staged
   `web/src/**/*.{ts,tsx}` files, then a project-wide `npm run typecheck` (`tsc -b` isn't
   file-scoped, so it runs the full check, but only when a `.ts`/`.tsx` file is staged);
   (b) back at the repo root, if any **`*.cs`** files are staged, `dotnet format
   ErsatzTV.sln --verify-no-changes --include <staged .cs>` — a formatting violation
   blocks the commit. The .cs step is **skipped entirely when no .cs is staged**, so
   web-only commits don't pay the sln-load cost; when it does run it's scoped to the staged
   files (~6-7s wall in practice, dominated by the workspace load).
2. **`pre-push`** — CI-parity gate: `cd web && npm run check:api && npm run lint && npm run
   typecheck && npm run build`. `check:api` guards generated-OpenAPI drift
   (`ErsatzTV/wwwroot/openapi/v1.json` → `web/src/api/generated/v1.d.ts`); the full
   lint/typecheck/build catch a staged change that breaks an *unstaged* file (lint-staged
   only sees staged files). Any failure blocks the push.
3. **`commit-msg`** — enforces the CLAUDE.md protocol: the message must carry a
   `Co-Authored-By:` trailer, else the commit is rejected. Merge commits are exempt
   (detected via `git rev-parse --verify MERGE_HEAD`).

- **Worktree/subdir gotcha**: git exports `GIT_DIR` (and friends) while running hooks. In a
  worktree or any subdir, an explicit `GIT_DIR` makes nested `git` commands mislocate the
  working tree — `pre-push`'s `check:api` (`git diff --exit-code`, run from `web/`) then
  silently reports "no diff" and lets drift through. `pre-push` therefore `unset`s
  `GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE` first. (`pre-commit`'s `.cs` collection uses
  `git diff --cached`, index-vs-HEAD, which needs only `GIT_DIR` and is unaffected.)
- **Practical effect**: a fresh `web/` `npm install` (after cloning or pulling this change)
  installs all four hooks automatically — no separate setup step. Commits that touch only
  non-`web/`, non-`.cs` files skip linting/formatting (lint-staged no-ops with nothing to
  run, the `.cs` step is skipped).

## Registry

Gitea Packages, HTTP-only at `192.168.1.95:3000`. the `ci-runner` VM's Docker daemon (192.168.1.127) has it as an
insecure-registry (server-management#172; runner relocated off jazz in #570). Images: `192.168.1.95:3000/timothy/ersatztv:<tag>`.

## Test / prod environments

Container/compose wiring lives in **server-management** (project boundary): test
`ersatztv-test` on 8410 (`:latest`), prod `ersatztv` on 8409 (`:prod`). See
server-management#481 for the full spec (registry pull on jazz, volumes, Jellyfin
isolation for test, Watchtower/manual promotion).

## Retired upstream workflows

The upstream `.github/workflows/` (`ci.yml`, `docker.yml`, `artifacts.yml`,
`release.yml`, `pr.yml`, `issue-stale.yml`) were removed — they targeted
GHCR/DockerHub + Azure/Apple signing and called reusable workflows at dead
`ersatztv/ersatztv@main` paths, and ran as noise (incl. a daily stale-issue cron) on
the Gitea runner. Upstream is archived, so there are no future merges to preserve them
for. The dead `.github/dependabot.yml` and `FUNDING.yml` (upstream-pointed) were also
removed.

## Known follow-ups

- Pin third-party actions to commit SHAs (currently floating major tags cloned from
  github.com at runtime) — low priority for a homelab; tracked informally.