feat(ci): #315 migration-on-prod-copy smoke for the release path

CI's migrations job only proves a migration is well-formed against a fresh,
empty DB. It never exercises the migration — or ErsatzTV's startup data steps
(DatabaseMigratorService -> DbInitializer + PopulatePathHashes over the real
MediaFile table) — against the accumulated prod SQLite, so a migration green on
a fresh DB can still fail/corrupt on prod, found only mid-deploy.

scripts/migration-smoke.sh rehearses it on a THROWAWAY copy of the latest prod
backup: boots the new image against the copy, gates PASS on the "Done applying
database migrations" log line (the migrator is a BackgroundService running
concurrently with Kestrel, so HTTP readiness alone doesn't prove migrations
finished), FAILs on early container exit / migration exception / timeout / not
serving afterwards. Always operates on a copy, never the live DB; tears down its
container + temp dir (incl. the container's root-owned config files) on exit.

Validated live 2026-07-12: :latest vs a copy of the 283MB prod backup ->
migrations applied cleanly, app booted+served, temp dir removed.

Home split: this repo owns the script + docs; wiring it into the Komodo
pre-deploy step is server-management#589 (cross-repo). Docs: docs/ci-cd.md
(Migration integrity), docs/decisions.md (new entry, pure insertion).

fixes #315

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/ci-cd.md

@@ -281,6 +281,42 @@ migration failure fails deterministically on every attempt, so the retry never m
 still flakes past the retry, re-trigger (Gitea has no rerun API on this version — push, or the run
 drains); don't treat a lone MySql-apply red as a code problem without checking the failure mode.
 
+### Migration-on-prod-copy smoke — release path (`scripts/migration-smoke.sh`, ersatztv#315)
+
+The `migrations` job above only proves a migration is **well-formed against a fresh, empty DB**. It
+can't prove it applies cleanly to the **accumulated prod SQLite** — real row volume, historical values,
+and the post-migration data steps ErsatzTV runs on startup: `DatabaseMigratorService` (a
+`BackgroundService`) applies pending migrations, then `DbInitializer.Initialize` + `PopulatePathHashes`
+(an `UPDATE` over the real `MediaFile` table). A migration green on a fresh DB can still fail or corrupt
+on prod, and today you'd only find out mid-deploy after the container recreates.
+
+`scripts/migration-smoke.sh` rehearses it on a **throwaway copy** of the latest prod backup — it never
+touches the live DB:
+
+```
+scripts/migration-smoke.sh --image <ref-about-to-be-promoted> [--db <backup.sqlite3>] [--timeout 180]
+```
+
+It copies the backup into a temp config dir, boots the new image against it (`ETV_CONFIG_FOLDER`), and
+**gates PASS on the `Done applying database migrations` log line** — not merely on HTTP readiness, since
+the migrator runs concurrently with Kestrel, so the web server can serve before/while migrations run.
+FAIL = the container exits before finishing, a migration exception appears in the logs, migrations
+don't finish within `--timeout`, or the app won't serve `/iptv/channels.m3u` afterwards. The smoke
+container, the DB copy, and the temp dir are always torn down on exit (the ErsatzTV image runs as root,
+so cleanup deletes its root-owned config files from inside a throwaway root container — otherwise each
+run would leak the multi-hundred-MB copy). Exit 0 = clean, 1 = migration/boot failure, 2 = usage error.
+
+- **`--db` default**: the newest `ersatztv.sqlite3` under `$ETV_BACKUP_DIR`
+  (default `~/downloadswarm/ersatztv-backups` — where the host-side pre-deploy backup hook writes
+  timestamped snapshots). Pass `--image` = the version tag about to be promoted.
+- **Where it runs**: it's meant to run **on the docker host** as a **Komodo pre-deploy step** (which
+  already produces the backup — see "Cutting a release" and the #553 pre-deploy backup caveat), so a
+  bad migration aborts the promote before the live container recreates. Wiring it into that hook is a
+  server-management concern (cross-repo — this repo owns the script + docs, server-management owns the
+  Komodo hook). Until wired, run it by hand before cutting a migration-bearing release.
+- Validated live 2026-07-12: `:latest` against a copy of the 283 MB prod backup → migrations applied
+  cleanly, app booted and served, temp dir removed.
+
 ## Pre-commit hooks (web/)
 
 The repo uses **husky** git hooks (installed via `web/`'s **lint-staged** + npm) to catch

docs/decisions.md

@@ -79,6 +79,7 @@ keep append-only from accreting stale, contradictory, or unreadably-large histor
 - [2026-07-12 — Live-E2E is a required step for API write-path handler changes (#303)](#2026-07-12--live-e2e-is-a-required-step-for-api-write-path-handler-changes-303)
 - [2026-07-12 — Formatting-as-you-touch, enforced; rebase-not-merge for PR branches (#311 H11 + format CI)](#2026-07-12--formatting-as-you-touch-enforced-rebase-not-merge-for-pr-branches-311-h11--format-ci)
 - [2026-07-12 — Merge-consent gate auto-grants when satisfied (no redundant prompt); state IS the consent (#314)](#2026-07-12--merge-consent-gate-auto-grants-when-satisfied-no-redundant-prompt-state-is-the-consent-314)
+- [2026-07-12 — Release path rehearses migrations on a prod-DB copy before promoting (#315)](#2026-07-12--release-path-rehearses-migrations-on-a-prod-db-copy-before-promoting-315)
 
 ---
 
@@ -1554,3 +1555,22 @@ no-creds→ask, non-merge→passthrough).
 also ask conversationally to merge a PR whose gate auto-grants. A separate human confirmation is still
 warranted only when the gate **asks** (state not derivable). This supersedes the "always confirm merge
 consent in-conversation per session" phrasing in the kickoff HARD CONSTRAINTS (updated in the same PR).
+
+## 2026-07-12 — Release path rehearses migrations on a prod-DB copy before promoting (#315)
+
+The CI `migrations` job proves a migration is well-formed against a **fresh, empty** DB (model-drift +
+apply-to-fresh, per provider). That is necessary but not sufficient: it never exercises the migration —
+or ErsatzTV's startup data steps (`DatabaseMigratorService` → `DbInitializer` + `PopulatePathHashes`
+over the real `MediaFile` table) — against the **accumulated prod SQLite**, where row volume and
+historical values differ. A migration green on a fresh DB can still fail or corrupt on prod, discovered
+only mid-deploy after the container recreates.
+
+Decision: before promoting a migration-bearing release, **rehearse** the new image's migrations against
+a **throwaway copy of the latest prod backup** via `scripts/migration-smoke.sh` — boot the new image
+against the copy, gate PASS on the `Done applying database migrations` log line (the migrator is a
+`BackgroundService` running concurrently with Kestrel, so HTTP-readiness alone does *not* prove
+migrations finished), FAIL on early container exit / a migration exception / timeout / not serving
+afterwards. Always operates on a copy, never the live DB. Home: the script + docs are ours; wiring it
+into the Komodo **pre-deploy** step (which already produces the backup) is a server-management concern.
+Rationale: data-plane rigor — catch a bad migration on a disposable copy, not on live prod data.
+See `docs/ci-cd.md` → Migration-on-prod-copy smoke. Cross-repo wiring tracked in server-management.

scripts/migration-smoke.sh

@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+# scripts/migration-smoke.sh — rehearse a release's EF migrations against a COPY of the real prod
+# database before promoting the new image (ersatztv#315). Data-plane release rigor.
+#
+# WHY: CI's `migrations` job only proves a migration is well-formed against a FRESH, empty DB
+# (model-drift + apply-to-fresh per provider). It cannot prove the migration applies cleanly to the
+# accumulated prod SQLite — real row volume, historical values, and the post-migration data steps
+# ErsatzTV runs on startup (DatabaseMigratorService → DbInitializer + PopulatePathHashes over the real
+# MediaFile table). A migration green on a fresh DB can still fail or corrupt on prod. This smoke boots
+# the NEW image against a THROWAWAY copy of the latest prod backup, waits for the migrator to finish,
+# and reports pass/fail — so a bad migration is caught on a copy, not mid-deploy on live data.
+#
+# It is designed to run ON the docker host (bumblebee) as a Komodo pre-deploy step (server-management
+# owns that wiring — see ersatztv#315), or by hand before a release. It NEVER touches the live prod DB.
+#
+# Usage:
+#   scripts/migration-smoke.sh --image <ref> [--db <path>] [--timeout <sec>]
+#
+#   --image <ref>   Image to rehearse — the one about to be PROMOTED (e.g.
+#                   192.168.1.95:3000/timothy/ersatztv:26.6.0). Required.
+#   --db <path>     A prod SQLite backup file to copy and migrate. Default: the newest
+#                   $ETV_BACKUP_DIR/*/ersatztv.sqlite3 (ETV_BACKUP_DIR defaults to
+#                   ~/downloadswarm/ersatztv-backups — where the Komodo pre-deploy hook writes them).
+#   --timeout <sec> Seconds to wait for migrations to finish (default 180). Big prod DBs + the path-hash
+#                   backfill can take a while on first upgrade.
+#
+# Exit 0 = the new image applied all pending migrations to the prod-copy and booted cleanly.
+# Exit 1 = a migration failed / the container exited before finishing / it never booted (logs dumped).
+# Exit 2 = usage / precondition error (no image, no backup found, docker missing).
+#
+# The container, the temp config dir, and the DB copy are always cleaned up on exit.
+
+set -euo pipefail
+
+IMAGE=""
+DB=""
+TIMEOUT=180
+BACKUP_DIR="${ETV_BACKUP_DIR:-$HOME/downloadswarm/ersatztv-backups}"
+
+die() { echo "migration-smoke: $*" >&2; exit 2; }
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --image) IMAGE="${2:-}"; shift 2 ;;
+    --db)    DB="${2:-}"; shift 2 ;;
+    --timeout) TIMEOUT="${2:-}"; shift 2 ;;
+    -h|--help) sed -n '2,40p' "$0"; exit 0 ;;
+    *) die "unknown argument: $1 (see --help)" ;;
+  esac
+done
+
+command -v docker >/dev/null 2>&1 || die "docker not found on PATH (run this on the docker host)"
+[ -n "$IMAGE" ] || die "--image <ref> is required (the image about to be promoted)"
+
+# Resolve the backup DB: explicit --db, else newest ersatztv.sqlite3 under $BACKUP_DIR.
+if [ -z "$DB" ]; then
+  DB=$(find "$BACKUP_DIR" -type f -name 'ersatztv.sqlite3' -printf '%T@ %p\n' 2>/dev/null \
+       | sort -rn | head -1 | cut -d' ' -f2-)
+  [ -n "$DB" ] || die "no ersatztv.sqlite3 backup found under $BACKUP_DIR (pass --db explicitly)"
+  echo "migration-smoke: using newest backup: $DB"
+fi
+[ -f "$DB" ] || die "backup DB not found: $DB"
+
+NAME="etv-migsmoke-$$"
+WORK="$(mktemp -d)"
+# shellcheck disable=SC2329  # invoked via 'trap cleanup EXIT'
+cleanup() {
+  docker rm -f "$NAME" >/dev/null 2>&1 || true
+  # The container runs as root and writes root-owned files into $WORK/config (cache, logs, search-index,
+  # data-protection), which a non-root invoker cannot delete. Remove them from inside a throwaway root
+  # container that mounts $WORK, then drop the (host-owned) temp dir — otherwise each run leaks the
+  # multi-hundred-MB DB copy + config.
+  if [ -n "${IMAGE:-}" ]; then
+    docker run --rm -v "$WORK:/work" --entrypoint /bin/sh "$IMAGE" -c 'rm -rf /work/config' >/dev/null 2>&1 || true
+  fi
+  rm -rf "$WORK" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+# Seed a throwaway config dir with a WRITABLE copy of the backup (single checkpointed file — we do NOT
+# copy any -wal/-shm; a backup snapshot is self-contained, and the container will create its own WAL).
+mkdir -p "$WORK/config"
+cp "$DB" "$WORK/config/ersatztv.sqlite3"
+chmod u+rw "$WORK/config/ersatztv.sqlite3"
+SIZE=$(du -h "$WORK/config/ersatztv.sqlite3" | cut -f1)
+echo "migration-smoke: image=$IMAGE  db-copy=${SIZE}  timeout=${TIMEOUT}s"
+
+docker pull "$IMAGE" >/dev/null 2>&1 || echo "migration-smoke: (pull failed/offline — using local image if present)"
+
+# Boot the new image against the prod-copy. DatabaseMigratorService (a BackgroundService) applies
+# pending migrations on startup; it logs "Applying database migrations" then "Done applying database
+# migrations", and on failure the host stops (default BackgroundServiceExceptionBehavior = StopHost),
+# so the container exits. We gate PASS on the "Done" line, FAIL on early exit / a migration exception.
+docker run -d --name "$NAME" --memory 2g \
+  -e ETV_CONFIG_FOLDER=/config \
+  -e ETV_TRANSCODE_FOLDER=/tmp/etv/transcode \
+  -v "$WORK/config:/config" \
+  "$IMAGE" >/dev/null || die "docker run failed for $IMAGE"
+
+DONE_RE='Done applying database migrations'
+FAIL_RE='error occurred while|Unhandled exception|SqliteException|BackgroundService failed|An exception occurred|Failed to migrate'
+
+mig_done=0
+deadline=$((SECONDS + TIMEOUT))
+while [ $SECONDS -lt $deadline ]; do
+  logs=$(docker logs "$NAME" 2>&1 || true)
+  if printf '%s' "$logs" | grep -qE "$DONE_RE"; then mig_done=1; break; fi
+  if printf '%s' "$logs" | grep -qiE "$FAIL_RE"; then
+    echo "migration-smoke: FAIL — migration error in logs:"; printf '%s\n' "$logs" | tail -n 40
+    exit 1
+  fi
+  if [ -z "$(docker ps -q --filter name="$NAME" --filter status=running)" ]; then
+    echo "migration-smoke: FAIL — container exited before finishing migrations:"; printf '%s\n' "$logs" | tail -n 40
+    exit 1
+  fi
+  sleep 2
+done
+
+if [ "$mig_done" != "1" ]; then
+  echo "migration-smoke: FAIL — migrations did not finish within ${TIMEOUT}s:"; docker logs "$NAME" 2>&1 | tail -n 40
+  exit 1
+fi
+echo "migration-smoke: migrations applied cleanly to the prod-copy."
+
+# Boot sanity: confirm the app then serves a DB-backed IPTV endpoint (proves DatabaseIsReady + the
+# app is healthy post-migration, not just that the migrator finished). The image ships python3.
+serve_ok=0
+for _ in $(seq 1 30); do
+  if docker exec "$NAME" python3 -c "import urllib.request,sys; urllib.request.urlopen('http://localhost:8409/iptv/channels.m3u',timeout=5)" >/dev/null 2>&1; then
+    serve_ok=1; break
+  fi
+  sleep 2
+done
+if [ "$serve_ok" != "1" ]; then
+  echo "migration-smoke: FAIL — migrations finished but the app did not serve /iptv/channels.m3u post-boot:"
+  docker logs "$NAME" 2>&1 | tail -n 40
+  exit 1
+fi
+
+echo "migration-smoke: PASS — $IMAGE migrated the prod-copy and booted healthy."
+exit 0