From 4da2b67ab24810f272081ffb479b81e0667ab286 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 12 Jul 2026 21:18:16 +0200 Subject: [PATCH 1/3] feat(ci): #315 migration-on-prod-copy smoke for the release path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's migrations job only proves a migration is well-formed against a fresh, empty DB. It never exercises the migration — or ErsatzTV's startup data steps (DatabaseMigratorService -> DbInitializer + PopulatePathHashes over the real MediaFile table) — against the accumulated prod SQLite, so a migration green on a fresh DB can still fail/corrupt on prod, found only mid-deploy. scripts/migration-smoke.sh rehearses it on a THROWAWAY copy of the latest prod backup: boots the new image against the copy, gates PASS on the "Done applying database migrations" log line (the migrator is a BackgroundService running concurrently with Kestrel, so HTTP readiness alone doesn't prove migrations finished), FAILs on early container exit / migration exception / timeout / not serving afterwards. Always operates on a copy, never the live DB; tears down its container + temp dir (incl. the container's root-owned config files) on exit. Validated live 2026-07-12: :latest vs a copy of the 283MB prod backup -> migrations applied cleanly, app booted+served, temp dir removed. Home split: this repo owns the script + docs; wiring it into the Komodo pre-deploy step is server-management#589 (cross-repo). Docs: docs/ci-cd.md (Migration integrity), docs/decisions.md (new entry, pure insertion). fixes #315 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/ci-cd.md | 36 ++++++++++ docs/decisions.md | 20 ++++++ scripts/migration-smoke.sh | 141 +++++++++++++++++++++++++++++++++++++ 3 files changed, 197 insertions(+) create mode 100755 scripts/migration-smoke.sh diff --git a/docs/ci-cd.md b/docs/ci-cd.md index 209f94cd5..2fb86c248 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -281,6 +281,42 @@ migration failure fails deterministically on every attempt, so the retry never m still flakes past the retry, re-trigger (Gitea has no rerun API on this version — push, or the run drains); don't treat a lone MySql-apply red as a code problem without checking the failure mode. +### Migration-on-prod-copy smoke — release path (`scripts/migration-smoke.sh`, ersatztv#315) + +The `migrations` job above only proves a migration is **well-formed against a fresh, empty DB**. It +can't prove it applies cleanly to the **accumulated prod SQLite** — real row volume, historical values, +and the post-migration data steps ErsatzTV runs on startup: `DatabaseMigratorService` (a +`BackgroundService`) applies pending migrations, then `DbInitializer.Initialize` + `PopulatePathHashes` +(an `UPDATE` over the real `MediaFile` table). A migration green on a fresh DB can still fail or corrupt +on prod, and today you'd only find out mid-deploy after the container recreates. + +`scripts/migration-smoke.sh` rehearses it on a **throwaway copy** of the latest prod backup — it never +touches the live DB: + +``` +scripts/migration-smoke.sh --image [--db ] [--timeout 180] +``` + +It copies the backup into a temp config dir, boots the new image against it (`ETV_CONFIG_FOLDER`), and +**gates PASS on the `Done applying database migrations` log line** — not merely on HTTP readiness, since +the migrator runs concurrently with Kestrel, so the web server can serve before/while migrations run. +FAIL = the container exits before finishing, a migration exception appears in the logs, migrations +don't finish within `--timeout`, or the app won't serve `/iptv/channels.m3u` afterwards. The smoke +container, the DB copy, and the temp dir are always torn down on exit (the ErsatzTV image runs as root, +so cleanup deletes its root-owned config files from inside a throwaway root container — otherwise each +run would leak the multi-hundred-MB copy). Exit 0 = clean, 1 = migration/boot failure, 2 = usage error. + +- **`--db` default**: the newest `ersatztv.sqlite3` under `$ETV_BACKUP_DIR` + (default `~/downloadswarm/ersatztv-backups` — where the host-side pre-deploy backup hook writes + timestamped snapshots). Pass `--image` = the version tag about to be promoted. +- **Where it runs**: it's meant to run **on the docker host** as a **Komodo pre-deploy step** (which + already produces the backup — see "Cutting a release" and the #553 pre-deploy backup caveat), so a + bad migration aborts the promote before the live container recreates. Wiring it into that hook is a + server-management concern (cross-repo — this repo owns the script + docs, server-management owns the + Komodo hook). Until wired, run it by hand before cutting a migration-bearing release. +- Validated live 2026-07-12: `:latest` against a copy of the 283 MB prod backup → migrations applied + cleanly, app booted and served, temp dir removed. + ## Pre-commit hooks (web/) The repo uses **husky** git hooks (installed via `web/`'s **lint-staged** + npm) to catch diff --git a/docs/decisions.md b/docs/decisions.md index f248b6a20..1a595aee3 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -79,6 +79,7 @@ keep append-only from accreting stale, contradictory, or unreadably-large histor - [2026-07-12 — Live-E2E is a required step for API write-path handler changes (#303)](#2026-07-12--live-e2e-is-a-required-step-for-api-write-path-handler-changes-303) - [2026-07-12 — Formatting-as-you-touch, enforced; rebase-not-merge for PR branches (#311 H11 + format CI)](#2026-07-12--formatting-as-you-touch-enforced-rebase-not-merge-for-pr-branches-311-h11--format-ci) - [2026-07-12 — Merge-consent gate auto-grants when satisfied (no redundant prompt); state IS the consent (#314)](#2026-07-12--merge-consent-gate-auto-grants-when-satisfied-no-redundant-prompt-state-is-the-consent-314) +- [2026-07-12 — Release path rehearses migrations on a prod-DB copy before promoting (#315)](#2026-07-12--release-path-rehearses-migrations-on-a-prod-db-copy-before-promoting-315) --- @@ -1554,3 +1555,22 @@ no-creds→ask, non-merge→passthrough). also ask conversationally to merge a PR whose gate auto-grants. A separate human confirmation is still warranted only when the gate **asks** (state not derivable). This supersedes the "always confirm merge consent in-conversation per session" phrasing in the kickoff HARD CONSTRAINTS (updated in the same PR). + +## 2026-07-12 — Release path rehearses migrations on a prod-DB copy before promoting (#315) + +The CI `migrations` job proves a migration is well-formed against a **fresh, empty** DB (model-drift + +apply-to-fresh, per provider). That is necessary but not sufficient: it never exercises the migration — +or ErsatzTV's startup data steps (`DatabaseMigratorService` → `DbInitializer` + `PopulatePathHashes` +over the real `MediaFile` table) — against the **accumulated prod SQLite**, where row volume and +historical values differ. A migration green on a fresh DB can still fail or corrupt on prod, discovered +only mid-deploy after the container recreates. + +Decision: before promoting a migration-bearing release, **rehearse** the new image's migrations against +a **throwaway copy of the latest prod backup** via `scripts/migration-smoke.sh` — boot the new image +against the copy, gate PASS on the `Done applying database migrations` log line (the migrator is a +`BackgroundService` running concurrently with Kestrel, so HTTP-readiness alone does *not* prove +migrations finished), FAIL on early container exit / a migration exception / timeout / not serving +afterwards. Always operates on a copy, never the live DB. Home: the script + docs are ours; wiring it +into the Komodo **pre-deploy** step (which already produces the backup) is a server-management concern. +Rationale: data-plane rigor — catch a bad migration on a disposable copy, not on live prod data. +See `docs/ci-cd.md` → Migration-on-prod-copy smoke. Cross-repo wiring tracked in server-management. diff --git a/scripts/migration-smoke.sh b/scripts/migration-smoke.sh new file mode 100755 index 000000000..45f7129e7 --- /dev/null +++ b/scripts/migration-smoke.sh @@ -0,0 +1,141 @@ +#!/usr/bin/env bash +# scripts/migration-smoke.sh — rehearse a release's EF migrations against a COPY of the real prod +# database before promoting the new image (ersatztv#315). Data-plane release rigor. +# +# WHY: CI's `migrations` job only proves a migration is well-formed against a FRESH, empty DB +# (model-drift + apply-to-fresh per provider). It cannot prove the migration applies cleanly to the +# accumulated prod SQLite — real row volume, historical values, and the post-migration data steps +# ErsatzTV runs on startup (DatabaseMigratorService → DbInitializer + PopulatePathHashes over the real +# MediaFile table). A migration green on a fresh DB can still fail or corrupt on prod. This smoke boots +# the NEW image against a THROWAWAY copy of the latest prod backup, waits for the migrator to finish, +# and reports pass/fail — so a bad migration is caught on a copy, not mid-deploy on live data. +# +# It is designed to run ON the docker host (bumblebee) as a Komodo pre-deploy step (server-management +# owns that wiring — see ersatztv#315), or by hand before a release. It NEVER touches the live prod DB. +# +# Usage: +# scripts/migration-smoke.sh --image [--db ] [--timeout ] +# +# --image Image to rehearse — the one about to be PROMOTED (e.g. +# 192.168.1.95:3000/timothy/ersatztv:26.6.0). Required. +# --db A prod SQLite backup file to copy and migrate. Default: the newest +# $ETV_BACKUP_DIR/*/ersatztv.sqlite3 (ETV_BACKUP_DIR defaults to +# ~/downloadswarm/ersatztv-backups — where the Komodo pre-deploy hook writes them). +# --timeout Seconds to wait for migrations to finish (default 180). Big prod DBs + the path-hash +# backfill can take a while on first upgrade. +# +# Exit 0 = the new image applied all pending migrations to the prod-copy and booted cleanly. +# Exit 1 = a migration failed / the container exited before finishing / it never booted (logs dumped). +# Exit 2 = usage / precondition error (no image, no backup found, docker missing). +# +# The container, the temp config dir, and the DB copy are always cleaned up on exit. + +set -euo pipefail + +IMAGE="" +DB="" +TIMEOUT=180 +BACKUP_DIR="${ETV_BACKUP_DIR:-$HOME/downloadswarm/ersatztv-backups}" + +die() { echo "migration-smoke: $*" >&2; exit 2; } + +while [ $# -gt 0 ]; do + case "$1" in + --image) IMAGE="${2:-}"; shift 2 ;; + --db) DB="${2:-}"; shift 2 ;; + --timeout) TIMEOUT="${2:-}"; shift 2 ;; + -h|--help) sed -n '2,40p' "$0"; exit 0 ;; + *) die "unknown argument: $1 (see --help)" ;; + esac +done + +command -v docker >/dev/null 2>&1 || die "docker not found on PATH (run this on the docker host)" +[ -n "$IMAGE" ] || die "--image is required (the image about to be promoted)" + +# Resolve the backup DB: explicit --db, else newest ersatztv.sqlite3 under $BACKUP_DIR. +if [ -z "$DB" ]; then + DB=$(find "$BACKUP_DIR" -type f -name 'ersatztv.sqlite3' -printf '%T@ %p\n' 2>/dev/null \ + | sort -rn | head -1 | cut -d' ' -f2-) + [ -n "$DB" ] || die "no ersatztv.sqlite3 backup found under $BACKUP_DIR (pass --db explicitly)" + echo "migration-smoke: using newest backup: $DB" +fi +[ -f "$DB" ] || die "backup DB not found: $DB" + +NAME="etv-migsmoke-$$" +WORK="$(mktemp -d)" +# shellcheck disable=SC2329 # invoked via 'trap cleanup EXIT' +cleanup() { + docker rm -f "$NAME" >/dev/null 2>&1 || true + # The container runs as root and writes root-owned files into $WORK/config (cache, logs, search-index, + # data-protection), which a non-root invoker cannot delete. Remove them from inside a throwaway root + # container that mounts $WORK, then drop the (host-owned) temp dir — otherwise each run leaks the + # multi-hundred-MB DB copy + config. + if [ -n "${IMAGE:-}" ]; then + docker run --rm -v "$WORK:/work" --entrypoint /bin/sh "$IMAGE" -c 'rm -rf /work/config' >/dev/null 2>&1 || true + fi + rm -rf "$WORK" 2>/dev/null || true +} +trap cleanup EXIT + +# Seed a throwaway config dir with a WRITABLE copy of the backup (single checkpointed file — we do NOT +# copy any -wal/-shm; a backup snapshot is self-contained, and the container will create its own WAL). +mkdir -p "$WORK/config" +cp "$DB" "$WORK/config/ersatztv.sqlite3" +chmod u+rw "$WORK/config/ersatztv.sqlite3" +SIZE=$(du -h "$WORK/config/ersatztv.sqlite3" | cut -f1) +echo "migration-smoke: image=$IMAGE db-copy=${SIZE} timeout=${TIMEOUT}s" + +docker pull "$IMAGE" >/dev/null 2>&1 || echo "migration-smoke: (pull failed/offline — using local image if present)" + +# Boot the new image against the prod-copy. DatabaseMigratorService (a BackgroundService) applies +# pending migrations on startup; it logs "Applying database migrations" then "Done applying database +# migrations", and on failure the host stops (default BackgroundServiceExceptionBehavior = StopHost), +# so the container exits. We gate PASS on the "Done" line, FAIL on early exit / a migration exception. +docker run -d --name "$NAME" --memory 2g \ + -e ETV_CONFIG_FOLDER=/config \ + -e ETV_TRANSCODE_FOLDER=/tmp/etv/transcode \ + -v "$WORK/config:/config" \ + "$IMAGE" >/dev/null || die "docker run failed for $IMAGE" + +DONE_RE='Done applying database migrations' +FAIL_RE='error occurred while|Unhandled exception|SqliteException|BackgroundService failed|An exception occurred|Failed to migrate' + +mig_done=0 +deadline=$((SECONDS + TIMEOUT)) +while [ $SECONDS -lt $deadline ]; do + logs=$(docker logs "$NAME" 2>&1 || true) + if printf '%s' "$logs" | grep -qE "$DONE_RE"; then mig_done=1; break; fi + if printf '%s' "$logs" | grep -qiE "$FAIL_RE"; then + echo "migration-smoke: FAIL — migration error in logs:"; printf '%s\n' "$logs" | tail -n 40 + exit 1 + fi + if [ -z "$(docker ps -q --filter name="$NAME" --filter status=running)" ]; then + echo "migration-smoke: FAIL — container exited before finishing migrations:"; printf '%s\n' "$logs" | tail -n 40 + exit 1 + fi + sleep 2 +done + +if [ "$mig_done" != "1" ]; then + echo "migration-smoke: FAIL — migrations did not finish within ${TIMEOUT}s:"; docker logs "$NAME" 2>&1 | tail -n 40 + exit 1 +fi +echo "migration-smoke: migrations applied cleanly to the prod-copy." + +# Boot sanity: confirm the app then serves a DB-backed IPTV endpoint (proves DatabaseIsReady + the +# app is healthy post-migration, not just that the migrator finished). The image ships python3. +serve_ok=0 +for _ in $(seq 1 30); do + if docker exec "$NAME" python3 -c "import urllib.request,sys; urllib.request.urlopen('http://localhost:8409/iptv/channels.m3u',timeout=5)" >/dev/null 2>&1; then + serve_ok=1; break + fi + sleep 2 +done +if [ "$serve_ok" != "1" ]; then + echo "migration-smoke: FAIL — migrations finished but the app did not serve /iptv/channels.m3u post-boot:" + docker logs "$NAME" 2>&1 | tail -n 40 + exit 1 +fi + +echo "migration-smoke: PASS — $IMAGE migrated the prod-copy and booted healthy." +exit 0 -- 2.47.3 From 5c5a90afde94e45ccdd4e4c21be8db3c8ff09f71 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 12 Jul 2026 21:23:45 +0200 Subject: [PATCH 2/3] =?UTF-8?q?fix(ci):=20#315=20address=20review=20nits?= =?UTF-8?q?=20=E2=80=94=20drop=20fragile=20FAIL=5FRE,=20validate=20--timeo?= =?UTF-8?q?ut,=20log=20image=20id?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial review (MERGEABLE-WITH-NITS) findings: - Remove the broad FAIL_RE log-scan (matched benign ErsatzTV startup noise — library scans against absent media mounts, EF connection retries — risking a false-FAIL that blocks a good release). It was also redundant: a failed migration faults the BackgroundService -> default StopHost -> container exit, which the early-exit check already catches reliably (per the reviewer's own analysis). Migration failure is now detected by early container exit + timeout + the post-boot serve probe. - Validate --timeout is a positive integer (was: '--timeout abc' -> 0 -> instant false-FAIL). - Log the resolved image id after (attempted) pull, so a pull-failure that rehearses a stale local :latest is visible to the operator. Re-validated live on bumblebee: :latest vs the 283MB prod-copy -> migrations clean, PASS, image digest logged, no leftover temp dir/container. shellcheck + bash -n clean. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/migration-smoke.sh | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/scripts/migration-smoke.sh b/scripts/migration-smoke.sh index 45f7129e7..bc12a9478 100755 --- a/scripts/migration-smoke.sh +++ b/scripts/migration-smoke.sh @@ -51,6 +51,8 @@ done command -v docker >/dev/null 2>&1 || die "docker not found on PATH (run this on the docker host)" [ -n "$IMAGE" ] || die "--image is required (the image about to be promoted)" +case "$TIMEOUT" in ''|*[!0-9]*) die "--timeout must be a positive integer of seconds (got: '$TIMEOUT')";; esac +[ "$TIMEOUT" -gt 0 ] || die "--timeout must be greater than 0" # Resolve the backup DB: explicit --db, else newest ersatztv.sqlite3 under $BACKUP_DIR. if [ -z "$DB" ]; then @@ -85,7 +87,13 @@ chmod u+rw "$WORK/config/ersatztv.sqlite3" SIZE=$(du -h "$WORK/config/ersatztv.sqlite3" | cut -f1) echo "migration-smoke: image=$IMAGE db-copy=${SIZE} timeout=${TIMEOUT}s" -docker pull "$IMAGE" >/dev/null 2>&1 || echo "migration-smoke: (pull failed/offline — using local image if present)" +if ! docker pull "$IMAGE" >/dev/null 2>&1; then + # Not fatal (offline / locally-built images), but for a mutable tag (:latest) a stale local copy + # would rehearse the wrong image — so we log the resolved image ID below for the operator to verify. + echo "migration-smoke: WARNING — pull failed; rehearsing whatever '$IMAGE' resolves to locally" +fi +IMG_ID=$(docker image inspect "$IMAGE" --format '{{.Id}}' 2>/dev/null || echo "unknown") +echo "migration-smoke: rehearsing $IMAGE ($IMG_ID)" # Boot the new image against the prod-copy. DatabaseMigratorService (a BackgroundService) applies # pending migrations on startup; it logs "Applying database migrations" then "Done applying database @@ -97,20 +105,19 @@ docker run -d --name "$NAME" --memory 2g \ -v "$WORK/config:/config" \ "$IMAGE" >/dev/null || die "docker run failed for $IMAGE" +# PASS gate = the migrator's completion log line. FAIL = the container exiting before it (a failed +# migration faults the BackgroundService → default StopHost → the host stops → the container exits, so +# early-exit IS the reliable migration-failure signal), or the timeout below. We deliberately do NOT +# grep logs for error strings: ErsatzTV's startup is noisy (library scans against absent media mounts, +# EF connection retries) and a broad error-regex would false-FAIL a good migration. DONE_RE='Done applying database migrations' -FAIL_RE='error occurred while|Unhandled exception|SqliteException|BackgroundService failed|An exception occurred|Failed to migrate' - mig_done=0 deadline=$((SECONDS + TIMEOUT)) while [ $SECONDS -lt $deadline ]; do - logs=$(docker logs "$NAME" 2>&1 || true) - if printf '%s' "$logs" | grep -qE "$DONE_RE"; then mig_done=1; break; fi - if printf '%s' "$logs" | grep -qiE "$FAIL_RE"; then - echo "migration-smoke: FAIL — migration error in logs:"; printf '%s\n' "$logs" | tail -n 40 - exit 1 - fi + if docker logs "$NAME" 2>&1 | grep -qE "$DONE_RE"; then mig_done=1; break; fi if [ -z "$(docker ps -q --filter name="$NAME" --filter status=running)" ]; then - echo "migration-smoke: FAIL — container exited before finishing migrations:"; printf '%s\n' "$logs" | tail -n 40 + echo "migration-smoke: FAIL — container exited before finishing migrations (a failed migration stops the host):" + docker logs "$NAME" 2>&1 | tail -n 40 exit 1 fi sleep 2 -- 2.47.3 From 9dc48118d190c9b70b55e62b0a99c5a7559fc7ff Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 12 Jul 2026 21:33:43 +0200 Subject: [PATCH 3/3] chore: re-trigger CI (api-docs runner checkout-cache flake on 5c5a90af; identical tree) Co-Authored-By: Claude Opus 4.8 (1M context) -- 2.47.3