CI's migrations job proves a migration is well-formed against a fresh, empty DB only. It never exercises the migration — or ErsatzTV's startup data steps (DatabaseMigratorService → DbInitializer + PopulatePathHashes over the real MediaFile table) — against the accumulated prod SQLite. A migration green on a fresh DB can still fail/corrupt on prod, found only mid-deploy after the container recreates.
What
scripts/migration-smoke.sh --image <ref> [--db <backup>] [--timeout] rehearses it on a throwaway copy of the latest prod backup (auto-picks newest ~/downloadswarm/ersatztv-backups/*/ersatztv.sqlite3):
boots the new image against the copy (ETV_CONFIG_FOLDER),
gates PASS on the Done applying database migrations log line — not HTTP readiness, because the migrator is a BackgroundService running concurrently with Kestrel, so the web server can serve before/while migrations run,
FAILs on early container exit / a migration exception in logs / timeout / not serving /iptv/channels.m3u afterwards,
always operates on a copy, never the live DB; tears down the container + temp dir on exit (deletes the container's root-owned config files via a throwaway root container — else each run leaks the multi-hundred-MB copy).
Ran twice on bumblebee: :latest vs a copy of the 283 MB prod backup → migrations applied cleanly … PASS, app served /iptv/channels.m3u, and (after the teardown fix) no leftover temp dir / container. bash -n + shellcheck clean.
Home split (cross-repo)
This repo owns the script + docs; wiring it into the Komodo pre-deploy step (which produces the backup) is server-management#589. Until wired, run by hand before a migration-bearing release (documented in docs/ci-cd.md).
Docs
docs/ci-cd.md → new "Migration-on-prod-copy smoke" subsection; docs/decisions.md → new entry (pure insertion, append-only-safe).
Closes #315.
## Why
CI's `migrations` job proves a migration is well-formed against a **fresh, empty** DB only. It never exercises the migration — or ErsatzTV's startup data steps (`DatabaseMigratorService` → `DbInitializer` + `PopulatePathHashes` over the real `MediaFile` table) — against the **accumulated prod SQLite**. A migration green on a fresh DB can still fail/corrupt on prod, found only mid-deploy after the container recreates.
## What
`scripts/migration-smoke.sh --image <ref> [--db <backup>] [--timeout]` rehearses it on a **throwaway copy** of the latest prod backup (auto-picks newest `~/downloadswarm/ersatztv-backups/*/ersatztv.sqlite3`):
- boots the new image against the copy (`ETV_CONFIG_FOLDER`),
- **gates PASS on the `Done applying database migrations` log line** — not HTTP readiness, because the migrator is a `BackgroundService` running concurrently with Kestrel, so the web server can serve before/while migrations run,
- FAILs on early container exit / a migration exception in logs / timeout / not serving `/iptv/channels.m3u` afterwards,
- always operates on a **copy**, never the live DB; tears down the container + temp dir on exit (deletes the container's root-owned config files via a throwaway root container — else each run leaks the multi-hundred-MB copy).
- Exit 0/1/2 = clean / migration-boot failure / usage error.
## Validation (live)
Ran twice on bumblebee: `:latest` vs a copy of the **283 MB** prod backup → `migrations applied cleanly … PASS`, app served `/iptv/channels.m3u`, and (after the teardown fix) **no leftover temp dir / container**. `bash -n` + shellcheck clean.
## Home split (cross-repo)
This repo owns the script + docs; wiring it into the **Komodo pre-deploy step** (which produces the backup) is **server-management#589**. Until wired, run by hand before a migration-bearing release (documented in `docs/ci-cd.md`).
## Docs
`docs/ci-cd.md` → new "Migration-on-prod-copy smoke" subsection; `docs/decisions.md` → new entry (pure insertion, append-only-safe).
CI's migrations job only proves a migration is well-formed against a fresh,
empty DB. It never exercises the migration — or ErsatzTV's startup data steps
(DatabaseMigratorService -> DbInitializer + PopulatePathHashes over the real
MediaFile table) — against the accumulated prod SQLite, so a migration green on
a fresh DB can still fail/corrupt on prod, found only mid-deploy.
scripts/migration-smoke.sh rehearses it on a THROWAWAY copy of the latest prod
backup: boots the new image against the copy, gates PASS on the "Done applying
database migrations" log line (the migrator is a BackgroundService running
concurrently with Kestrel, so HTTP readiness alone doesn't prove migrations
finished), FAILs on early container exit / migration exception / timeout / not
serving afterwards. Always operates on a copy, never the live DB; tears down its
container + temp dir (incl. the container's root-owned config files) on exit.
Validated live 2026-07-12: :latest vs a copy of the 283MB prod backup ->
migrations applied cleanly, app booted+served, temp dir removed.
Home split: this repo owns the script + docs; wiring it into the Komodo
pre-deploy step is server-management#589 (cross-repo). Docs: docs/ci-cd.md
(Migration integrity), docs/decisions.md (new entry, pure insertion).
fixes#315
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review (MERGEABLE-WITH-NITS) findings:
- Remove the broad FAIL_RE log-scan (matched benign ErsatzTV startup noise — library
scans against absent media mounts, EF connection retries — risking a false-FAIL that
blocks a good release). It was also redundant: a failed migration faults the
BackgroundService -> default StopHost -> container exit, which the early-exit check
already catches reliably (per the reviewer's own analysis). Migration failure is now
detected by early container exit + timeout + the post-boot serve probe.
- Validate --timeout is a positive integer (was: '--timeout abc' -> 0 -> instant false-FAIL).
- Log the resolved image id after (attempted) pull, so a pull-failure that rehearses a
stale local :latest is visible to the operator.
Re-validated live on bumblebee: :latest vs the 283MB prod-copy -> migrations clean, PASS,
image digest logged, no leftover temp dir/container. shellcheck + bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review (2 independent cold passes) + CI flake note
Pass 1 — initial script (4da2b67a): MERGEABLE-WITH-NITS. Confirmed the safety-critical properties: never touches the live prod DB (only ever copies a backup file into a mktemp dir; only that dir is mounted), no confirmed false-PASS (the serve-probe backstops the post-Done window), complete teardown. Nits: over-broad FAIL_RE (benign-startup-noise false-FAIL risk), unvalidated --timeout, stale-local-image transparency.
Fix (5c5a90af): dropped the fragile FAIL_RE log-scan (a failed migration faults the BackgroundService → StopHost → container exit, already caught by the early-exit check — so this removes a false-FAIL without adding a false-PASS), validated --timeout, and now log the resolved image id.
Pass 2 — re-review of the fix commit (5c5a90af): MERGEABLE. Confirmed removing FAIL_RE opens no false-PASS (the Done applying database migrations gate is emitted only on genuine migrator completion; every failure mode is still caught by early-exit / timeout / the post-boot serve probe), the --timeout guard is correct and well-placed, and the live-DB safety + teardown are unchanged.
Live-validated twice on bumblebee (:latest vs a copy of the 283 MB prod backup → migrations clean, PASS, clean teardown). bash -n + shellcheck clean.
CI note: the api-docs check went red on 5c5a90af due to a runner infrastructure flake, not this diff — the job died in actions/checkout@v4 itself (Cannot find module '/var/run/act/actions/.../dist/index.js', corrupted act_runner action cache), before any step evaluated my content. This PR touches no API surface (ErsatzTV/Controllers/Api/ or ErsatzTV.Core/Api/), so the api-docs job path-gates to a trivial skip; the prior head 4da2b67a passed it. Re-running on the same sha.
## Adversarial review (2 independent cold passes) + CI flake note
**Pass 1 — initial script (`4da2b67a`)**: MERGEABLE-WITH-NITS. Confirmed the safety-critical properties: never touches the live prod DB (only ever copies a *backup* file into a `mktemp` dir; only that dir is mounted), no confirmed false-PASS (the serve-probe backstops the post-`Done` window), complete teardown. Nits: over-broad `FAIL_RE` (benign-startup-noise false-FAIL risk), unvalidated `--timeout`, stale-local-image transparency.
**Fix (`5c5a90af`)**: dropped the fragile `FAIL_RE` log-scan (a failed migration faults the `BackgroundService` → StopHost → container exit, already caught by the early-exit check — so this removes a false-FAIL without adding a false-PASS), validated `--timeout`, and now log the resolved image id.
**Pass 2 — re-review of the fix commit (`5c5a90af`)**: MERGEABLE. Confirmed removing `FAIL_RE` opens **no** false-PASS (the `Done applying database migrations` gate is emitted only on genuine migrator completion; every failure mode is still caught by early-exit / timeout / the post-boot serve probe), the `--timeout` guard is correct and well-placed, and the live-DB safety + teardown are unchanged.
Live-validated twice on bumblebee (`:latest` vs a copy of the 283 MB prod backup → migrations clean, PASS, clean teardown). `bash -n` + shellcheck clean.
**CI note:** the `api-docs` check went red on `5c5a90af` due to a **runner infrastructure flake**, not this diff — the job died in `actions/checkout@v4` itself (`Cannot find module '/var/run/act/actions/.../dist/index.js'`, corrupted act_runner action cache), before any step evaluated my content. This PR touches no API surface (`ErsatzTV/Controllers/Api/` or `ErsatzTV.Core/Api/`), so the api-docs job path-gates to a trivial skip; the prior head `4da2b67a` passed it. Re-running on the same sha.
Review-verdict: MERGEABLE @ 5c5a90af
Re-triggered CI with an empty commit9dc48118 (tree byte-identical to the reviewed 5c5a90af — verified git rev-parse HEAD^{tree} matches). No code change, so the fix-commit re-review carries over unchanged.
On 5c5a90af every substantive check passed — Build & test ✓, EF migration integrity ✓, Formatting ✓, decisions-guard ✓ — and the lone red api-docs was a runner checkout-cache flake (job died in actions/checkout@v4, Cannot find module …/dist/index.js, before the detect step). #321 changes no API surface, so api-docs path-gates to a trivial skip; the separate latent build-before-regen bug in that gate (being fixed elsewhere on docker-build.yml) does not affect this PR.
Re-triggered CI with an **empty commit** `9dc48118` (tree byte-identical to the reviewed `5c5a90af` — verified `git rev-parse HEAD^{tree}` matches). No code change, so the fix-commit re-review carries over unchanged.
On `5c5a90af` every substantive check passed — **Build & test ✓, EF migration integrity ✓, Formatting ✓, decisions-guard ✓** — and the lone red `api-docs` was a runner **checkout-cache flake** (job died in `actions/checkout@v4`, `Cannot find module …/dist/index.js`, before the detect step). #321 changes no API surface, so api-docs path-gates to a trivial skip; the separate latent build-before-regen bug in that gate (being fixed elsewhere on `docker-build.yml`) does not affect this PR.
Review-verdict: MERGEABLE @ 9dc48118
timothy
merged commit 9dc48118d1 into main2026-07-12 21:47:43 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #315.
Why
CI's
migrationsjob proves a migration is well-formed against a fresh, empty DB only. It never exercises the migration — or ErsatzTV's startup data steps (DatabaseMigratorService→DbInitializer+PopulatePathHashesover the realMediaFiletable) — against the accumulated prod SQLite. A migration green on a fresh DB can still fail/corrupt on prod, found only mid-deploy after the container recreates.What
scripts/migration-smoke.sh --image <ref> [--db <backup>] [--timeout]rehearses it on a throwaway copy of the latest prod backup (auto-picks newest~/downloadswarm/ersatztv-backups/*/ersatztv.sqlite3):ETV_CONFIG_FOLDER),Done applying database migrationslog line — not HTTP readiness, because the migrator is aBackgroundServicerunning concurrently with Kestrel, so the web server can serve before/while migrations run,/iptv/channels.m3uafterwards,Validation (live)
Ran twice on bumblebee:
:latestvs a copy of the 283 MB prod backup →migrations applied cleanly … PASS, app served/iptv/channels.m3u, and (after the teardown fix) no leftover temp dir / container.bash -n+ shellcheck clean.Home split (cross-repo)
This repo owns the script + docs; wiring it into the Komodo pre-deploy step (which produces the backup) is server-management#589. Until wired, run by hand before a migration-bearing release (documented in
docs/ci-cd.md).Docs
docs/ci-cd.md→ new "Migration-on-prod-copy smoke" subsection;docs/decisions.md→ new entry (pure insertion, append-only-safe).Adversarial review (2 independent cold passes) + CI flake note
Pass 1 — initial script (
4da2b67a): MERGEABLE-WITH-NITS. Confirmed the safety-critical properties: never touches the live prod DB (only ever copies a backup file into amktempdir; only that dir is mounted), no confirmed false-PASS (the serve-probe backstops the post-Donewindow), complete teardown. Nits: over-broadFAIL_RE(benign-startup-noise false-FAIL risk), unvalidated--timeout, stale-local-image transparency.Fix (
5c5a90af): dropped the fragileFAIL_RElog-scan (a failed migration faults theBackgroundService→ StopHost → container exit, already caught by the early-exit check — so this removes a false-FAIL without adding a false-PASS), validated--timeout, and now log the resolved image id.Pass 2 — re-review of the fix commit (
5c5a90af): MERGEABLE. Confirmed removingFAIL_REopens no false-PASS (theDone applying database migrationsgate is emitted only on genuine migrator completion; every failure mode is still caught by early-exit / timeout / the post-boot serve probe), the--timeoutguard is correct and well-placed, and the live-DB safety + teardown are unchanged.Live-validated twice on bumblebee (
:latestvs a copy of the 283 MB prod backup → migrations clean, PASS, clean teardown).bash -n+ shellcheck clean.CI note: the
api-docscheck went red on5c5a90afdue to a runner infrastructure flake, not this diff — the job died inactions/checkout@v4itself (Cannot find module '/var/run/act/actions/.../dist/index.js', corrupted act_runner action cache), before any step evaluated my content. This PR touches no API surface (ErsatzTV/Controllers/Api/orErsatzTV.Core/Api/), so the api-docs job path-gates to a trivial skip; the prior head4da2b67apassed it. Re-running on the same sha.Review-verdict: MERGEABLE @
5c5a90afRe-triggered CI with an empty commit
9dc48118(tree byte-identical to the reviewed5c5a90af— verifiedgit rev-parse HEAD^{tree}matches). No code change, so the fix-commit re-review carries over unchanged.On
5c5a90afevery substantive check passed — Build & test ✓, EF migration integrity ✓, Formatting ✓, decisions-guard ✓ — and the lone redapi-docswas a runner checkout-cache flake (job died inactions/checkout@v4,Cannot find module …/dist/index.js, before the detect step). #321 changes no API surface, so api-docs path-gates to a trivial skip; the separate latent build-before-regen bug in that gate (being fixed elsewhere ondocker-build.yml) does not affect this PR.Review-verdict: MERGEABLE @
9dc48118