Release path: migration-on-prod-copy smoke (restore backup → apply migrations → boot) before promoting #315

Closed
opened 2026-07-12 20:38:02 +02:00 by timothy · 0 comments
Owner

Spun off from #303 (process-hardening tracker) as its second remaining non-hook follow-up. Data-plane release rigor.

Why

CI validates migrations against fresh, empty DBs (the #13 migrations job: model-drift + apply-to-fresh-DB per provider, SQLite + MySql). That proves a migration is well-formed, not that it applies cleanly to the accumulated prod SQLite — real data volume, WAL state, historical rows, and column values a fresh EnsureCreated schema never exercises. A migration that is green on a fresh DB can still fail (or silently corrupt) on the real prod database, and we only find out mid-deploy after the container recreates.

The Komodo pre-deploy hook already backs up the prod SQLite before recreating (server-management, media-servers stack). This issue adds a smoke that uses that backup as a rehearsal: restore the copy → apply pending EF migrations → boot ErsatzTV against it → health check, before promoting the new image.

Scope

  • A smoke step (script or CI job) that:
    1. takes the pre-deploy prod SQLite backup copy (never the live file),
    2. applies pending EF migrations against it,
    3. boots ErsatzTV pointed at the migrated copy,
    4. hits the health endpoint and asserts a clean start.
  • Decide where it runs: Komodo pre-deploy hook step (server-management), CI job, or documented release-checklist step.

Done-when

  • Smoke step implemented (script or job) doing restore-copy → migrate → boot → health-check — scripts/migration-smoke.sh (PR #321), gates on the Done applying database migrations log line + a post-boot /iptv/channels.m3u probe; validated live vs a copy of the 283 MB prod backup
  • Wired into the release/pre-deploy path (or documented as a mandatory release-checklist step), with the server-management side filed as a companion issue — documented in docs/ci-cd.md as a pre-deploy/release step; server-management#589 filed to wire it into the Komodo pre-deploy hook
  • Documented in docs/ci-cd.md release section — new "Migration-on-prod-copy smoke" subsection + docs/decisions.md entry
  • adversarial review passed — 2 independent cold passes (initial MERGEABLE-WITH-NITS → nits fixed → fix-commit re-review MERGEABLE @ 5c5a90af)

Notes

  • Operate only on a copy of the backup — never the live prod DB.
  • Lower urgency than the #197 security-scan sibling (not a go-live gate), but it protects prod data on every future release.
Spun off from #303 (process-hardening tracker) as its second remaining non-hook follow-up. Data-plane release rigor. ## Why CI validates migrations against **fresh, empty DBs** (the #13 `migrations` job: model-drift + apply-to-fresh-DB per provider, SQLite + MySql). That proves a migration is *well-formed*, not that it applies cleanly to the **accumulated prod SQLite** — real data volume, WAL state, historical rows, and column values a fresh `EnsureCreated` schema never exercises. A migration that is green on a fresh DB can still fail (or silently corrupt) on the real prod database, and we only find out mid-deploy after the container recreates. The Komodo pre-deploy hook already **backs up** the prod SQLite before recreating (server-management, `media-servers` stack). This issue adds a smoke that *uses* that backup as a rehearsal: restore the copy → apply pending EF migrations → boot ErsatzTV against it → health check, before promoting the new image. ## Scope - A smoke step (script or CI job) that: 1. takes the pre-deploy prod SQLite backup copy (never the live file), 2. applies pending EF migrations against it, 3. boots ErsatzTV pointed at the migrated copy, 4. hits the health endpoint and asserts a clean start. - Decide **where it runs**: Komodo pre-deploy hook step (server-management), CI job, or documented release-checklist step. ## Done-when - [x] Smoke step implemented (script or job) doing restore-copy → migrate → boot → health-check — `scripts/migration-smoke.sh` (PR #321), gates on the `Done applying database migrations` log line + a post-boot `/iptv/channels.m3u` probe; validated live vs a copy of the 283 MB prod backup - [x] Wired into the release/pre-deploy path (or documented as a mandatory release-checklist step), with the server-management side filed as a companion issue — documented in `docs/ci-cd.md` as a pre-deploy/release step; **server-management#589** filed to wire it into the Komodo pre-deploy hook - [x] Documented in `docs/ci-cd.md` release section — new "Migration-on-prod-copy smoke" subsection + `docs/decisions.md` entry - [x] adversarial review passed — 2 independent cold passes (initial MERGEABLE-WITH-NITS → nits fixed → fix-commit re-review MERGEABLE @ 5c5a90af) ## Notes - Operate only on a **copy** of the backup — never the live prod DB. - Lower urgency than the #197 security-scan sibling (not a go-live gate), but it protects prod data on every future release.
timothy added the enhancementci-cdpriority: medium labels 2026-07-12 20:38:02 +02:00
timothy added the in-progress label 2026-07-12 21:09:43 +02:00
timothy removed the in-progress label 2026-07-12 22:08:07 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#315