ci: make the MySql migration-apply resilient to concurrent-runner contention #294

Merged
timothy merged 1 commits from ci/resilient-mysql-migration into main 2026-07-12 00:51:03 +02:00
Owner

Root cause

The EF migration integrity job's MySql apply-all-migrations-to-a-fresh-DB step flakes when two migration jobs land on the same runner host at once (common when several PRs push together). Each job spins its own services: mysql:8.4 container, and the paired MySQL servers starve each other replaying the 787 migrations. Diagnosed from run logs tonight:

Run Runner MySql apply outcome
513 ci-runner ❌ The Command Timeout expired (>30s command under load)
516 bumblebee-runner ❌ MySqlEndOfStreamException (connection dropped mid-replay)
515 bumblebee-runner ✅ passed (ran concurrently with 516 — one host, two MySQLs, one starved)

It is pure infra contention, not a model/code problem: has-pending-model-changes (the actual model check) passes for both providers, and the identical tree passes on a quieter host. Both runners have both passed and failed, so it isn't one bad runner.

Fix (runner-agnostic; repo-owned workflow only)

  • DefaultCommandTimeout=300 on the MySql connection string (up from MySqlConnector's 30s default) — survives a slow DDL command under load.
  • 3× retry around the apply that resumes from __EFMigrationsHistory — EF commits each migration in its own transaction, so an interrupted one rolls back cleanly and the retry continues where it left off. A genuine migration failure fails deterministically on every attempt, so the retry can never mask a real problem.

No runner-host change is needed (that would be server-management); this hardens the job itself, de-flaking it for every in-flight PR.

Verified

  • YAML parses; the retry block passes bash -n.
  • This PR's own EF migration integrity job exercises the hardened step.

Docs: docs/ci-cd.md migration-integrity section documents the contention + retry.

Refs #13 #236

🤖 Generated with Claude Code

## Root cause The `EF migration integrity` job's **MySql apply-all-migrations-to-a-fresh-DB** step flakes when two migration jobs land on the **same runner host at once** (common when several PRs push together). Each job spins its own `services: mysql:8.4` container, and the paired MySQL servers starve each other replaying the 787 migrations. Diagnosed from run logs tonight: | Run | Runner | MySql apply outcome | |-----|--------|--------------------| | 513 | ci-runner | ❌ `The Command Timeout expired` (>30s command under load) | | 516 | bumblebee-runner | ❌ `MySqlEndOfStreamException` (connection dropped mid-replay) | | 515 | bumblebee-runner | ✅ passed (ran concurrently with 516 — one host, two MySQLs, one starved) | It is **pure infra contention, not a model/code problem**: `has-pending-model-changes` (the actual model check) passes for both providers, and the *identical tree* passes on a quieter host. Both runners have both passed and failed, so it isn't one bad runner. ## Fix (runner-agnostic; repo-owned workflow only) - **`DefaultCommandTimeout=300`** on the MySql connection string (up from MySqlConnector's 30s default) — survives a slow DDL command under load. - **3× retry** around the apply that **resumes from `__EFMigrationsHistory`** — EF commits each migration in its own transaction, so an interrupted one rolls back cleanly and the retry continues where it left off. A *genuine* migration failure fails deterministically on every attempt, so the retry can never mask a real problem. No runner-host change is needed (that would be server-management); this hardens the job itself, de-flaking it for **every** in-flight PR. ## Verified - YAML parses; the retry block passes `bash -n`. - This PR's own `EF migration integrity` job exercises the hardened step. Docs: `docs/ci-cd.md` migration-integrity section documents the contention + retry. Refs #13 #236 🤖 Generated with [Claude Code](https://claude.com/claude-code)
timothy added 1 commit 2026-07-12 00:43:05 +02:00
ci: make the MySql migration-apply resilient to concurrent-runner contention
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 8s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 4m14s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 7m12s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Build ErsatzTV Image / Docs update reminder (push) Has been skipped
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m24s
Build ErsatzTV Image / Build & push image (amd64) (push) Has been cancelled
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Has been cancelled
fa2d787ac1
Root cause (diagnosed from run logs 513/515/516): the EF migration-integrity
job's "MySql apply all migrations to a fresh DB" step flakes when two migration
jobs land on the SAME runner host at once — each `services: mysql:8.4` container
starves the other, so the 787-migration replay either exceeds MySqlConnector's
30s default command timeout ("Command Timeout expired", run 513 on ci-runner) or
has its connection dropped mid-replay ("MySqlEndOfStreamException", run 516 on
bumblebee-runner). It's pure infra contention: `has-pending-model-changes` (the
model check) passes both providers, and the identical tree passes on a quieter
host (run 515). Both runners have both passed and failed — not one bad runner.

Fix (runner-agnostic, repo-owned workflow only — no runner-host change needed):
- Raise `DefaultCommandTimeout` to 300s in the MySql connection string.
- Wrap the apply in a 3× retry that resumes from `__EFMigrationsHistory` (EF
  commits each migration in its own transaction, so an interrupted one rolls back
  and the retry continues). A real migration failure fails on every attempt, so
  the retry can't mask a genuine problem.

Docs: ci-cd.md migration-integrity section documents the contention + retry.

Refs #13 #236

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
timothy merged commit fa2d787ac1 into main 2026-07-12 00:51:03 +02:00
timothy deleted branch ci/resilient-mysql-migration 2026-07-12 00:51:03 +02:00
Sign in to join this conversation.