The EF migration integrity job's MySql apply-all-migrations-to-a-fresh-DB step flakes when two migration jobs land on the same runner host at once (common when several PRs push together). Each job spins its own services: mysql:8.4 container, and the paired MySQL servers starve each other replaying the 787 migrations. Diagnosed from run logs tonight:
Run
Runner
MySql apply outcome
513
ci-runner
❌The Command Timeout expired (>30s command under load)
✅ passed (ran concurrently with 516 — one host, two MySQLs, one starved)
It is pure infra contention, not a model/code problem: has-pending-model-changes (the actual model check) passes for both providers, and the identical tree passes on a quieter host. Both runners have both passed and failed, so it isn't one bad runner.
Fix (runner-agnostic; repo-owned workflow only)
DefaultCommandTimeout=300 on the MySql connection string (up from MySqlConnector's 30s default) — survives a slow DDL command under load.
3× retry around the apply that resumes from __EFMigrationsHistory — EF commits each migration in its own transaction, so an interrupted one rolls back cleanly and the retry continues where it left off. A genuine migration failure fails deterministically on every attempt, so the retry can never mask a real problem.
No runner-host change is needed (that would be server-management); this hardens the job itself, de-flaking it for every in-flight PR.
Verified
YAML parses; the retry block passes bash -n.
This PR's own EF migration integrity job exercises the hardened step.
Docs: docs/ci-cd.md migration-integrity section documents the contention + retry.
## Root cause
The `EF migration integrity` job's **MySql apply-all-migrations-to-a-fresh-DB** step flakes when two migration jobs land on the **same runner host at once** (common when several PRs push together). Each job spins its own `services: mysql:8.4` container, and the paired MySQL servers starve each other replaying the 787 migrations. Diagnosed from run logs tonight:
| Run | Runner | MySql apply outcome |
|-----|--------|--------------------|
| 513 | ci-runner | ❌ `The Command Timeout expired` (>30s command under load) |
| 516 | bumblebee-runner | ❌ `MySqlEndOfStreamException` (connection dropped mid-replay) |
| 515 | bumblebee-runner | ✅ passed (ran concurrently with 516 — one host, two MySQLs, one starved) |
It is **pure infra contention, not a model/code problem**: `has-pending-model-changes` (the actual model check) passes for both providers, and the *identical tree* passes on a quieter host. Both runners have both passed and failed, so it isn't one bad runner.
## Fix (runner-agnostic; repo-owned workflow only)
- **`DefaultCommandTimeout=300`** on the MySql connection string (up from MySqlConnector's 30s default) — survives a slow DDL command under load.
- **3× retry** around the apply that **resumes from `__EFMigrationsHistory`** — EF commits each migration in its own transaction, so an interrupted one rolls back cleanly and the retry continues where it left off. A *genuine* migration failure fails deterministically on every attempt, so the retry can never mask a real problem.
No runner-host change is needed (that would be server-management); this hardens the job itself, de-flaking it for **every** in-flight PR.
## Verified
- YAML parses; the retry block passes `bash -n`.
- This PR's own `EF migration integrity` job exercises the hardened step.
Docs: `docs/ci-cd.md` migration-integrity section documents the contention + retry.
Refs #13 #236
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Root cause (diagnosed from run logs 513/515/516): the EF migration-integrity
job's "MySql apply all migrations to a fresh DB" step flakes when two migration
jobs land on the SAME runner host at once — each `services: mysql:8.4` container
starves the other, so the 787-migration replay either exceeds MySqlConnector's
30s default command timeout ("Command Timeout expired", run 513 on ci-runner) or
has its connection dropped mid-replay ("MySqlEndOfStreamException", run 516 on
bumblebee-runner). It's pure infra contention: `has-pending-model-changes` (the
model check) passes both providers, and the identical tree passes on a quieter
host (run 515). Both runners have both passed and failed — not one bad runner.
Fix (runner-agnostic, repo-owned workflow only — no runner-host change needed):
- Raise `DefaultCommandTimeout` to 300s in the MySql connection string.
- Wrap the apply in a 3× retry that resumes from `__EFMigrationsHistory` (EF
commits each migration in its own transaction, so an interrupted one rolls back
and the retry continues). A real migration failure fails on every attempt, so
the retry can't mask a genuine problem.
Docs: ci-cd.md migration-integrity section documents the contention + retry.
Refs #13#236
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
timothy
merged commit fa2d787ac1 into main2026-07-12 00:51:03 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Root cause
The
EF migration integrityjob's MySql apply-all-migrations-to-a-fresh-DB step flakes when two migration jobs land on the same runner host at once (common when several PRs push together). Each job spins its ownservices: mysql:8.4container, and the paired MySQL servers starve each other replaying the 787 migrations. Diagnosed from run logs tonight:The Command Timeout expired(>30s command under load)MySqlEndOfStreamException(connection dropped mid-replay)It is pure infra contention, not a model/code problem:
has-pending-model-changes(the actual model check) passes for both providers, and the identical tree passes on a quieter host. Both runners have both passed and failed, so it isn't one bad runner.Fix (runner-agnostic; repo-owned workflow only)
DefaultCommandTimeout=300on the MySql connection string (up from MySqlConnector's 30s default) — survives a slow DDL command under load.__EFMigrationsHistory— EF commits each migration in its own transaction, so an interrupted one rolls back cleanly and the retry continues where it left off. A genuine migration failure fails deterministically on every attempt, so the retry can never mask a real problem.No runner-host change is needed (that would be server-management); this hardens the job itself, de-flaking it for every in-flight PR.
Verified
bash -n.EF migration integrityjob exercises the hardened step.Docs:
docs/ci-cd.mdmigration-integrity section documents the contention + retry.Refs #13 #236
🤖 Generated with Claude Code
Root cause (diagnosed from run logs 513/515/516): the EF migration-integrity job's "MySql apply all migrations to a fresh DB" step flakes when two migration jobs land on the SAME runner host at once — each `services: mysql:8.4` container starves the other, so the 787-migration replay either exceeds MySqlConnector's 30s default command timeout ("Command Timeout expired", run 513 on ci-runner) or has its connection dropped mid-replay ("MySqlEndOfStreamException", run 516 on bumblebee-runner). It's pure infra contention: `has-pending-model-changes` (the model check) passes both providers, and the identical tree passes on a quieter host (run 515). Both runners have both passed and failed — not one bad runner. Fix (runner-agnostic, repo-owned workflow only — no runner-host change needed): - Raise `DefaultCommandTimeout` to 300s in the MySql connection string. - Wrap the apply in a 3× retry that resumes from `__EFMigrationsHistory` (EF commits each migration in its own transaction, so an interrupted one rolls back and the retry continues). A real migration failure fails on every attempt, so the retry can't mask a genuine problem. Docs: ci-cd.md migration-integrity section documents the contention + retry. Refs #13 #236 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>