ci: make the MySql migration-apply resilient to concurrent-runner contention
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 8s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 4m14s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 7m12s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Build ErsatzTV Image / Docs update reminder (push) Has been skipped
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m24s
Build ErsatzTV Image / Build & push image (amd64) (push) Has been cancelled
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Has been cancelled

Root cause (diagnosed from run logs 513/515/516): the EF migration-integrity
job's "MySql apply all migrations to a fresh DB" step flakes when two migration
jobs land on the SAME runner host at once — each `services: mysql:8.4` container
starves the other, so the 787-migration replay either exceeds MySqlConnector's
30s default command timeout ("Command Timeout expired", run 513 on ci-runner) or
has its connection dropped mid-replay ("MySqlEndOfStreamException", run 516 on
bumblebee-runner). It's pure infra contention: `has-pending-model-changes` (the
model check) passes both providers, and the identical tree passes on a quieter
host (run 515). Both runners have both passed and failed — not one bad runner.

Fix (runner-agnostic, repo-owned workflow only — no runner-host change needed):
- Raise `DefaultCommandTimeout` to 300s in the MySql connection string.
- Wrap the apply in a 3× retry that resumes from `__EFMigrationsHistory` (EF
  commits each migration in its own transaction, so an interrupted one rolls back
  and the retry continues). A real migration failure fails on every attempt, so
  the retry can't mask a genuine problem.

Docs: ci-cd.md migration-integrity section documents the contention + retry.

Refs #13 #236

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit was merged in pull request #294.
This commit is contained in:
2026-07-12 00:42:49 +02:00
co-authored by Claude Opus 4.8
parent ee6be81c22
commit fa2d787ac1
2 changed files with 37 additions and 3 deletions
+23 -3
View File
@@ -169,7 +169,12 @@ jobs:
# service container above. MySql__ConnectionString maps to config key "MySql:ConnectionString".
- name: MySql — model drift + apply all migrations to a fresh DB
env:
MySql__ConnectionString: "Server=mysql;Port=3306;Database=ersatztv_migrations;Uid=root;Pwd=ersatztv;"
# DefaultCommandTimeout is raised from MySqlConnector's 30s default: replaying every
# migration to a fresh DB issues DDL commands that can exceed 30s when two migration jobs
# share a runner host (each spins its own mysql:8.4 service) and starve each other. That
# contention produced both "Command Timeout expired" and mid-replay connection drops
# (MySqlEndOfStreamException) — neither is a model problem. See #13 / #236.
MySql__ConnectionString: "Server=mysql;Port=3306;Database=ersatztv_migrations;Uid=root;Pwd=ersatztv;DefaultCommandTimeout=300;"
run: |
set -euo pipefail
export PATH="$PATH:$HOME/.dotnet/tools"
@@ -178,8 +183,23 @@ jobs:
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql
echo "::endgroup::"
echo "::group::MySql apply all migrations to a fresh DB"
dotnet ef database update --no-build --configuration Release \
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql
# Retry the apply: under concurrent-runner MySQL contention the server can drop the
# connection mid-replay. Each attempt resumes from __EFMigrationsHistory (EF wraps each
# migration in its own transaction, so an interrupted migration rolls back cleanly and the
# retry continues from the last committed one) — so this only papers over infra flakiness,
# never a real migration failure, which fails deterministically on every attempt.
attempt=1
max=3
until dotnet ef database update --no-build --configuration Release \
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql; do
if [ "$attempt" -ge "$max" ]; then
echo "MySql apply failed after ${max} attempts" >&2
exit 1
fi
echo "MySql apply attempt ${attempt} failed (likely runner MySQL contention); retrying in 15s..." >&2
attempt=$((attempt + 1))
sleep 15
done
echo "::endgroup::"
build:
+14
View File
@@ -245,6 +245,20 @@ outside a transaction and warn at startup; they can't be rolled back mid-migrati
migrations carefully (this is part of what motivated the apply-to-fresh check before the prod
cutover, server-management#481).
**Resilience — the MySql apply is retried (concurrent-runner contention, not a model bug)**: both
runners (`ci-runner` VM 127 + `bumblebee-runner`) serve `ubuntu-latest`, and when two migration jobs
land on the **same host at once** (common when several PRs push together), each spins its own
`mysql:8.4` service container and they starve each other — producing intermittent `Command Timeout
expired` **or** mid-replay `MySqlEndOfStreamException` (dropped connection) on the MySql
apply-to-fresh-DB step. This is pure infra flakiness — `has-pending-model-changes` (the actual model
check) still passes, and the *same commit* passes on a quieter host. The job hardens against it two
ways: the connection string sets `DefaultCommandTimeout=300` (up from MySqlConnector's 30s default),
and the apply is wrapped in a **3× retry** that resumes from `__EFMigrationsHistory` (EF commits each
migration in its own transaction, so an interrupted one rolls back and the retry continues). A *real*
migration failure fails deterministically on every attempt, so the retry never masks it. If a run
still flakes past the retry, re-trigger (Gitea has no rerun API on this version — push, or the run
drains); don't treat a lone MySql-apply red as a code problem without checking the failure mode.
## Pre-commit hooks (web/)
The repo uses **husky** git hooks (installed via `web/`'s **lint-staged** + npm) to catch