ci: make the MySql migration-apply resilient to concurrent-runner contention #294

Merged
timothy merged 1 commits from ci/resilient-mysql-migration into main 2026-07-12 00:51:03 +02:00
2 changed files with 37 additions and 3 deletions
+23 -3
View File
@@ -169,7 +169,12 @@ jobs:
# service container above. MySql__ConnectionString maps to config key "MySql:ConnectionString".
- name: MySql — model drift + apply all migrations to a fresh DB
env:
MySql__ConnectionString: "Server=mysql;Port=3306;Database=ersatztv_migrations;Uid=root;Pwd=ersatztv;"
# DefaultCommandTimeout is raised from MySqlConnector's 30s default: replaying every
# migration to a fresh DB issues DDL commands that can exceed 30s when two migration jobs
# share a runner host (each spins its own mysql:8.4 service) and starve each other. That
# contention produced both "Command Timeout expired" and mid-replay connection drops
# (MySqlEndOfStreamException) — neither is a model problem. See #13 / #236.
MySql__ConnectionString: "Server=mysql;Port=3306;Database=ersatztv_migrations;Uid=root;Pwd=ersatztv;DefaultCommandTimeout=300;"
run: |
set -euo pipefail
export PATH="$PATH:$HOME/.dotnet/tools"
@@ -178,8 +183,23 @@ jobs:
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql
echo "::endgroup::"
echo "::group::MySql apply all migrations to a fresh DB"
dotnet ef database update --no-build --configuration Release \
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql
# Retry the apply: under concurrent-runner MySQL contention the server can drop the
# connection mid-replay. Each attempt resumes from __EFMigrationsHistory (EF wraps each
# migration in its own transaction, so an interrupted migration rolls back cleanly and the
# retry continues from the last committed one) — so this only papers over infra flakiness,
# never a real migration failure, which fails deterministically on every attempt.
attempt=1
max=3
until dotnet ef database update --no-build --configuration Release \
--context TvContext --startup-project ErsatzTV --project ErsatzTV.Infrastructure.MySql -- --provider MySql; do
if [ "$attempt" -ge "$max" ]; then
echo "MySql apply failed after ${max} attempts" >&2
exit 1
fi
echo "MySql apply attempt ${attempt} failed (likely runner MySQL contention); retrying in 15s..." >&2
attempt=$((attempt + 1))
sleep 15
done
echo "::endgroup::"
build:
+14
View File
@@ -245,6 +245,20 @@ outside a transaction and warn at startup; they can't be rolled back mid-migrati
migrations carefully (this is part of what motivated the apply-to-fresh check before the prod
cutover, server-management#481).
**Resilience — the MySql apply is retried (concurrent-runner contention, not a model bug)**: both
runners (`ci-runner` VM 127 + `bumblebee-runner`) serve `ubuntu-latest`, and when two migration jobs
land on the **same host at once** (common when several PRs push together), each spins its own
`mysql:8.4` service container and they starve each other — producing intermittent `Command Timeout
expired` **or** mid-replay `MySqlEndOfStreamException` (dropped connection) on the MySql
apply-to-fresh-DB step. This is pure infra flakiness — `has-pending-model-changes` (the actual model
check) still passes, and the *same commit* passes on a quieter host. The job hardens against it two
ways: the connection string sets `DefaultCommandTimeout=300` (up from MySqlConnector's 30s default),
and the apply is wrapped in a **3× retry** that resumes from `__EFMigrationsHistory` (EF commits each
migration in its own transaction, so an interrupted one rolls back and the retry continues). A *real*
migration failure fails deterministically on every attempt, so the retry never masks it. If a run
still flakes past the retry, re-trigger (Gitea has no rerun API on this version — push, or the run
drains); don't treat a lone MySql-apply red as a code problem without checking the failure mode.
## Pre-commit hooks (web/)
The repo uses **husky** git hooks (installed via `web/`'s **lint-staged** + npm) to catch