fix(ci): repin the toolchain image — 32747a0 vanished from the registry #771

Closed
timothy wants to merge 1 commits from fix/ci-image-repin into main
Owner

CI is red repo-wide and this restores it. Not a code defect — the pinned toolchain image is simply gone.

Every container: job in docker-build.yml (test, migrations, functional-e2e, api-docs, format) fails after 1–2s at the image pull:

Error response from daemon: failed to resolve reference
"192.168.1.95:3000/timothy/ersatztv-ci:32747a0": not found

Both required contexts (Build & test (.NET), EF migration integrity) are among them, so nothing in the repo could merge.

Evidence

  • The pin is unchanged on main — no PR touched it.
  • GET /v2/timothy/ersatztv-ci/tags/list → 3a6174c, 40dc752, 7280a91, 9881d1f, 9928be8, 9dc360c, buildcache, latest. No 32747a0.
  • Last known good: run 1933 (push to main f9f8f65c), container jobs started fine at 2026-08-11T19:26Z.
  • First observed failure: run 1937, 2026-08-13T17:19Z.

So the tag was removed from the registry inside that ~2-day window. By what is not established — that is the open question and it is deliberately not guessed at here.

How the shape was spotted quickly: the scan job from #767 stayed green throughout. It is the only job with no container:, so it was the only one that could still start — which isolated the fault to the image pull rather than the workflow or the diff.

The fix

The documented two-step from docs/ci-cd.md → "CI toolchain image":

  1. ci-image.yml dispatched on main → republished the toolchain as :f9f8f65. It tags by build-commit sha, so a rebuild produces a new tag rather than resurrecting 32747a0.
  2. This PR moves all six pin sites onto it.

This PR's own green run is the proof the new image works — which is precisely what the two-step exists for.

Deliberately not fixed here

Nothing prevents a recurrence. A pin whose image can silently disappear is a standing fail-open: CI goes red repo-wide and the only signal is a pull error buried inside each job's log. Restoring service and hardening against the class are separate changes, and bundling them would make this PR unreviewable while CI is down. Tracked separately.

**CI is red repo-wide and this restores it.** Not a code defect — the pinned toolchain image is simply gone. Every `container:` job in `docker-build.yml` (`test`, `migrations`, `functional-e2e`, `api-docs`, `format`) fails after 1–2s at the image pull: ``` Error response from daemon: failed to resolve reference "192.168.1.95:3000/timothy/ersatztv-ci:32747a0": not found ``` Both **required** contexts (`Build & test (.NET)`, `EF migration integrity`) are among them, so nothing in the repo could merge. ## Evidence - The pin is **unchanged on `main`** — no PR touched it. - `GET /v2/timothy/ersatztv-ci/tags/list` → `3a6174c, 40dc752, 7280a91, 9881d1f, 9928be8, 9dc360c, buildcache, latest`. **No `32747a0`.** - **Last known good:** run 1933 (push to `main` `f9f8f65c`), container jobs started fine at `2026-08-11T19:26Z`. - **First observed failure:** run 1937, `2026-08-13T17:19Z`. So the tag was removed from the registry inside that ~2-day window. **By what is not established** — that is the open question and it is deliberately not guessed at here. How the shape was spotted quickly: the `scan` job from #767 stayed **green** throughout. It is the only job with no `container:`, so it was the only one that could still start — which isolated the fault to the image pull rather than the workflow or the diff. ## The fix The documented two-step from `docs/ci-cd.md` → "CI toolchain image": 1. `ci-image.yml` dispatched on `main` → republished the toolchain as **`:f9f8f65`**. It tags by build-commit sha, so a rebuild produces a *new* tag rather than resurrecting `32747a0`. 2. This PR moves all **six** pin sites onto it. This PR's own green run is the proof the new image works — which is precisely what the two-step exists for. ## Deliberately not fixed here Nothing prevents a recurrence. A pin whose image can silently disappear is a standing fail-open: CI goes red repo-wide and the only signal is a pull error buried inside each job's log. Restoring service and hardening against the class are separate changes, and bundling them would make this PR unreviewable while CI is down. Tracked separately.
timothy added 1 commit 2026-08-13 19:26:56 +02:00
fix(ci): repin the toolchain image — 32747a0 vanished from the registry
review-verdict/h10 Awaiting review verdict for e7b3675
Review verdict / Set review-verdict status (pull_request_target) Successful in 14s
PR Gates / Docs update reminder (pull_request) Successful in 12s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Script tests (pytest) (pull_request) Successful in 2m16s
PR Gates / CI image pin matches docker/ci (pull_request) Failing after 6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m21s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 0s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Canceled after 0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Canceled after 45s
e7b3675a6b
Every `container:` job in docker-build.yml (`test`, `migrations`, `functional-e2e`,
`api-docs`, `format`) has been failing after 1-2s with:

  Error response from daemon: failed to resolve reference
  "192.168.1.95:3000/timothy/ersatztv-ci:32747a0": not found

That is repo-wide and blocks BOTH required contexts, so nothing could merge. It is not a
code defect: the pin is unchanged on main, and the image simply is not in the registry any
more. `GET /v2/timothy/ersatztv-ci/tags/list` returned 3a6174c, 40dc752, 7280a91, 9881d1f,
9928be8, 9dc360c, buildcache, latest — no 32747a0.

Last known good: run 1933 (push to main f9f8f65c, 2026-08-11T19:26Z) started its container
jobs successfully. First observed failure: run 1937, 2026-08-13T17:19Z. So the tag was
removed from the registry somewhere in that ~2-day window; by what is not established, and
that is the open question, tracked separately.

`scan` (ersatztv#767) was green throughout, which is how the shape was identified quickly —
it is the one job with no `container:`, so it was the only one that could still start.

The fix is the documented two-step: `ci-image.yml` was dispatched on main, which republished
the toolchain as `:f9f8f65` (it tags by build-commit sha, so a rebuild yields a NEW tag rather
than restoring the old one), and this PR moves all six pin sites onto it. This PR's own green
run is the proof the new image works — that is exactly what the two-step exists for.

NOT done here: nothing has been changed to prevent a recurrence. A pin whose image can
disappear is a standing fail-open — CI goes red repo-wide and the only signal is a container
pull error buried in each job. Tracked separately rather than bundled into a restore.

Refs: #767
timothy closed this pull request 2026-08-13 19:39:23 +02:00
Author
Owner

Closing unmerged — this was the wrong fix, and the way it failed is the most useful thing here.

ci-image-pin correctly rejected it:

CI toolchain image pin is stale: docker-build.yml pins ersatztv-ci:f9f8f65
(f9f8f65cee87...), but docker/ci was last changed in 32747a067e60...

The gate requires the pin to equal the last commit that touched docker/ci / ci-image.yml, because that is how ci-image.yml names what it publishes. But a workflow_dispatch rebuild tags by git rev-parse --short HEAD — main's HEAD, not the image-source commit.

So the documented recovery is self-defeating. docs/ci-cd.md says "dispatch a rebuild, then update the pin", and that works when bumping the toolchain (the dispatch follows a docker/ci change, so HEAD is the image-source commit). It cannot work when restoring a lost tag: the rebuild produces a tag that ci-image-pin will reject by construction, and repinning to it means the jobs test an image not built from this PR's docker/ci — exactly what the gate exists to prevent. Chasing it would have meant either disabling the gate or landing a knowingly-wrong pin.

What actually fixed it: docker/ci has not changed since 32747a0, so the image published by the rebuild is built from identical sources. Re-tagging it to the sha the convention demands restores the exact artifact the pin names:

docker tag  …/ersatztv-ci:f9f8f65 …/ersatztv-ci:32747a0
docker push …/ersatztv-ci:32747a0
→ 32747a0: digest sha256:7a8db810… size 3539

GET /v2/timothy/ersatztv-ci/manifests/32747a0 → 200. No code change was needed at all; main's pin was correct the whole time.

One honesty note: the restored 32747a0 is a rebuild from the same sources, not the byte-identical original — a fresh build pulls current base layers, so anything that moved upstream since is now in it. That is what any rebuild would have given, but it means "restored" is not "recovered".

Follow-ups on #772, which now has a concrete second failure mode to record: the recovery procedure in docs/ci-cd.md only covers upgrading the toolchain, and following it during an outage produces a pin the repo's own gate refuses.

Closing unmerged — **this was the wrong fix, and the way it failed is the most useful thing here.** `ci-image-pin` correctly rejected it: ``` CI toolchain image pin is stale: docker-build.yml pins ersatztv-ci:f9f8f65 (f9f8f65cee87...), but docker/ci was last changed in 32747a067e60... ``` The gate requires the pin to equal **the last commit that touched `docker/ci` / `ci-image.yml`**, because that is how `ci-image.yml` names what it publishes. But a `workflow_dispatch` rebuild tags by **`git rev-parse --short HEAD`** — main's HEAD, not the image-source commit. **So the documented recovery is self-defeating.** `docs/ci-cd.md` says "dispatch a rebuild, then update the pin", and that works when bumping the toolchain (the dispatch follows a `docker/ci` change, so HEAD *is* the image-source commit). It cannot work when *restoring* a lost tag: the rebuild produces a tag that `ci-image-pin` will reject by construction, and repinning to it means the jobs test an image not built from this PR's `docker/ci` — exactly what the gate exists to prevent. Chasing it would have meant either disabling the gate or landing a knowingly-wrong pin. **What actually fixed it:** `docker/ci` has not changed since `32747a0`, so the image published by the rebuild is built from identical sources. Re-tagging it to the sha the convention demands restores the exact artifact the pin names: ``` docker tag …/ersatztv-ci:f9f8f65 …/ersatztv-ci:32747a0 docker push …/ersatztv-ci:32747a0 → 32747a0: digest sha256:7a8db810… size 3539 ``` `GET /v2/timothy/ersatztv-ci/manifests/32747a0` → **200**. No code change was needed at all; `main`'s pin was correct the whole time. One honesty note: the restored `32747a0` is a **rebuild from the same sources**, not the byte-identical original — a fresh build pulls current base layers, so anything that moved upstream since is now in it. That is what any rebuild would have given, but it means "restored" is not "recovered". Follow-ups on #772, which now has a concrete second failure mode to record: the recovery procedure in `docs/ci-cd.md` only covers *upgrading* the toolchain, and following it during an *outage* produces a pin the repo's own gate refuses.

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.