fix(ci): repin the toolchain image — 32747a0 vanished from the registry
#771
Closed
timothy
wants to merge 1 commits from
fix/ci-image-repin into main
pull from: fix/ci-image-repin
merge into: :main
:main
:renovate/dotnet-monorepo
:renovate/meziantou.analyzer-3.x
:renovate/lucene.net
:renovate/sqlitepclraw.bundle_e_sqlite3-3.x
:renovate/cliwrap-3.x
:913-scripted-playout-401
:release/v26.15.0-notes
:issue-806-guard-populations
:scratch/767b-poisoned
:scratch/767b-control
:release/v26.14.0-notes
:release/v26.14.0
:docs/510-skill-logo-bug-policy
:fix/510-watermark-resolution-policy
:fix/629-verdict-classifier-falseopens
:fix/609-decisions-edit-token-scope
:issue-135-clear-to-none
:release/v26.12.0-notes
:fix/409b-lastscan-api-parity
:fix/401-updatechannel-mirror-422
:fix/327-playlist-rename-validation
:fix/410-scancancel-log-level
:fix/409-447-librariesscreen-neverscanned
:fix/338-zap-exit-code
:fix/367-plex-budget-message
:fix/310-debom-legacy-cs
:ci/604-lane-rebalance
:feat/388-design-mirror
:feat/247-test-ownership
:feat/247-primary-action
:feat/357-player-owned-playback
:feat/357-jellyfin-plugin-poc
:fix/289-mcp-hardening
:issue58-mcp
:feat/244-channels-extract
:ci/auto-bump-prod-compose
:feat/multi-rerun-collections-api
:feat/collections-api
:feat/quick-wins
:feat/185-docs-part2
:feat/140-collections-screen
:feat/146-channel-edit
:feat/147-classic-ui-link
:issue22-renovate-dashboard
:feat/91-cutover
:feat/63-composite-create
:feat/65-library-browse
:feat/85-epg
:feat/86-schedule-editor
:feat/109-dashboard-data
:feat/99-session-tracking
:fix/dockerfile-node-tag
:feat/59-spa-foundation
:docs/59-ui-redesign-brief
:feat/102-json-guide
:feat/111-schedule-durations
:feat/104-artwork-upload
:feat/103-media-sources-api
:feat/playouts-read-api
:feat/108-health-api
:feat/105-picker-list-endpoints
:issue-97-channel-state-api
:issue42-jellyfin-musicvideos
:issue46-rest-api-error-contract
:dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
:qsv-improvements
:hdr-vulkan-cuda-test
No Reviewers
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#771
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
CI is red repo-wide and this restores it. Not a code defect — the pinned toolchain image is simply gone.
Every
container:job indocker-build.yml(test,migrations,functional-e2e,api-docs,format) fails after 1–2s at the image pull:Both required contexts (
Build & test (.NET),EF migration integrity) are among them, so nothing in the repo could merge.Evidence
main— no PR touched it.GET /v2/timothy/ersatztv-ci/tags/list→3a6174c, 40dc752, 7280a91, 9881d1f, 9928be8, 9dc360c, buildcache, latest. No32747a0.mainf9f8f65c), container jobs started fine at2026-08-11T19:26Z.2026-08-13T17:19Z.So the tag was removed from the registry inside that ~2-day window. By what is not established — that is the open question and it is deliberately not guessed at here.
How the shape was spotted quickly: the
scanjob from #767 stayed green throughout. It is the only job with nocontainer:, so it was the only one that could still start — which isolated the fault to the image pull rather than the workflow or the diff.The fix
The documented two-step from
docs/ci-cd.md→ "CI toolchain image":ci-image.ymldispatched onmain→ republished the toolchain as:f9f8f65. It tags by build-commit sha, so a rebuild produces a new tag rather than resurrecting32747a0.This PR's own green run is the proof the new image works — which is precisely what the two-step exists for.
Deliberately not fixed here
Nothing prevents a recurrence. A pin whose image can silently disappear is a standing fail-open: CI goes red repo-wide and the only signal is a pull error buried inside each job's log. Restoring service and hardening against the class are separate changes, and bundling them would make this PR unreviewable while CI is down. Tracked separately.
32747a0vanished from the registryClosing unmerged — this was the wrong fix, and the way it failed is the most useful thing here.
ci-image-pincorrectly rejected it:The gate requires the pin to equal the last commit that touched
docker/ci/ci-image.yml, because that is howci-image.ymlnames what it publishes. But aworkflow_dispatchrebuild tags bygit rev-parse --short HEAD— main's HEAD, not the image-source commit.So the documented recovery is self-defeating.
docs/ci-cd.mdsays "dispatch a rebuild, then update the pin", and that works when bumping the toolchain (the dispatch follows adocker/cichange, so HEAD is the image-source commit). It cannot work when restoring a lost tag: the rebuild produces a tag thatci-image-pinwill reject by construction, and repinning to it means the jobs test an image not built from this PR'sdocker/ci— exactly what the gate exists to prevent. Chasing it would have meant either disabling the gate or landing a knowingly-wrong pin.What actually fixed it:
docker/cihas not changed since32747a0, so the image published by the rebuild is built from identical sources. Re-tagging it to the sha the convention demands restores the exact artifact the pin names:GET /v2/timothy/ersatztv-ci/manifests/32747a0→ 200. No code change was needed at all;main's pin was correct the whole time.One honesty note: the restored
32747a0is a rebuild from the same sources, not the byte-identical original — a fresh build pulls current base layers, so anything that moved upstream since is now in it. That is what any rebuild would have given, but it means "restored" is not "recovered".Follow-ups on #772, which now has a concrete second failure mode to record: the recovery procedure in
docs/ci-cd.mdonly covers upgrading the toolchain, and following it during an outage produces a pin the repo's own gate refuses.Pull request closed