timothyandClaude Fable 5.1
786d1c3c4b
`scripts/ci-toolchain-image-resolves.sh` and `docs/ci-cd.md` both stated that the #772 container jobs "died after 1-2s", and the header used the same number to argue the preflight needs no `needs:` gate. Nobody measured it, and it cannot be measured from a working session without reproducing a deleted-tag incident. What the number stood for is structural and IS known: a container job that cannot pull its image fails AT the pull, before it runs a step, so it wastes no work waiting to be told and the argument against serialising the five jobs survives intact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QqCpYFsKgnAnx6jVwrKiV
300 lines
19 KiB
Bash
Executable File
300 lines
19 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Preflight: does the PINNED CI toolchain image still exist in the registry? (ersatztv#772)
|
|
#
|
|
# WHY THIS EXISTS. `docker-build.yml` pins its five `container:` jobs to an immutable
|
|
# `ersatztv-ci:<sha>`. Between 2026-08-11 and 2026-08-13 that tag was deleted from the Gitea
|
|
# registry and every one of those jobs — including BOTH required contexts — died at image pull,
|
|
# before running a step, with
|
|
#
|
|
# Error response from daemon: failed to resolve reference "…/ersatztv-ci:<the pinned sha>": not found
|
|
#
|
|
# buried in each job's log. Nothing said "your toolchain image is gone", so the natural first
|
|
# reading was "my diff broke the build". This job says it in one line, in a job whose NAME says it.
|
|
#
|
|
# "Immutable" was taken to mean "will always exist", and those are different claims. The cause was
|
|
# an owner-level Gitea package cleanup rule (keep_count 15, remove_days 1, remove_pattern `.*`, and
|
|
# a keep_pattern no 7-hex sha can match), so a pinned tag is deleted once 15 newer versions of the
|
|
# package exist. The rule lives in the registry's repo — the durable fix is
|
|
# timothy/server-management#842 — and THIS script does not fix it. It converts a five-job pull
|
|
# failure into one actionable message, which is all a consumer of someone else's registry can do.
|
|
#
|
|
# WHY IT DOES NOT GATE THE CONTAINER JOBS with `needs:`. Serialising five jobs behind a checkout +
|
|
# one curl would tax every green run to speed up the rare red one, and a container job that cannot
|
|
# pull its image fails at the pull, before it runs a step, so it wastes no work waiting to be told.
|
|
# This runs in PARALLEL: the diagnosis is present the moment anyone looks, and the happy path pays
|
|
# nothing.
|
|
#
|
|
# UNKNOWN IS NOT A PASS. Warning and exiting 0 on every answer that is not 200 or
|
|
# 404 makes "curl is missing from this runner", "the
|
|
# registry moved", and "DNS changed" all indistinguishable from a healthy pin — a job that is green
|
|
# forever having checked nothing, in a file whose header claims the opposite. Unknown answers are
|
|
# RETRIED (they are usually transient) and then FAIL. The message stays distinct from the deleted
|
|
# case: "could not verify" and "IS GONE" send an operator to different places.
|
|
#
|
|
# IT READS THE REGISTRY ANONYMOUSLY, AND THAT IS LOAD-BEARING RATHER THAN INCIDENTAL. This job runs
|
|
# on the `pull_request` route, where the workflow YAML is supplied by the PR head, so it may hold no
|
|
# stored secret (ersatztv#885, `ci.pr-route-carries-no-stored-credential`). A naive unauthenticated
|
|
# GET of a manifest is a 401 for every tag, present or deleted — which is why this used to demand a
|
|
# credential — but that 401 is a Bearer CHALLENGE, and this registry issues an anonymous pull token
|
|
# for a public package against it. Measured 2026-09-04: the token endpoint answers 200 with no
|
|
# credential, that token reads the pinned manifest and its config blob (200), and a tag that does
|
|
# not exist answers 404 rather than 401 — so the deleted-tag diagnosis this whole script exists for
|
|
# survives the change. What does NOT survive it is the `ersatztv-ci` PACKAGE ceasing to be
|
|
# anonymously pullable: the token leg then refuses, and this fails loudly with a message that names
|
|
# that cause rather than reporting could-not-tell. That is the package and not this repo — the
|
|
# package is linked to no repository (measured 2026-09-05: every version of it reports
|
|
# `"repository": null`), so `timothy/ersatztv` going private would not take the pull token away.
|
|
# What that WOULD take away is the unauthenticated commit-status GET in
|
|
# scripts/ci-detect-already-validated.sh, which is a different dependency with the opposite failure
|
|
# direction: it falls through to `skip=false` and stays green. Do not cite this script's loudness
|
|
# for that one.
|
|
#
|
|
# Env (all optional; the defaults are the live values):
|
|
# ETV_CI_REGISTRY registry host:port (default 192.168.1.95:3000)
|
|
# ETV_CI_IMAGE_REPO package path inside the registry (default timothy/ersatztv-ci)
|
|
# ETV_CI_WORKFLOW workflow file to read the pin from (default .gitea/workflows/docker-build.yml)
|
|
# ETV_CI_ATTEMPTS tries per pin before an unknown becomes a failure (default 3)
|
|
# ETV_CI_RETRY_SECONDS pause between those tries (default 5)
|
|
set -euo pipefail
|
|
|
|
registry="${ETV_CI_REGISTRY:-192.168.1.95:3000}"
|
|
image_repo="${ETV_CI_IMAGE_REPO:-timothy/ersatztv-ci}"
|
|
workflow="${ETV_CI_WORKFLOW:-.gitea/workflows/docker-build.yml}"
|
|
|
|
fail() { printf '::error::ci-toolchain-image-resolves: %s\n' "$*" >&2; exit 1; }
|
|
|
|
[ -f "$workflow" ] || fail "cannot read $workflow to find the toolchain pin"
|
|
|
|
# The same expression `pr-checks.yml::ci-image-pin` greps with, so the two cannot disagree about
|
|
# what "the pin" is. Note it is written so THIS line cannot match itself: the character after the
|
|
# colon here is `[`, which is not in [0-9a-f].
|
|
pins=$(grep -oE 'ersatztv-ci:[0-9a-f]+' "$workflow" | cut -d: -f2 | sort -u || true)
|
|
[ -n "$pins" ] || fail "no ersatztv-ci pin found in $workflow — if the grep pattern stopped matching, fix it here and in pr-checks.yml::ci-image-pin together"
|
|
|
|
accept='application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json'
|
|
attempts="${ETV_CI_ATTEMPTS:-3}"
|
|
retry_seconds="${ETV_CI_RETRY_SECONDS:-5}"
|
|
rc=0
|
|
|
|
# The anonymous pull token, acquired once and reused for every pin. Empty until the registry
|
|
# challenges. `token_leg_done` separates "not tried yet" from "tried", which is what lets the
|
|
# refusal arms below report what the run actually DID rather than what it might have done.
|
|
#
|
|
# `token_leg_retryable` separates the two ways the leg can come back empty, and they are not the
|
|
# same event. A token endpoint that ANSWERED and named no token — or a challenge naming no realm, or
|
|
# no challenge at all — has told us something, and asking it again cannot change the answer: that is
|
|
# the registry genuinely refusing anonymous reads, and it is asked once per RUN. A token endpoint
|
|
# that could not be REACHED, or that answered 5xx, told us nothing; it is the same transport blip the
|
|
# manifest read absorbs, so it is retried on the same `ETV_CI_ATTEMPTS` budget. Without that split
|
|
# the two legs of one read had opposite flake tolerances — measured 2026-09-05 on the predecessor of
|
|
# this commit with `ETV_CI_ATTEMPTS=3`, an unreachable token endpoint failed after 1 token call and
|
|
# 1 manifest call while a flaky manifest read got 3 retries — and this job's red denies a merge (the
|
|
# consent hook reads the COMBINED status, ersatztv#598), so a one-second token-endpoint outage
|
|
# blocked a merge until someone re-ran it.
|
|
token=""
|
|
token_leg_done=0
|
|
token_leg_retryable=0
|
|
token_leg_attempts=0
|
|
headers=$(mktemp)
|
|
trap 'rm -f "$headers"' EXIT
|
|
|
|
# One directive out of a `Www-Authenticate: Bearer realm="…",service="…"` challenge. The realm is
|
|
# read from the challenge rather than assumed, so a registry that moves its token endpoint is
|
|
# followed instead of guessed at; `service` is optional (this registry issues a token without it,
|
|
# measured 2026-09-04) and is passed through when the challenge names one.
|
|
#
|
|
# The directive NAME is matched in any case, through a character class generated from the key, and
|
|
# the header name likewise (below): RFC 7235 §2.1 makes auth-param names case-insensitive and RFC
|
|
# 9110 §5.1 makes field names case-insensitive, so a registry answering `WWW-Authenticate: Bearer
|
|
# Realm="…"` is spelling this legally. Matching one spelling would send that challenge down the
|
|
# "named no realm" arm and `fail` the job with the wrong diagnosis — loud, but pointing an operator
|
|
# at a token endpoint that is fine. The VALUE keeps its case: a realm URL is case-sensitive.
|
|
challenge_field() {
|
|
local key="$2" pattern="" index char
|
|
for ((index = 0; index < ${#key}; index++)); do
|
|
char=${key:index:1}
|
|
pattern="${pattern}[${char}$(printf '%s' "$char" | tr '[:lower:]' '[:upper:]')]"
|
|
done
|
|
printf '%s' "$1" | sed -n "s/.*[ ,]$pattern=\"\([^\"]*\)\".*/\1/p" | head -1
|
|
}
|
|
|
|
# The token leg. Returns non-zero on every shape that leaves us without a bearer — no challenge, no
|
|
# realm in it, a token endpoint that will not answer, or an answer carrying no token. Each of those
|
|
# is "could not establish anonymous access", which the caller turns into a REFUSAL rather than a
|
|
# could-not-tell: an empty token would otherwise fall through to a second 401 and read as an
|
|
# ordinary auth failure with no cause named. It also SORTS them, into `token_leg_retryable`: only
|
|
# the shapes where the endpoint said nothing at all are worth asking again.
|
|
acquire_token() {
|
|
local challenge realm service url resp body http
|
|
token_leg_done=1
|
|
token_leg_retryable=0
|
|
token_leg_attempts=$((token_leg_attempts + 1))
|
|
challenge=$(tr -d '\r' < "$headers" |
|
|
awk -F: 'tolower($1) == "www-authenticate" { sub(/^[^:]*:[[:space:]]*/, ""); print; exit }')
|
|
[ -n "$challenge" ] || return 1
|
|
realm=$(challenge_field "$challenge" realm)
|
|
[ -n "$realm" ] || return 1
|
|
service=$(challenge_field "$challenge" service)
|
|
url="$realm?scope=repository:$image_repo:pull"
|
|
# Spelled as a full `if` rather than `[ … ] && …`: as a bare statement the latter returns the
|
|
# test's exit status, which is 1 whenever `service` is absent — a legal challenge shape.
|
|
if [ -n "$service" ]; then url="$url&service=$service"; fi
|
|
# The status code is read for the same reason the manifest read reads it: it is the only thing
|
|
# that separates "this endpoint REFUSED" from "this endpoint was not there", and only the second
|
|
# is worth a second ask. A `curl` that fails outright (DNS, connect, reset) leaves no response at
|
|
# all and lands in the same class.
|
|
resp=$(curl -s -w '\n%{http_code}' "$url") || resp=""
|
|
if [ -z "$resp" ]; then
|
|
token_leg_retryable=1
|
|
return 1
|
|
fi
|
|
http=${resp##*$'\n'}
|
|
body=${resp%$'\n'*}
|
|
case "$http" in
|
|
000|5??) token_leg_retryable=1; return 1 ;;
|
|
esac
|
|
token=$(printf '%s' "$body" | sed -n 's/.*"token"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -1)
|
|
[ -n "$token" ] || return 1
|
|
return 0
|
|
}
|
|
|
|
# One GET, recording the HTTP code and whether the body is a manifest. The body is fetched rather
|
|
# than a HEAD sent, because
|
|
# HTTP 200 alone does not mean "the manifest is there": a proxy, a captive login page or an error
|
|
# document all answer 200 with a body that is not a manifest, and a check that reads only the status
|
|
# line reports those as "resolves". A manifest always carries `schemaVersion`, so the body is matched
|
|
# for it — with a shell `case`, so nothing depends on jq being installed and no pipeline can invert
|
|
# the result on a large body.
|
|
#
|
|
# It ASSIGNS `code`/`is_manifest` rather than printing them, and so does `probe` — because the token
|
|
# and the "already tried the token leg" flag must survive from one pin to the next, and a
|
|
# `$(probe …)` command substitution runs in a SUBSHELL whose assignments are discarded. Reading the
|
|
# answer through a subshell would silently re-run the whole two-leg exchange for every pin.
|
|
request() {
|
|
local url="$1" resp body
|
|
if [ -n "$token" ]; then
|
|
resp=$(curl -s -w '\n%{http_code}' -D "$headers" -H "Authorization: Bearer $token" -H "Accept: $accept" "$url") || resp=""
|
|
else
|
|
resp=$(curl -s -w '\n%{http_code}' -D "$headers" -H "Accept: $accept" "$url") || resp=""
|
|
fi
|
|
if [ -z "$resp" ]; then
|
|
code="000"
|
|
is_manifest="no"
|
|
return 0
|
|
fi
|
|
code=${resp##*$'\n'}
|
|
body=${resp%$'\n'*}
|
|
case "$body" in
|
|
*'"schemaVersion"'*) is_manifest="yes" ;;
|
|
*) is_manifest="no" ;;
|
|
esac
|
|
}
|
|
|
|
# The two legs of an anonymous registry read: the plain GET, and — only if it is challenged and the
|
|
# leg is not already settled — the token exchange followed by ONE retry carrying the bearer. A 401
|
|
# that survives the retry is left as a 401 and refused by the caller; it is never rounded off to
|
|
# could-not-tell, because the cause (this package is no longer readable without a credential) sends
|
|
# an operator somewhere quite different from a flaky registry.
|
|
#
|
|
# "Settled" is `token_leg_done` AND NOT `token_leg_retryable`: a leg abandoned because its endpoint
|
|
# could not be reached settled nothing, so the next attempt asks again.
|
|
probe() {
|
|
request "$1"
|
|
if [ "$code" = "401" ] && { [ "$token_leg_done" -eq 0 ] || [ "$token_leg_retryable" -eq 1 ]; }; then
|
|
if acquire_token; then
|
|
request "$1"
|
|
fi
|
|
fi
|
|
}
|
|
|
|
code=""
|
|
is_manifest="no"
|
|
for pin in $pins; do
|
|
url="http://$registry/v2/$image_repo/manifests/$pin"
|
|
attempt=1
|
|
while : ; do
|
|
probe "$url"
|
|
case "$code" in
|
|
200|404|403) break ;;
|
|
401)
|
|
# A 401 is an answer — unless the only reason it still stands is a token leg whose endpoint
|
|
# could not be reached, in which case it is an unknown wearing the first read's status code
|
|
# and belongs in the retry class below with every other unknown.
|
|
if [ "$token_leg_retryable" -eq 0 ]; then break; fi
|
|
;;
|
|
esac
|
|
# Only the unknown answers are retried: 200/404 are answers, and an auth failure will not cure
|
|
# itself. A transient registry is the common case for the rest, and absorbing it here is what
|
|
# lets the unknown be a FAILURE at the end rather than a warning nobody reads.
|
|
[ "$attempt" -lt "$attempts" ] || break
|
|
attempt=$((attempt + 1))
|
|
sleep "$retry_seconds"
|
|
done
|
|
|
|
case "$code" in
|
|
200)
|
|
if [ "$is_manifest" = "yes" ]; then
|
|
printf 'ci-toolchain-image-resolves: %s/%s:%s resolves (HTTP 200, manifest present)\n' "$registry" "$image_repo" "$pin"
|
|
else
|
|
printf '::error::ci-toolchain-image-resolves: %s/%s:%s answered HTTP 200 with a body that is not a manifest (no schemaVersion). Something is answering for the registry — a proxy, a login page, or an error document. The pin was NOT verified.\n' \
|
|
"$registry" "$image_repo" "$pin" >&2
|
|
rc=1
|
|
fi
|
|
;;
|
|
404)
|
|
# The one unambiguous answer, and the outage this exists for.
|
|
printf '::error::ci-toolchain-image-resolves: the pinned CI toolchain image %s/%s:%s IS GONE from the registry (HTTP 404). Every container: job in docker-build.yml will fail at image pull, including both required contexts, and NO diff caused it. Recovery does not need CI: rebuild that exact tag from the commit it names and push it — see docs/ci-cd.md -> "CI toolchain image" -> "When the pinned tag disappears". Root cause + the durable fix: timothy/server-management#842.\n' \
|
|
"$registry" "$image_repo" "$pin" >&2
|
|
rc=1
|
|
;;
|
|
401|403)
|
|
# `fail` rather than `rc=1`: unlike a 404, this says nothing about the pin, and it will say
|
|
# the same thing about every remaining one. Abandoning the loop keeps the log to one cause.
|
|
#
|
|
# THREE ways to get here, and they send an operator to three different places, so they are
|
|
# worded apart — the same reason `404` and `could NOT VERIFY` are. Each message states only
|
|
# what actually ran, because a message naming a step that did not happen is evidence for a
|
|
# diagnosis nobody performed:
|
|
#
|
|
# * a refusal that survived a GOOD token is an answer about this PACKAGE, and since
|
|
# ersatztv#885 it is not a preflight-only problem: every `container:` job pulls the same
|
|
# image with no credential, so they fail at image pull too, including both required
|
|
# contexts;
|
|
# * a challenge that yielded no usable token is an infrastructure answer about the TOKEN
|
|
# ENDPOINT;
|
|
# * a FIRST-READ 403 never reached the token leg at all — `probe` enters it on a 401 only —
|
|
# so nothing was ever asked for. This is an answer about ACCESS to the registry. Its
|
|
# message says NO TOKEN WAS EVER REQUESTED rather than "the registry issued no challenge":
|
|
# a 403 MAY carry a `Www-Authenticate` and this script would still not follow it, so only
|
|
# the first is something the run observed. Note the boundary too: a 401 carrying no
|
|
# challenge DOES enter the token leg and abandon it, so it lands in the arm above, not
|
|
# this one. The branch order below is `token` first precisely so this case cannot borrow
|
|
# either of the other two mechanisms.
|
|
#
|
|
# A token leg that ANSWERED is not retried, deliberately: `token_leg_done` is set before the
|
|
# attempt and `token_leg_retryable` stays 0, so a registry genuinely refusing anonymous reads
|
|
# is asked once per RUN rather than once per pin or once per attempt. A token endpoint that
|
|
# could not be REACHED settled nothing and IS retried, on the same `ETV_CI_ATTEMPTS` budget the
|
|
# manifest read uses — the two legs of one read must not have opposite flake tolerances when a
|
|
# red here denies a merge. The message below reports how many token-leg attempts actually ran,
|
|
# so the two cases are distinguishable in the log rather than only in this comment.
|
|
if [ -n "$token" ]; then
|
|
fail "the registry refused an ANONYMOUS read (HTTP $code) of $registry/$image_repo:$pin even after a Bearer token was obtained, so the pin could not be checked. Every container: job pulls this image without a credential too, so they will fail at image pull. Check that the ersatztv-ci package is still PUBLIC (it is linked to no repository, so this is the package's own visibility, not the repo's) — do not read this as a pass."
|
|
fi
|
|
if [ "$token_leg_done" -eq 1 ]; then
|
|
fail "could NOT OBTAIN an anonymous pull token for $registry/$image_repo:$pin after $token_leg_attempts token-leg attempt(s) — the registry answered HTTP $code and the token leg produced none: either there was no Www-Authenticate challenge, or it named no realm, or the token endpoint did not answer with a token. The pin was NOT checked. Look at the registry's token endpoint, not at the pin."
|
|
fi
|
|
fail "the registry refused an ANONYMOUS read (HTTP $code) of $registry/$image_repo:$pin and NO TOKEN WAS EVER REQUESTED: the token leg is entered on a 401 only, so this answer was never followed as a Bearer challenge and the pin could not be checked. An outright refusal is an answer about ACCESS to the registry, not about the pin. Every container: job pulls this image without a credential too, so they will fail at image pull. Check that the ersatztv-ci package is still PUBLIC (it is linked to no repository, so this is the package's own visibility, not the repo's), and that nothing (a proxy, an ACL) is answering for the registry — do not read this as a pass."
|
|
;;
|
|
*)
|
|
# NOT gone, and NOT a pass either. Deliberately worded apart from the 404 message: this sends
|
|
# an operator to the registry's health, not to a rebuild of a tag that may be sitting there.
|
|
printf '::error::ci-toolchain-image-resolves: could NOT VERIFY %s/%s:%s after %s attempt(s) (last answer: HTTP %s). This is not evidence the image is gone — it is evidence the check could not run, which fails rather than passing so the preflight cannot quietly become a no-op.\n' \
|
|
"$registry" "$image_repo" "$pin" "$attempts" "$code" >&2
|
|
rc=1
|
|
;;
|
|
esac
|
|
done
|
|
|
|
exit "$rc"
|