fix(772,792): name the missing toolchain image; stop a refusal leaving a verdict comment #825
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#825
Reference in New Issue
Block a user
fixes #772
fixes #792
#772 — a deleted toolchain image took all container CI down with no signal
Root cause established, and it is registry-side. An enabled Gitea owner-level container cleanup rule (
keep_count15,remove_days1,remove_pattern.*, and akeep_patternno 7-hex sha can match) deletes a sha tag once 15 newer versions of the package exist.cleanup_packagesruns@midnightand ran nightly through the outage window. The corroboration is the siblingersatztvpackage carrying the same rule's fingerprint exactly — every:<sha>older than the 15-slot window gone, everykeep_patterntag kept back to26.3.1.Stated limit: package-version deletes leave no audit row, so the specific run cannot be replayed. The durable fix belongs to the registry's repo and is filed as timothy/server-management#842;
docs/ci-cd.mdsays what to conclude if a pin vanishes after that lands (cause refuted, not re-applied).What lands here is what a consumer of someone else's registry can do:
toolchain-preflight— a container-free job (a job consuming the missing image could not run to report it) resolving every pin against the registry and failing with a message that names the tag and the recovery. Deliberately not aneeds:of the five jobs it diagnoses: they already fail fast, and gating them would tax every green run to speed up a rare red one.IS GONE; a 200 whose body is not a manifest, rejected credentials, and an unverifiable answer (afterETV_CI_ATTEMPTStries) are all red, each with its own wording because they send an operator to different places. An absent secret interpolates to":", which is refused before any query.200→ deleted →404).#792 — the premise was wrong; the half-state was real
The issue reported the script exiting 0 having written no status. Re-measured: that is false on the current tree — every no-status path exits non-zero, and the eight refusal modes now asserted here pass against
origin/main's script, which is how we know the defect was never in it. The observed 0 came from the invocation, not the script. The contract is asserted anyway: it held by convention, through one shareddiehelper, and nothing had ever executed it.What WAS broken is the issue's second half. The comment was written before the status, so every refusal left
Review-verdict: MERGEABLE @ <head>on a PR with no gating status behind it — an artifact that reads as granted consent, produced by the run that refused to grant it. The two writes are now ordered status-then-comment, which makes the only reachable half-state the safe one: a status with no comment leaves the hook's condition (c) with nothing to classify →ask. The refusals themselves are untouched.Ordering rather than compensating deletion: an orphaned-comment cleanup needs a Gitea call, and these refusals are usually caused by Gitea being unreachable. Recorded as
release.verdict-writes-status-before-comment.Verification
scripts/tests— 910 passed, 2 skipped;ruff check/formatclean; decisions validator + catalog clean; doc-narrative clean on 3 files.":"→ refused before any query, and a two-pin file (one live, one gone) → correct per-pin verdicts.404)→4040)) is executed every suite and reddens the named proof on its diagnostic — since an unverifiable answer fails too, the exit code no longer separates the cases.origin/main's script, the 8 exit-code tests stay green.Review
Three adversarial rounds, two reviewers, one cross-family. Round 1 and 2 independently found the same defect — the preflight WARNED and exited 0 on any answer that was not 200/404, i.e. a permanently green no-op if
curlwere missing or the registry moved. Fixed. Rounds 3 and 4 found the same shape twice more (a corrected claim with an uncorrected twin elsewhere in the tree) and are closed. Final verdict MERGEABLE.Deferred with a home:
curl -uputs credentials in argv at fivescripts/call sites — a class, filed as #821, named at the site rather than fixed here.Docs
docs/ci-cd.md(failure mode, arm table, recovery, root cause + reproduction),docs/guard-inventory.md(+2 rows, counts),docs/remote-state-inventory.md(+1 row), a new decision record, catalog regenerated, plus theci.required-job-step-execution-markerscarve-out, which named one container-free job and now has two.Two independent reviewers (one cross-family) converged on the same defect, and it was the important one: the preflight WARNED and exited 0 on every answer that was not 200 or 404, so a missing `curl`, a moved registry or a DNS change would have left it green forever — "the check could not run" presenting as "the pin is fine", in a script whose own header disclaimed exactly that. Unknown answers are now retried (3x, 5s) and then FAIL, with wording kept distinct from the deleted case because the two send an operator to different places. Also from the reviews: * An absent secret does not arrive as an unset variable. `${{ secrets.X }}:${{ secrets.Y }}` interpolates to ":", a perfectly non-empty and perfectly useless credential, and the tests covered only the unset shape. Both halves are now required, and the parametrised test drives the production shape. * HTTP 200 is not a manifest. A proxy or a login page answers 200 too, so the body is fetched and matched for `schemaVersion` (a shell `case`, so no jq dependency and no pipeline that can inject). * The curl stub ignored `-u` and answered 200 regardless, so deleting the real `-u` would have left the suite green while the live registry rejected every request. It now 401s an unauthenticated read, as the registry does. * The mutation's declared diagnostic changed with the script: now that unknown fails too, the exit code no longer separates "deleted" from "could not check", so the proof turns on the message and `expect` says so. * docs/ci-cd.md: `scan` is no longer the only `docker-build.yml` job on the small lane, so the tag-push exclusivity claim and the lane membership were both false. Fixed. * "Immutable" was overstated: `ci-image.yml` tags `rev-parse --short HEAD`, so a dispatch or a weekly no-cache run at the same HEAD republishes that tag from a rebuilt image. Stated, along with what the rebuild recovery does NOT restore (mutable bases and apt, so equivalent rather than bit-identical). * The recovery recipe left you in a worktree checked out at the pin commit — where the verify script does not exist, and where the workflow carries the pre-bump pin. It now keeps `$repo`, returns, and removes the worktree. It also needed BuildKit's `http = true` caveat: the container driver does not inherit the daemon's insecure-registries. * The root cause carries its evidentiary limit and its reproduction commands, and says what to conclude if a pin vanishes after server-management#842 lands (refuted, not re-applied). * The `ci.required-job-step-execution-markers` carve-out named one container-free job; there are two now, and the membership is what rots. * The decision record's `''` YAML escapes leaked into rendered prose; "status, no comment -> ask" is qualified (a prior positive verdict for the SAME head still satisfies condition (c)); "exits 1" is "exits non-zero" (usage exits 2, jq its own status, signals 128+n). refs #772 refs #792 Decisions-Edit: yesBoth remaining review findings were the same shape as the one before them, and it is the shape this repo keeps recording: a claim corrected in one place, its copy left standing somewhere else in the tree. * `docs/ci-cd.md` said "gates nothing" in the small-lane paragraph while the section 1441 lines below said the opposite. A red preflight lands in the PR's combined status, which the merge gate reads (#598) — what it does not do is SKIP the jobs it diagnoses, and that is now the sentence in both places. * Two docstrings in the preflight's test file still described the disarmed script as warning and exiting 0. Built the mutant and ran it: it emits an error and exits 1. The exit code separates nothing now that an unverifiable answer fails too — the DIAGNOSTIC is what the mutation destroys, which is what `mutation_manifest.py` already said and the prose next to it contradicted. Nits from the same pass: the admin-cron URL is quoted (`?` globs in zsh, the operator's shell); the retry assertion's message quoted a threshold it does not use; the arm table omitted the malformed-credential shape the code and tests both have; `buildx inspect` no longer `--bootstrap`s a builder just to read its name, and an empty capture no longer produces a noisy `buildx use ""`. Swept the tree for the shape rather than the two reported lines: the surviving "exits 0" and "could-not-tell" hits are other subsystems, or the concept named as a concept. refs #772Review-verdict: MERGEABLE @
78fc283Three adversarial rounds across two reviewers (one cross-family, GPT-5.6 via codex exec). Rounds 1+2 independently found the same real defect: the preflight WARNED and exited 0 on any answer that was not 200/404, so a missing curl or a moved registry would have left it green forever. Fixed and re-reviewed; rounds 3+4 each found one more instance of the same shape (a corrected claim with an uncorrected twin elsewhere in the tree) and both are closed. Final round verified by executing mutants and live registry calls rather than reading the diff. Local: 910 passed / 2 skipped, ruff clean, decisions validator + catalog clean, doc-narrative clean. The preflight is observed green on a real runner (run 2250 job 9519: 'resolves (HTTP 200, manifest present)' plus its step-execution marker), which settles the two environmental preconditions this branch could not assume. Deferred with a home: #821 (curl -u credentials in argv, five call sites, a class not a site).