Files
ersatztv/scripts/tests/test_workflow_persist_credentials.py
T
timothy 226f132b1c fix(744,835): review round 1 — ref-class residual, retracted two-step, guard escapes
Two independent cold reviews (Codex GPT-5.6, and a fresh Opus reviewer in an isolated
worktree) both returned BLOCKED. All findings addressed:

- The "what this does not close" statement sorted by the `branches:` KEYWORD, which
  mis-sorts `docker-build.yml`: it has `branches: [main]` and would pass, yet also
  `tags: ['v*']` and `pull_request:`, both head-resolved on the same registry
  credential — and a tag push is explicitly outside `release.main-direct-push-disabled`,
  so it needs no dispatch at all. Restated over the REF CLASS a trigger admits, in the
  record's `rule:` and in ci-cd.md, and #853 updated to match.
- The recovery text offered "dispatch on `main` after the merge and bump the pin" as an
  alternative. It cannot work: `expected` is the last `docker/ci` commit, so a pin naming
  the workflow-only commit fails `ci-image-pin` and leaving the pin keeps consuming the
  old image. A `docker/ci/**` touch is the only remedy.
- `ci-image.yml`'s own ROLLOUT NOTE still described the retracted merge-then-follow-up-PR
  two-step, five lines below the trigger summary the change had updated.
- The guard: `uses: https://github.com/actions/checkout@v4` is a valid Gitea spelling and
  escaped both the walk and the text cross-check — canonicalised, and both spellings are
  now pinned in the negative control. Added two absence tests so the population's
  assumptions REPORT themselves rather than going blind: no tracked workflow below
  `.gitea/workflows/` (the direct-children derivation is shared with two sibling guards),
  and no local composite action (whose `action.yml` is invisible to both halves of the
  cross-check — the docstring had cited the wrong test for that case).
- `ci-image-pin` no longer claims to prove PUBLICATION: it compares git shas and never
  queries the registry, which matters more now that publishing is a human dispatch.
  Existence stays `toolchain-preflight`'s job.
- The `:latest` guard is now the SOLE protection on the one event that never exercised it
  (a dispatch selects any ref), and says so.
- Stale contracts: the credential-scoping record's `rule:` frontmatter and body still said
  15-of-16; the pin job's `::error::` still said "let ci-image.yml publish".

The `docker/ci/Dockerfile` comment fixes were made, reddened `ci-image-pin` (any byte
there re-points `expected`), and were reverted — the trap is now recorded in ci-cd.md and
the record, with both stale line ranges deferred to #854.

refs #744
refs #835
Decisions-Edit: yes
2026-08-27 20:31:56 +02:00

222 lines
11 KiB
Python

"""Every `actions/checkout` drops the persisted credential (ersatztv#835).
WHAT THIS IS PROTECTING. `actions/checkout` writes an `Authorization` header into `.git/config`
unless `persist-credentials: false` is set, and this instance's Actions default token permission is
`permissive` — so that header is write-capable, and every later step in the job inherits it, not
just the checkout. ersatztv#746 set the flag on every checkout but one; nothing held the convention
afterwards. A job added tomorrow gets a checkout without the flag, and **nothing goes
red** — the silent-by-construction shape `docs/defect-shapes-773.md` §4 exists to catch.
WHY THERE IS NO EXEMPTION LIST. `ci-image.yml`'s checkout was the one left unset, for a mechanical
reason in another issue (#744), and this guard was deliberately held back rather than shipped with a
one-entry exemption for it: an exemption outlives its reason silently — once #744 set the flag there,
the entry would still have passed and the guard would have been permanently blind to the very file it
was written for. #744 landed with this, so the assertion is universal over the derived population.
POPULATION. The git index, via `tracked_files.tracked_paths` — not a filesystem walk and not a
hand-written list (`testing.guard-derives-population-from-source`, ersatztv#806). Both `*.yml` and
`*.yaml` are matched: Gitea accepts either spelling, so a `.yaml` workflow would otherwise be
structurally invisible to a check that reads as covering all of them.
"""
from __future__ import annotations
import re
from pathlib import Path
import yaml
from scripts.tests.tracked_files import _git_ls_files, tracked_paths
WORKFLOW_DIR = ".gitea/workflows"
WORKFLOWS = (WORKFLOW_DIR, ("*.yml", "*.yaml"))
CHECKOUT = "actions/checkout"
# Gitea accepts a full action URL as well as `owner/repo`, so `https://github.com/actions/checkout@v4`
# is the same step spelled differently and must not escape. Canonicalise before comparing.
_ACTION_URL = re.compile(r"^(?:https?://)?(?:[^/]+\.[^/]+)/(?=[^/]+/[^/]+)")
# Matches the `uses:` line the YAML walk is supposed to reach, in either spelling. Used ONLY as an
# independent second opinion on how many there are — never as the population itself.
_USES_CHECKOUT = re.compile(
r"^\s*-?\s*uses:\s*['\"]?(?:(?:https?://)?[^/'\"\s]+\.[^/'\"\s]+/)?actions/checkout(@|\s|['\"]|$)",
re.M,
)
def _canonical_action(uses: str) -> str:
"""`https://github.com/actions/checkout@v4` and `actions/checkout@v4` are the same action."""
return _ACTION_URL.sub("", uses.strip().strip("'\"")).split("@", 1)[0].strip()
def workflow_files() -> list[Path]:
return tracked_paths(*WORKFLOWS)
def _checkout_steps(doc: object) -> list[tuple[str, int, dict]]:
"""Every `actions/checkout` step in a parsed workflow, as (job id, index within job, step).
Reads `jobs.<id>.steps` only, which is the whole shape this repo uses. Two different escapes are
reported by two different tests, and they are not interchangeable:
`test_the_walk_finds_every_actions_checkout_the_TEXT_does` covers a shape INSIDE a workflow file
that the walk cannot reach, because both halves read the same files. A **composite action** is
invisible to both halves — its `action.yml` is not in the workflow population at all — so that
case is reported by `test_no_LOCAL_COMPOSITE_ACTION_carries_an_unchecked_checkout` instead.
"""
found: list[tuple[str, int, dict]] = []
if not isinstance(doc, dict):
return found
jobs = doc.get("jobs")
if not isinstance(jobs, dict):
return found
for job_id, job in jobs.items():
if not isinstance(job, dict):
continue
steps = job.get("steps")
if not isinstance(steps, list):
continue
for index, step in enumerate(steps):
if not isinstance(step, dict):
continue
uses = step.get("uses")
if isinstance(uses, str) and _canonical_action(uses) == CHECKOUT:
found.append((str(job_id), index, step))
return found
def checkout_faults(rel: str, doc: object) -> list[str]:
"""Human-readable faults for one workflow — accumulated, not failed fast.
A missing key and an explicit `true` are the same defect and are reported the same way: what
matters is whether the credential is left behind, not how the step spelled it.
"""
faults: list[str] = []
for job_id, index, step in _checkout_steps(doc):
with_block = step.get("with")
value = with_block.get("persist-credentials") if isinstance(with_block, dict) else None
if value is False:
continue
how = "does not set it at all" if value is None else f"sets it to {value!r}"
faults.append(
f"{rel}: job `{job_id}` step #{index} uses {CHECKOUT} and {how} — so the action "
f"persists a write-capable Authorization header into .git/config, and every later "
f"step in that job inherits it. Add `persist-credentials: false` under `with:` "
f"(ersatztv#746, guarded by ersatztv#835). There is deliberately no exemption list: "
f"if a checkout genuinely needs the credential, say so in the step and change this "
f"guard in the same PR."
)
return faults
def test_every_actions_checkout_DROPS_the_persisted_credential() -> None:
faults: list[str] = []
for path in workflow_files():
rel = path.relative_to(Path(__file__).resolve().parents[2]).as_posix()
faults.extend(checkout_faults(rel, yaml.safe_load(path.read_text())))
assert not faults, "\n".join(faults)
def test_the_walk_finds_every_actions_checkout_the_TEXT_does() -> None:
"""The YAML walk and a plain text scan must agree on the count, per file.
Without this, a step list the walk cannot reach — a composite action, a reusable workflow, a
shape act_runner grows later — makes checkouts structurally invisible while the guard above
still reports a clean run. Absence of output is not evidence; make the system report it.
"""
disagreements: list[str] = []
for path in workflow_files():
text = path.read_text()
walked = len(_checkout_steps(yaml.safe_load(text)))
scanned = len(_USES_CHECKOUT.findall(text))
if walked != scanned:
disagreements.append(
f"{path.name}: the jobs.<id>.steps walk found {walked} {CHECKOUT} step(s) but the "
f"text scan found {scanned}. The walk is what the flag assertion runs over, so the "
f"difference is checkouts this guard cannot see — widen `_checkout_steps`."
)
assert not disagreements, "\n".join(disagreements)
def test_the_population_and_the_checkout_set_are_NOT_empty() -> None:
"""Anti-vacuity, in both directions.
An empty population, or a population of workflows in which the walk finds no checkout at all,
makes the assertion above pass while measuring nothing — which is the failure this guard exists
to prevent, arriving through the guard itself.
"""
files = workflow_files()
assert files, (
"`git ls-files` reported no .gitea/workflows/*.yml — the derivation is broken, not the repo, "
"and every assertion in this file would pass vacuously."
)
total = sum(len(_checkout_steps(yaml.safe_load(p.read_text()))) for p in files)
assert total > 0, (
f"Parsed {len(files)} workflow(s) and found no {CHECKOUT} step in any of them. Either the "
"repo really has none (then delete this guard and say why) or `_checkout_steps` stopped "
"matching, in which case the flag assertion is measuring an empty set."
)
def test_the_fault_collector_reports_a_checkout_that_OMITS_the_flag() -> None:
"""Negative control on the collector itself, over both defect spellings.
The mutation proof (`mutation_manifest.py`) drives the real tree; this drives the collector
directly so a collector that silently stopped reporting is caught without waiting for the
harness, and so `true` is covered as well as an absent key.
"""
omitted = {"jobs": {"j": {"steps": [{"uses": "actions/checkout@v4"}]}}}
explicit_true = {"jobs": {"j": {"steps": [{"uses": "actions/checkout@v4", "with": {"persist-credentials": True}}]}}}
compliant = {"jobs": {"j": {"steps": [{"uses": "actions/checkout@v4", "with": {"persist-credentials": False}}]}}}
assert len(checkout_faults("synthetic.yml", omitted)) == 1
assert "does not set it at all" in checkout_faults("synthetic.yml", omitted)[0]
assert len(checkout_faults("synthetic.yml", explicit_true)) == 1
assert "sets it to True" in checkout_faults("synthetic.yml", explicit_true)[0]
assert checkout_faults("synthetic.yml", compliant) == []
# Gitea accepts a full action URL, so the same step spelled as a URL must not escape either —
# pinned rather than probed once, because a canonicalisation that quietly stopped working would
# leave the guard green over a checkout it no longer recognises as one.
for spelling in (
"https://github.com/actions/checkout@v4",
"github.com/actions/checkout@v4",
):
as_url = {"jobs": {"j": {"steps": [{"uses": spelling}]}}}
assert len(checkout_faults("synthetic.yml", as_url)) == 1, spelling
assert _USES_CHECKOUT.search(f" - uses: {spelling}"), spelling
assert not _USES_CHECKOUT.search(" - uses: actions/setup-node@v4")
def test_no_workflow_hides_in_a_SUBDIRECTORY() -> None:
"""The population is direct children of `.gitea/workflows`; prove nothing sits below it.
`tracked_children` is direct-children-only by design (see `tracked_files`), which is right for a
flat directory and blind the moment one stops being flat. This does not widen the population —
it makes the assumption REPORT itself, so a nested workflow reddens the guard instead of being
silently uncovered, and it covers the two sibling guards that derive the same directory the same
way (`test_ci_image_pin_population.py`, `test_pr_changed_files.py`).
"""
nested = sorted(
path for path in _git_ls_files() if path.startswith(f"{WORKFLOW_DIR}/") and "/" in path[len(WORKFLOW_DIR) + 1 :]
)
assert not nested, (
f"tracked workflow file(s) below {WORKFLOW_DIR}/: {nested}. The population here is direct "
"children only, so these are NOT checked for `persist-credentials: false` — widen the "
"derivation (and the sibling guards that share it) before adding them."
)
def test_no_LOCAL_COMPOSITE_ACTION_carries_an_unchecked_checkout() -> None:
"""A checkout inside a local composite action runs in the job but is outside the workflow set.
The repo has no `action.yml`/`action.yaml` today, so rather than write a parser for a shape that
does not exist, assert the absence: the first one added reddens this and its author widens the
guard, instead of the guard reading as complete while a whole class walked around it.
"""
actions = sorted(path for path in _git_ls_files() if path.rpartition("/")[2] in {"action.yml", "action.yaml"})
assert not actions, (
f"local composite action definition(s) found: {actions}. A composite action's steps run in "
"the job, so an `actions/checkout` in one is subject to the same rule and is NOT covered by "
"the workflow-file population above — extend `_checkout_steps` to walk them."
)