Files
ersatztv/scripts/tests/test_workflow_job_guards.py
T
timothy f420649fa2 fix(786,789): harden the step-text predicate and sharpen the guard/none rule
Self-review findings on the previous commit, all measured before fixing.

THE PREDICATE HAD BOTH FAILURE DIRECTIONS. Measured against constructed inputs:

  full-line comment       -> ['dotnet']      false POSITIVE
  echo string / heredoc   -> ['dotnet']      false POSITIVE
  dotnet-ef migrations    -> []              false NEGATIVE
  playwright install      -> []              false NEGATIVE

The false positive is the dangerous one: a bare-runner job whose COMMENT merely mentions dotnet
would redden the suite on a correct tree, and a correct guard going red on a correct tree is the
fastest route to that guard being deleted. Full-line comments are now stripped before matching.
`dotnet-ef` and `playwright` join the tool list, and the alternation is sorted longest-first so the
match does not depend on backtracking out of the `dotnet` prefix.

The residue is stated in the code rather than papered over: a TRAILING `# ...` comment, an
`echo "run dotnet build"` and a heredoc body all still match. Parsing those means parsing shell,
which `testing.fixing-a-parser-bug-introduces-the-next-one` prices at three or more rounds. It is
tolerable only because of its DIRECTION — a loud red naming the job and the token, with an obvious
remedy — never a silent pass.

Verified the real workflow's classification is unchanged and exact after the change:
needs-toolchain derives to {test, migrations, functional-e2e, api-docs, format}, equal to the
declared set, zero violations.

THE GUARD/NONE RULE WAS UNDER-SPECIFIED. "Purpose, not whether a step can fail" does not decide
`docker-build.yml::test`, which blocks the merge as a required context. Sharpened everywhere it is
stated: a `guard` job enforces an invariant about the REPOSITORY or its ARTIFACTS; a job that builds
or exercises the PRODUCT is `none` even when it is a required check. No marker value changed — the
rule now explains the classifications that were already there.

Also: a row for test_workflow_job_guards.py in the file-populations table, and the scope-limit
preamble de-narrated (it recounted an earlier draft; the operative warning — an unlisted class reads
as covered — is kept, the autobiography dropped).

Refs #786
Refs #789
2026-08-28 22:14:44 +02:00

294 lines
14 KiB
Python

"""Every workflow JOB that is a guard has a row in docs/guard-inventory.md (ersatztv#786).
WHAT THIS CLOSES. The inventory's population was guard FILES — `.claude/hooks/`, `.husky/`, and the
`scripts/…` paths workflows and hooks reference. A guard written INLINE in workflow YAML belonged to
no population at all, and the inventory said so in its own scope limit rather than implying coverage.
The motivating case lived exactly there: `pr-checks.yml:ci-image-pin` stated an invariant in its
error text — "Every container: job must pin ersatztv-ci:<7-char-sha>" — and did not check it (#774).
It was fixed by adding a structural test, not by bringing the job into any inventory, so the next one
would have been just as invisible.
WHY A MARKER RATHER THAN A LIST. "Which jobs are guards" is a judgement per job, and #786 says so:
the filesystem cannot supply it. The available shapes were a hand-written registry in this file, or a
declaration in the workflow. This takes the second, for the reason #789 rejected the first in
`test_ci_image_pin_population.py`: a literal list of members is a filter frozen at authoring time,
correct the day it is written and unable to report the day it stopped being. The judgement still has
to be made by a human — it is made in the workflow diff, next to the job, where a reviewer changing
that job sees it.
WHAT THE MARKER MEANS, because the obvious wrong reading makes the population useless. It records the
job's PURPOSE, not whether any step can fail: a `guard` job enforces an invariant about the
REPOSITORY or its ARTIFACTS, while a job that builds or exercises the PRODUCT is `none` even when it
is a required check. `docker-build.yml::test` blocks the merge and is still `none` — what it runs is
the product's own test suite, not a check on the repo — and `docker-build.yml::build` is `none`
despite a smoke test that fails the job. Read the marker as "any job containing a step that can
fail" and every job is a guard and the table stops distinguishing anything.
THE RESIDUE, stated rather than left for review. This proves every guard JOB has been classified by
someone and has a row; it cannot prove the classification is CORRECT, and it cannot prove an inline
assertion works — that needs the job to run. `Proof: NONE` on an inline row is therefore the honest
entry and not a gap to be filled with a neighbouring file's proof. Same residue `test_guard_inventory.py`
records for its own table, and for the same reason: the table is what review reads.
"""
from __future__ import annotations
import copy
import re
from pathlib import Path
import pytest
import yaml
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
SECTION = "Workflow-job guards (ersatztv#786)"
# From the GIT INDEX, never `Path.glob` (ersatztv#806): an untracked scratch workflow left in the
# directory would otherwise join the population on one checkout and not in CI — red locally, green on
# the server, which is #778's third shape. `*.yaml` alongside `*.yml` because Gitea accepts both.
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
ROLE_KEY = "CI_JOB_ROLE"
GUARD, ADVISORY, NONE = "guard", "advisory", "none"
# A VOCABULARY, not a population — the legal values of the marker, not a list of jobs.
ROLES = frozenset({GUARD, ADVISORY, NONE})
# role -> the Kind the row must carry. `none` is absent by construction: it takes no row.
ROLE_KIND = {GUARD: "GUARD", ADVISORY: "ADVISORY"}
_ROW = re.compile(r"^\|\s*`([a-z0-9.-]+\.ya?ml::[A-Za-z0-9_-]+)`\s*\|([^|]*)\|\s*([A-Z]+)\s*\|", re.M)
def workflow_files() -> list[Path]:
"""THE WORKFLOW POPULATION, from the git index. Named so the shared proof in
`test_guard_populations_derive_from_git.py` can assert it never admits an untracked file."""
return tracked_paths(*WORKFLOWS)
def declared_roles() -> dict[str, str | None]:
"""`<workflow>::<job>` -> the role it declares, or None where the marker is absent.
None is KEPT rather than dropped: an unmarked job is the defect `role_faults` reports, and a dict
that silently omitted it could not report anything.
"""
out: dict[str, str | None] = {}
for path in workflow_files():
doc = yaml.safe_load(path.read_text()) or {}
for jid, job in (doc.get("jobs") or {}).items():
if not isinstance(job, dict):
continue
value = (job.get("env") or {}).get(ROLE_KEY)
out[f"{path.name}::{jid}"] = str(value) if value is not None else None
return out
def role_faults(roles: dict[str, str | None]) -> list[str]:
"""MISSING OR UNKNOWN IS A HARD FAILURE.
A marker whose absence defaults to `none` would stop applying the moment someone adds a job and
forgets — and the job nobody remembered to mark is the job nobody reviewed. An unrecognised value
is rejected for the same reason rather than read as "not a guard".
"""
faults = []
for job, value in sorted(roles.items()):
if value is None:
faults.append(
f"job '{job}' declares no {ROLE_KEY}. Every job in every tracked workflow must "
f"declare one under its `env:` — {sorted(ROLES)} — so that 'not a guard' is a "
"recorded decision rather than an omission."
)
elif value not in ROLES:
faults.append(f"job '{job}' declares {ROLE_KEY}: {value!r}, which is not one of {sorted(ROLES)}.")
return faults
def inventory_section() -> str:
"""The body of the workflow-job section alone.
Scoped, not a whole-file scan: `docs/guard-inventory.md` holds several tables and the file-guard
one is read by `test_guard_inventory.py`. Two shape-keyed regexes over one document would each
pull in the other's rows and report them as phantoms. A missing heading raises rather than
returning empty — a silent empty section is the vacuous population this file exists to prevent.
"""
text = INVENTORY.read_text()
start = text.index(f"\n## {SECTION}\n")
nxt = text.find("\n## ", start + 1)
return text[start : nxt if nxt != -1 else len(text)]
def inventory_jobs() -> dict[str, str]:
"""`<workflow>::<job>` -> Kind, for each row of the workflow-job table."""
return {job.strip(): kind.strip() for job, _blocks, kind in _ROW.findall(inventory_section())}
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY FIRST — a population or a row regex that stopped matching would make every assertion
# below compare two empty sets and report a fully-covered inventory. That is the failure this file
# exists to prevent, so it is checked before anything depends on it.
# ------------------------------------------------------------------------------------------------
def test_the_workflow_population_is_not_empty():
files = workflow_files()
assert files, (
"no tracked workflow files were found — `tracked_paths` returned nothing. Every assertion "
"below would compare empty sets and pass having checked nothing."
)
roles = declared_roles()
assert roles, f"{len(files)} workflow file(s) parsed but no jobs were found in any of them"
def test_the_workflow_job_table_actually_parsed():
rows = inventory_jobs()
assert rows, (
f"the `## {SECTION}` table parsed to ZERO rows. Either the table is gone or `_ROW` no "
"longer matches its shape; both make the set equality below vacuous."
)
# ------------------------------------------------------------------------------------------------
# THE LIVE ASSERTIONS
# ------------------------------------------------------------------------------------------------
def test_every_job_declares_a_known_role():
faults = role_faults(declared_roles())
assert not faults, "workflow jobs with no usable role:\n " + "\n ".join(faults)
def test_the_inventory_covers_exactly_the_guard_JOBS_that_exist():
"""SET EQUALITY, both directions — the two failures are opposite mistakes.
Left-to-right: a job declared a guard with no row, which is the gap #786 was filed for.
Right-to-left: a row for a job that no longer declares itself a guard (renamed, deleted, or
downgraded), which is how a table becomes a comforting fiction.
"""
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
listed = set(inventory_jobs())
assert declared - listed == set(), (
f"these workflow jobs declare {ROLE_KEY} guard/advisory but have NO row in "
f"`## {SECTION}` — {sorted(declared - listed)}. Add a row saying what each blocks, where "
"its assertion lives and what proof it carries."
)
assert listed - declared == set(), (
f"`## {SECTION}` has rows for jobs that no longer declare a guard/advisory role — "
f"{sorted(listed - declared)}. Either the job was renamed or removed, or its marker "
"changed; the row must follow."
)
def test_each_row_carries_the_KIND_its_marker_declares():
"""The row and the workflow must not disagree about WHICH kind.
Set equality alone is satisfied by an `advisory` job holding a `GUARD` row — the table would then
claim a merge-blocking check where the workflow has one that cannot fail. `docs-reminder` is
exactly that job (`continue-on-error: true` on every check step), so the case is live, not
hypothetical.
"""
roles = declared_roles()
listed = inventory_jobs()
faults = [
f"'{job}' declares {ROLE_KEY}: {roles.get(job)} (Kind {ROLE_KIND.get(roles.get(job))}) but its row says {kind}"
for job, kind in sorted(listed.items())
if ROLE_KIND.get(roles.get(job)) != kind
]
assert not faults, "row Kind disagrees with the job's declared role:\n " + "\n ".join(faults)
# ------------------------------------------------------------------------------------------------
# MUTATION PROOFS — THE GUARD IS A TEST, so each mutation goes into the guarded ARTIFACT (the
# workflow, or the table) rather than into an assertion here. Disarming a checker makes it ABSENT
# rather than red; `testing.guard-ships-with-mutation-proof`'s checker-guard exception is what makes
# introducing the real defect the admissible proof.
# ------------------------------------------------------------------------------------------------
def _role_mutants():
for job in sorted(declared_roles()):
removed = dict(declared_roles())
removed[job] = None
yield f"{job}-marker-removed", removed
unknown = dict(declared_roles())
unknown[job] = "checker"
yield f"{job}-marker-unknown-value", unknown
_ROLE_MUTANTS = list(_role_mutants())
@pytest.mark.parametrize("roles", [m for _, m in _ROLE_MUTANTS], ids=[i for i, _ in _ROLE_MUTANTS])
def test_a_missing_or_UNKNOWN_role_is_DETECTED(roles):
assert role_faults(roles), "a job with no role, or an unrecognised one, was accepted"
def test_the_role_mutation_set_is_not_empty():
expected = 2 * len(declared_roles())
assert len(_ROLE_MUTANTS) == expected, f"expected 2 role mutations per job ({expected}), got {len(_ROLE_MUTANTS)}"
def test_MUTATION_a_guard_job_with_its_row_DELETED_is_reported():
"""The defect this guard exists to catch: a guard job that acquired no row.
Mutating the guarded ARTIFACT (the table), not this checker's population — a shrunken population
would make every real row report as a PHANTOM, demonstrating a false POSITIVE while proving
nothing about the missing-row detection the inventory row claims.
"""
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
listed = set(inventory_jobs())
assert declared and listed, "nothing to mutate — the population or the table is empty"
for victim in sorted(listed):
mutated = listed - {victim}
assert declared - mutated, f"deleting the row for '{victim}' was not reported as missing"
def test_MUTATION_a_PHANTOM_row_is_reported():
"""The opposite direction: a row for a job that does not declare a guard role."""
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
mutated = set(inventory_jobs()) | {"docker-build.yml::a-job-that-was-deleted"}
assert mutated - declared, "a row naming a non-guard job was not reported as a phantom"
def test_MUTATION_a_row_whose_KIND_disagrees_is_reported():
"""`advisory` holding a `GUARD` row must not pass — the case `docs-reminder` makes live."""
roles = declared_roles()
advisory = [j for j, r in roles.items() if r == ADVISORY]
assert advisory, "no advisory job exists, so this proof is vacuous"
listed = dict(inventory_jobs())
listed[advisory[0]] = "GUARD"
faults = [j for j, kind in listed.items() if ROLE_KIND.get(roles.get(j)) != kind]
assert faults, "an advisory job carrying a GUARD row was accepted"
def test_a_workflow_job_added_without_a_marker_is_DETECTED():
"""A NEW job is the realistic arrival path, and it must not default to `none`."""
roles = dict(declared_roles())
roles["docker-build.yml::a-brand-new-job"] = None
assert role_faults(roles), "a newly added job with no role marker was accepted"
def test_the_marker_is_read_from_the_PARSED_yaml_not_a_text_search():
"""Guards against the parser regressing to a grep.
A text search for `CI_JOB_ROLE: guard` would count this file's own prose, the inventory's, and a
commented-out marker. Asserting the parse produces exactly the tracked workflows' jobs is what
keeps the population a property of the YAML.
"""
roles = declared_roles()
for path in workflow_files():
doc = yaml.safe_load(path.read_text()) or {}
for jid in doc.get("jobs") or {}:
assert f"{path.name}::{jid}" in roles, f"{path.name}::{jid} was not parsed"
assert all("::" in k for k in roles)
def test_a_commented_out_marker_does_NOT_count():
"""The concrete form of the above: YAML comments are not values."""
doc = yaml.safe_load("jobs:\n x:\n env:\n # CI_JOB_ROLE: guard\n OTHER: 1\n steps: []\n")
job = doc["jobs"]["x"]
assert (job.get("env") or {}).get(ROLE_KEY) is None
mutated = copy.deepcopy(doc)
assert (mutated["jobs"]["x"].get("env") or {}).get(ROLE_KEY) is None