Self-review findings on the previous commit, all measured before fixing.
THE PREDICATE HAD BOTH FAILURE DIRECTIONS. Measured against constructed inputs:
full-line comment -> ['dotnet'] false POSITIVE
echo string / heredoc -> ['dotnet'] false POSITIVE
dotnet-ef migrations -> [] false NEGATIVE
playwright install -> [] false NEGATIVE
The false positive is the dangerous one: a bare-runner job whose COMMENT merely mentions dotnet
would redden the suite on a correct tree, and a correct guard going red on a correct tree is the
fastest route to that guard being deleted. Full-line comments are now stripped before matching.
`dotnet-ef` and `playwright` join the tool list, and the alternation is sorted longest-first so the
match does not depend on backtracking out of the `dotnet` prefix.
The residue is stated in the code rather than papered over: a TRAILING `# ...` comment, an
`echo "run dotnet build"` and a heredoc body all still match. Parsing those means parsing shell,
which `testing.fixing-a-parser-bug-introduces-the-next-one` prices at three or more rounds. It is
tolerable only because of its DIRECTION — a loud red naming the job and the token, with an obvious
remedy — never a silent pass.
Verified the real workflow's classification is unchanged and exact after the change:
needs-toolchain derives to {test, migrations, functional-e2e, api-docs, format}, equal to the
declared set, zero violations.
THE GUARD/NONE RULE WAS UNDER-SPECIFIED. "Purpose, not whether a step can fail" does not decide
`docker-build.yml::test`, which blocks the merge as a required context. Sharpened everywhere it is
stated: a `guard` job enforces an invariant about the REPOSITORY or its ARTIFACTS; a job that builds
or exercises the PRODUCT is `none` even when it is a required check. No marker value changed — the
rule now explains the classifications that were already there.
Also: a row for test_workflow_job_guards.py in the file-populations table, and the scope-limit
preamble de-narrated (it recounted an earlier draft; the operative warning — an unlisted class reads
as covered — is kept, the autobiography dropped).
Refs #786
Refs #789
294 lines
14 KiB
Python
294 lines
14 KiB
Python
"""Every workflow JOB that is a guard has a row in docs/guard-inventory.md (ersatztv#786).
|
|
|
|
WHAT THIS CLOSES. The inventory's population was guard FILES — `.claude/hooks/`, `.husky/`, and the
|
|
`scripts/…` paths workflows and hooks reference. A guard written INLINE in workflow YAML belonged to
|
|
no population at all, and the inventory said so in its own scope limit rather than implying coverage.
|
|
The motivating case lived exactly there: `pr-checks.yml:ci-image-pin` stated an invariant in its
|
|
error text — "Every container: job must pin ersatztv-ci:<7-char-sha>" — and did not check it (#774).
|
|
It was fixed by adding a structural test, not by bringing the job into any inventory, so the next one
|
|
would have been just as invisible.
|
|
|
|
WHY A MARKER RATHER THAN A LIST. "Which jobs are guards" is a judgement per job, and #786 says so:
|
|
the filesystem cannot supply it. The available shapes were a hand-written registry in this file, or a
|
|
declaration in the workflow. This takes the second, for the reason #789 rejected the first in
|
|
`test_ci_image_pin_population.py`: a literal list of members is a filter frozen at authoring time,
|
|
correct the day it is written and unable to report the day it stopped being. The judgement still has
|
|
to be made by a human — it is made in the workflow diff, next to the job, where a reviewer changing
|
|
that job sees it.
|
|
|
|
WHAT THE MARKER MEANS, because the obvious wrong reading makes the population useless. It records the
|
|
job's PURPOSE, not whether any step can fail: a `guard` job enforces an invariant about the
|
|
REPOSITORY or its ARTIFACTS, while a job that builds or exercises the PRODUCT is `none` even when it
|
|
is a required check. `docker-build.yml::test` blocks the merge and is still `none` — what it runs is
|
|
the product's own test suite, not a check on the repo — and `docker-build.yml::build` is `none`
|
|
despite a smoke test that fails the job. Read the marker as "any job containing a step that can
|
|
fail" and every job is a guard and the table stops distinguishing anything.
|
|
|
|
THE RESIDUE, stated rather than left for review. This proves every guard JOB has been classified by
|
|
someone and has a row; it cannot prove the classification is CORRECT, and it cannot prove an inline
|
|
assertion works — that needs the job to run. `Proof: NONE` on an inline row is therefore the honest
|
|
entry and not a gap to be filled with a neighbouring file's proof. Same residue `test_guard_inventory.py`
|
|
records for its own table, and for the same reason: the table is what review reads.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import copy
|
|
import re
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
import yaml
|
|
|
|
from scripts.tests.tracked_files import tracked_paths
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
|
|
SECTION = "Workflow-job guards (ersatztv#786)"
|
|
|
|
# From the GIT INDEX, never `Path.glob` (ersatztv#806): an untracked scratch workflow left in the
|
|
# directory would otherwise join the population on one checkout and not in CI — red locally, green on
|
|
# the server, which is #778's third shape. `*.yaml` alongside `*.yml` because Gitea accepts both.
|
|
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
|
|
|
|
ROLE_KEY = "CI_JOB_ROLE"
|
|
GUARD, ADVISORY, NONE = "guard", "advisory", "none"
|
|
# A VOCABULARY, not a population — the legal values of the marker, not a list of jobs.
|
|
ROLES = frozenset({GUARD, ADVISORY, NONE})
|
|
# role -> the Kind the row must carry. `none` is absent by construction: it takes no row.
|
|
ROLE_KIND = {GUARD: "GUARD", ADVISORY: "ADVISORY"}
|
|
|
|
_ROW = re.compile(r"^\|\s*`([a-z0-9.-]+\.ya?ml::[A-Za-z0-9_-]+)`\s*\|([^|]*)\|\s*([A-Z]+)\s*\|", re.M)
|
|
|
|
|
|
def workflow_files() -> list[Path]:
|
|
"""THE WORKFLOW POPULATION, from the git index. Named so the shared proof in
|
|
`test_guard_populations_derive_from_git.py` can assert it never admits an untracked file."""
|
|
return tracked_paths(*WORKFLOWS)
|
|
|
|
|
|
def declared_roles() -> dict[str, str | None]:
|
|
"""`<workflow>::<job>` -> the role it declares, or None where the marker is absent.
|
|
|
|
None is KEPT rather than dropped: an unmarked job is the defect `role_faults` reports, and a dict
|
|
that silently omitted it could not report anything.
|
|
"""
|
|
out: dict[str, str | None] = {}
|
|
for path in workflow_files():
|
|
doc = yaml.safe_load(path.read_text()) or {}
|
|
for jid, job in (doc.get("jobs") or {}).items():
|
|
if not isinstance(job, dict):
|
|
continue
|
|
value = (job.get("env") or {}).get(ROLE_KEY)
|
|
out[f"{path.name}::{jid}"] = str(value) if value is not None else None
|
|
return out
|
|
|
|
|
|
def role_faults(roles: dict[str, str | None]) -> list[str]:
|
|
"""MISSING OR UNKNOWN IS A HARD FAILURE.
|
|
|
|
A marker whose absence defaults to `none` would stop applying the moment someone adds a job and
|
|
forgets — and the job nobody remembered to mark is the job nobody reviewed. An unrecognised value
|
|
is rejected for the same reason rather than read as "not a guard".
|
|
"""
|
|
faults = []
|
|
for job, value in sorted(roles.items()):
|
|
if value is None:
|
|
faults.append(
|
|
f"job '{job}' declares no {ROLE_KEY}. Every job in every tracked workflow must "
|
|
f"declare one under its `env:` — {sorted(ROLES)} — so that 'not a guard' is a "
|
|
"recorded decision rather than an omission."
|
|
)
|
|
elif value not in ROLES:
|
|
faults.append(f"job '{job}' declares {ROLE_KEY}: {value!r}, which is not one of {sorted(ROLES)}.")
|
|
return faults
|
|
|
|
|
|
def inventory_section() -> str:
|
|
"""The body of the workflow-job section alone.
|
|
|
|
Scoped, not a whole-file scan: `docs/guard-inventory.md` holds several tables and the file-guard
|
|
one is read by `test_guard_inventory.py`. Two shape-keyed regexes over one document would each
|
|
pull in the other's rows and report them as phantoms. A missing heading raises rather than
|
|
returning empty — a silent empty section is the vacuous population this file exists to prevent.
|
|
"""
|
|
text = INVENTORY.read_text()
|
|
start = text.index(f"\n## {SECTION}\n")
|
|
nxt = text.find("\n## ", start + 1)
|
|
return text[start : nxt if nxt != -1 else len(text)]
|
|
|
|
|
|
def inventory_jobs() -> dict[str, str]:
|
|
"""`<workflow>::<job>` -> Kind, for each row of the workflow-job table."""
|
|
return {job.strip(): kind.strip() for job, _blocks, kind in _ROW.findall(inventory_section())}
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# ANTI-VACUITY FIRST — a population or a row regex that stopped matching would make every assertion
|
|
# below compare two empty sets and report a fully-covered inventory. That is the failure this file
|
|
# exists to prevent, so it is checked before anything depends on it.
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_workflow_population_is_not_empty():
|
|
files = workflow_files()
|
|
assert files, (
|
|
"no tracked workflow files were found — `tracked_paths` returned nothing. Every assertion "
|
|
"below would compare empty sets and pass having checked nothing."
|
|
)
|
|
roles = declared_roles()
|
|
assert roles, f"{len(files)} workflow file(s) parsed but no jobs were found in any of them"
|
|
|
|
|
|
def test_the_workflow_job_table_actually_parsed():
|
|
rows = inventory_jobs()
|
|
assert rows, (
|
|
f"the `## {SECTION}` table parsed to ZERO rows. Either the table is gone or `_ROW` no "
|
|
"longer matches its shape; both make the set equality below vacuous."
|
|
)
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# THE LIVE ASSERTIONS
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_every_job_declares_a_known_role():
|
|
faults = role_faults(declared_roles())
|
|
assert not faults, "workflow jobs with no usable role:\n " + "\n ".join(faults)
|
|
|
|
|
|
def test_the_inventory_covers_exactly_the_guard_JOBS_that_exist():
|
|
"""SET EQUALITY, both directions — the two failures are opposite mistakes.
|
|
|
|
Left-to-right: a job declared a guard with no row, which is the gap #786 was filed for.
|
|
Right-to-left: a row for a job that no longer declares itself a guard (renamed, deleted, or
|
|
downgraded), which is how a table becomes a comforting fiction.
|
|
"""
|
|
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
|
|
listed = set(inventory_jobs())
|
|
assert declared - listed == set(), (
|
|
f"these workflow jobs declare {ROLE_KEY} guard/advisory but have NO row in "
|
|
f"`## {SECTION}` — {sorted(declared - listed)}. Add a row saying what each blocks, where "
|
|
"its assertion lives and what proof it carries."
|
|
)
|
|
assert listed - declared == set(), (
|
|
f"`## {SECTION}` has rows for jobs that no longer declare a guard/advisory role — "
|
|
f"{sorted(listed - declared)}. Either the job was renamed or removed, or its marker "
|
|
"changed; the row must follow."
|
|
)
|
|
|
|
|
|
def test_each_row_carries_the_KIND_its_marker_declares():
|
|
"""The row and the workflow must not disagree about WHICH kind.
|
|
|
|
Set equality alone is satisfied by an `advisory` job holding a `GUARD` row — the table would then
|
|
claim a merge-blocking check where the workflow has one that cannot fail. `docs-reminder` is
|
|
exactly that job (`continue-on-error: true` on every check step), so the case is live, not
|
|
hypothetical.
|
|
"""
|
|
roles = declared_roles()
|
|
listed = inventory_jobs()
|
|
faults = [
|
|
f"'{job}' declares {ROLE_KEY}: {roles.get(job)} (Kind {ROLE_KIND.get(roles.get(job))}) but its row says {kind}"
|
|
for job, kind in sorted(listed.items())
|
|
if ROLE_KIND.get(roles.get(job)) != kind
|
|
]
|
|
assert not faults, "row Kind disagrees with the job's declared role:\n " + "\n ".join(faults)
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# MUTATION PROOFS — THE GUARD IS A TEST, so each mutation goes into the guarded ARTIFACT (the
|
|
# workflow, or the table) rather than into an assertion here. Disarming a checker makes it ABSENT
|
|
# rather than red; `testing.guard-ships-with-mutation-proof`'s checker-guard exception is what makes
|
|
# introducing the real defect the admissible proof.
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def _role_mutants():
|
|
for job in sorted(declared_roles()):
|
|
removed = dict(declared_roles())
|
|
removed[job] = None
|
|
yield f"{job}-marker-removed", removed
|
|
|
|
unknown = dict(declared_roles())
|
|
unknown[job] = "checker"
|
|
yield f"{job}-marker-unknown-value", unknown
|
|
|
|
|
|
_ROLE_MUTANTS = list(_role_mutants())
|
|
|
|
|
|
@pytest.mark.parametrize("roles", [m for _, m in _ROLE_MUTANTS], ids=[i for i, _ in _ROLE_MUTANTS])
|
|
def test_a_missing_or_UNKNOWN_role_is_DETECTED(roles):
|
|
assert role_faults(roles), "a job with no role, or an unrecognised one, was accepted"
|
|
|
|
|
|
def test_the_role_mutation_set_is_not_empty():
|
|
expected = 2 * len(declared_roles())
|
|
assert len(_ROLE_MUTANTS) == expected, f"expected 2 role mutations per job ({expected}), got {len(_ROLE_MUTANTS)}"
|
|
|
|
|
|
def test_MUTATION_a_guard_job_with_its_row_DELETED_is_reported():
|
|
"""The defect this guard exists to catch: a guard job that acquired no row.
|
|
|
|
Mutating the guarded ARTIFACT (the table), not this checker's population — a shrunken population
|
|
would make every real row report as a PHANTOM, demonstrating a false POSITIVE while proving
|
|
nothing about the missing-row detection the inventory row claims.
|
|
"""
|
|
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
|
|
listed = set(inventory_jobs())
|
|
assert declared and listed, "nothing to mutate — the population or the table is empty"
|
|
for victim in sorted(listed):
|
|
mutated = listed - {victim}
|
|
assert declared - mutated, f"deleting the row for '{victim}' was not reported as missing"
|
|
|
|
|
|
def test_MUTATION_a_PHANTOM_row_is_reported():
|
|
"""The opposite direction: a row for a job that does not declare a guard role."""
|
|
declared = {j for j, r in declared_roles().items() if r in ROLE_KIND}
|
|
mutated = set(inventory_jobs()) | {"docker-build.yml::a-job-that-was-deleted"}
|
|
assert mutated - declared, "a row naming a non-guard job was not reported as a phantom"
|
|
|
|
|
|
def test_MUTATION_a_row_whose_KIND_disagrees_is_reported():
|
|
"""`advisory` holding a `GUARD` row must not pass — the case `docs-reminder` makes live."""
|
|
roles = declared_roles()
|
|
advisory = [j for j, r in roles.items() if r == ADVISORY]
|
|
assert advisory, "no advisory job exists, so this proof is vacuous"
|
|
listed = dict(inventory_jobs())
|
|
listed[advisory[0]] = "GUARD"
|
|
faults = [j for j, kind in listed.items() if ROLE_KIND.get(roles.get(j)) != kind]
|
|
assert faults, "an advisory job carrying a GUARD row was accepted"
|
|
|
|
|
|
def test_a_workflow_job_added_without_a_marker_is_DETECTED():
|
|
"""A NEW job is the realistic arrival path, and it must not default to `none`."""
|
|
roles = dict(declared_roles())
|
|
roles["docker-build.yml::a-brand-new-job"] = None
|
|
assert role_faults(roles), "a newly added job with no role marker was accepted"
|
|
|
|
|
|
def test_the_marker_is_read_from_the_PARSED_yaml_not_a_text_search():
|
|
"""Guards against the parser regressing to a grep.
|
|
|
|
A text search for `CI_JOB_ROLE: guard` would count this file's own prose, the inventory's, and a
|
|
commented-out marker. Asserting the parse produces exactly the tracked workflows' jobs is what
|
|
keeps the population a property of the YAML.
|
|
"""
|
|
roles = declared_roles()
|
|
for path in workflow_files():
|
|
doc = yaml.safe_load(path.read_text()) or {}
|
|
for jid in doc.get("jobs") or {}:
|
|
assert f"{path.name}::{jid}" in roles, f"{path.name}::{jid} was not parsed"
|
|
assert all("::" in k for k in roles)
|
|
|
|
|
|
def test_a_commented_out_marker_does_NOT_count():
|
|
"""The concrete form of the above: YAML comments are not values."""
|
|
doc = yaml.safe_load("jobs:\n x:\n env:\n # CI_JOB_ROLE: guard\n OTHER: 1\n steps: []\n")
|
|
job = doc["jobs"]["x"]
|
|
assert (job.get("env") or {}).get(ROLE_KEY) is None
|
|
mutated = copy.deepcopy(doc)
|
|
assert (mutated["jobs"]["x"].get("env") or {}).get(ROLE_KEY) is None
|