timothyandClaude Opus 5
3ec73f3769
Round 6 found three more false greens and named the class they share, which is worth
more than any of the three fixes:
* `web/vitest.config.ts` OUTRANKS the pinned `vite.config.ts` — closed in the previous
commit, found by probing vitest rather than reading about it.
* A DECOY first `test: {` block. The comparison took `text.index("test: {")`, so a copy
of the pin placed above `defineConfig` satisfied it while the real block was narrowed.
Exactly one is now required — the same assertion this file already made about the
gating step's NAME, for the same reason, not carried across.
* A `needs:` edge matched by bare job id. `needs:` resolves within its own workflow, so
a SECOND workflow publishing this Dockerfile while needing its own unrelated job
called `test` satisfied it. Now bound to `GATING_WORKFLOW`. (The reviewer downgraded
this to MEDIUM on measuring that `test_remote_state_inventory.py` forces a human to
classify any new workflow — so the hole is "the guard is blind", not "silent". The
forced review asks about remote state, not about whether the image is gated, so the
one-line fix stands.)
* A vite PLUGIN can shell out to the suite from `buildStart()`. The plugin ARRAY is
pinned; the plugin BODIES are a stated residual, mitigated because
`trackedSourceFilesPlugin` is deliberately lazy — a fact its own comment now marks as
LOAD-BEARING for the image build rather than leaving as an optimisation note.
THE CLASS: **a pin assumes it is pinning the artifact that still decides.** Every route
found so far is authority moving where the pin is not looking — to another FILE, another
OCCURRENCE in the same file, another WORKFLOW, or a HOOK the pinned command invokes. That
question is now written down for the next person adding a pin, because a list of four
instances is not what generalises.
Prose, all refuted by execution: the residual naming the uncovered COPY shapes was wrong a
THIRD time at the same site (`/source/web /elsewhere` IS recognised — only the destination
is renamed — and the file's own test 700 lines below said so); "only an `ENV` is
unmodelled" was an absolute and is now a list; "Reach: N mutants, 0 missed" is restated as
a DEVELOPMENT BATTERY, since it is not in the repo, nothing re-derives it, and an
independent battery found misses against an earlier head; and `PUBLISH_ACTION` was claimed
covered by anti-vacuity, which proves the selector is non-empty and cannot prove it
complete.
Battery 61 -> 64, 0 missed.
Refs: #887
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019T79beF1Ufid3dXju4yqkF
1006 lines
54 KiB
Python
1006 lines
54 KiB
Python
"""The SPA vitest suite never runs inside an image build, and the image is gated on a step that does
|
|
run it, unfiltered and non-advisory (ersatztv#887).
|
|
|
|
WHAT THIS IS PROTECTING. `docker/Dockerfile`'s `web-build` stage is gitless twice over. It copies
|
|
only `web/` and `design-system/`, so it holds no `.git` — a property of the STAGE, not of the build
|
|
context, which is the repository root (`context: .`) and whose `.dockerignore` does not exclude
|
|
`.git`: the directory is there to be copied and simply is not. And `node:22-bookworm-slim` ships no
|
|
git binary, which is why a `COPY .git` would not help either. Members of the SPA suite need one or
|
|
the other — `pageSizeCallSites.guard.test.ts` and `completeAnnotations.guard.test.ts` derive their
|
|
file population from `git ls-files` and refuse to fall back to a directory walk
|
|
(`testing.guard-derives-population-from-source`), and `trackedSourceFiles.realgit.test.ts` builds a
|
|
real temporary repository.
|
|
|
|
Running the suite there anyway therefore requires a list of the members that cannot run, and that
|
|
list is a population nothing derives: ersatztv#883 added a third member without updating a
|
|
hand-written pair of `--exclude`s, and every image build failed from that commit on. The red was
|
|
structurally unreachable on a PR — `Build & push image (amd64)` carries
|
|
`if: github.event_name != 'pull_request'` — so it landed on `main` and on the `v*` tag path, where a
|
|
release cut fails at the image build and `:latest` stops being republished.
|
|
|
|
## WHY THIS GUARD PINS TEXT INSTEAD OF PARSING IT
|
|
|
|
The first three versions of this file asked "does this command RUN the suite, and can it FAIL?" of
|
|
arbitrary shell text. That predicate was wrong nine times across three review rounds, and every one
|
|
of the nine was the same mechanism:
|
|
|
|
* heredoc bodies — skipped as data, but BuildKit EXECUTES `RUN <<EOF`, and the opener regex also
|
|
fired inside quotes (`echo "tags<<__EOT__"`), which blinded the scan over the last 303 lines of
|
|
`docker-build.yml`. Wrong in BOTH directions at once.
|
|
* `shlex.shlex` does not clear `commenters` the way `shlex.split` does, so `#` truncated a command
|
|
mid-word — including the live `${#reports[@]}` idiom — and made this file's own stated residual
|
|
("only a LEADING `#` is stripped") false.
|
|
* compound punctuation (`);`, `;;`) welded two commands into one segment.
|
|
* `npm t`, `./node_modules/.bin/vitest`, `pnpm vitest`, `yarn vitest`, `node …/vitest.mjs`,
|
|
`timeout …`, `su -c …`, `if npm test; then` — all invisible.
|
|
* `true || npm test` counted as the gating run while never executing.
|
|
* `continue-on-error: ${{ … }}` passed a check that compared against two literals.
|
|
|
|
Nine defects from one mechanism is `process.enumerate-workaround-behaviors-before-deleting`'s signal
|
|
to remove it rather than patch a tenth time, and the mechanism is removed here. **Nothing in this
|
|
file decides what a command MEANS.** The commands that may run in the risky places are PINNED as
|
|
text; anything else — a heredoc, a wrapper, a new spelling, a rewrite — is simply not equal to the
|
|
pin and reddens.
|
|
|
|
WHAT A PIN IS AND IS NOT IMMUNE TO, stated precisely because an earlier version of this paragraph
|
|
claimed the absolute "a pin cannot have a false green" and cold review refuted it by execution. A
|
|
pin is immune to the class that defeated the parser: a DIFFERENT SPELLING of the command. It is not
|
|
immune to the same text MEANING something else. EIGHT such routes have been IDENTIFIED, and that
|
|
number is a running total, not a proof: it was written as five, then six, then seven, and each was
|
|
wrong when checked. Six are handled — `working-directory` and the gating step's own `shell:`; the workflow
|
|
`defaults.run.shell` and the job-level `defaults` that would override it; a stage `SHELL`, refused
|
|
outright; `web/package.json`'s script map, pinned WHOLE after selecting on the literal `vitest`
|
|
missed `npm run test`, `npm t`, and the `prebuild`/`preinstall` LIFECYCLE HOOKS npm runs for
|
|
`npm run build` and `npm ci`; `web/vite.config.ts`'s `test:` block, which decides what the suite
|
|
COLLECTS and is where a filter would now naturally be written; the ABSENCE of any
|
|
`vitest.config.*`/`vitest.workspace.*`/`vitest.projects.*`, which vitest resolves in PREFERENCE to
|
|
that file — pinning one config is worthless while a second can outrank it, and the measured effect
|
|
of adding one was a run that collected zero specs and exited 0, which is worse than a filtered one;
|
|
and the gating job's `container:` image, pinned by `test_ci_image_pin_population.py`
|
|
and CITED rather than re-asserted, because two guards on one condition mask each other (#685).
|
|
|
|
The eighth, an `ENV` rewriting `PATH` so a pinned `RUN` resolves a different `npm`, is not modelled
|
|
and is stated as a residual.
|
|
|
|
So: the pin removes the SPELLING problem outright and reduces the MEANING problem to a list that can
|
|
be worked through — a much better place than a predicate that was wrong nine times. It is not
|
|
"cannot be wrong", and the list's own history says so.
|
|
|
|
THE SHAPE THOSE ROUTES SHARE, which is the thing to check when adding a pin rather than a list to
|
|
memorise: **a pin assumes it is pinning the artifact that still DECIDES.** Every instance found so
|
|
far is authority moving somewhere the pin is not looking — to another FILE (`vitest.config.*`
|
|
outranks `vite.config.ts`), to another OCCURRENCE in the same file (a decoy first `test: {`), to
|
|
another WORKFLOW (a `needs:` edge naming a job called `test` that is not this one), or to a HOOK the
|
|
pinned command invokes (a vite plugin, an npm lifecycle script). Ask of any new pin: what else could
|
|
decide this, and would the pin still match?
|
|
|
|
The cost is stated rather than hidden: a legitimate edit to those command lines reddens this guard
|
|
and its author must update the pin. That is the same deliberate review trigger `KNOWN_GATE_*` uses,
|
|
applied to the two places where a silent change publishes an untested image.
|
|
|
|
## THE INVARIANT, in the three parts it has
|
|
|
|
* NEGATIVE — every Dockerfile STAGE that carries the SPA source runs exactly the pinned commands.
|
|
A suite run put back in any spelling is not equal to the pin.
|
|
* POSITIVE — every job publishing an image from an SPA-carrying Dockerfile transitively `needs:`
|
|
the job holding the pinned gating step.
|
|
* EFFECTIVE — that step's `run:` body, its `if:`, and the publishing step's own docs-only `if:`
|
|
are pinned; the step and its job carry no `continue-on-error` in ANY spelling; and the job
|
|
carries no job-level `if:`.
|
|
|
|
POPULATIONS, all derived from the git index via `tracked_files` — never a filesystem walk and never
|
|
a hand-written list (`testing.guard-derives-population-from-source`, ersatztv#806). The PINS are
|
|
hand-written by necessity, and the difference matters: a population decides WHAT IS CHECKED (so a
|
|
hand-written one goes silently short), a pin decides WHAT IS EXPECTED (so a stale one goes loudly
|
|
red). Only the second is safe to write by hand.
|
|
|
|
WHAT IS NOT COVERED, stated rather than implied:
|
|
|
|
* A Dockerfile stage that does not carry the SPA source is unpinned. The boundary is what
|
|
`copies_spa_source` recognises — a context copy of `web/` or `.`, a matching GLOB, or a stage
|
|
copy whose source names `web` — and NOT the wider "has the suite available to run", which is
|
|
what an earlier version of this sentence claimed. A stage copy that RENAMES the tree on the way
|
|
in (`COPY --from=web-build /source/web /elsewhere`) has no `web` segment in its source and is
|
|
not recognised. THAT CLAUSE WAS WRONG THREE TIMES and is now stated from a measurement rather
|
|
than from memory: `COPY --from=web-build /source/web /elsewhere` IS recognised (only the
|
|
DESTINATION is renamed, and only sources are read). What is not: a source that is an ANCESTOR of
|
|
the tree (`COPY --from=web-build /source /src` brings `/source/web` along without naming it) or
|
|
names it only as `/source/.`. Both measured, and both backstopped by the substring sweep — an
|
|
ancestor-copy stage running `npm t`, `npx vitest` or a new npm script is caught. The
|
|
false-NEGATIVE direction is what drops a stage out of the check, so the predicate is pinned in
|
|
both directions by its own test, and the substring sweep below is the backstop for whatever it
|
|
still misses.
|
|
* A step or stage OTHER than a pinned one could run the suite, and the substring sweep is what
|
|
reports it. `SUITE_MENTIONS` is a hand-written SELECTOR — a third category the population/pin
|
|
split does not name, and the one that behaves worst: a population going short is caught by an
|
|
equality, a pin going stale goes loudly red, but a SELECTOR going short is silent. It was short
|
|
by exactly one entry (`npm t`, npm's own alias, which this file names among the spellings that
|
|
defeated the parser) until cold review measured a stage escaping through it. `PUBLISH_ACTION` is
|
|
the same category and is covered by the anti-vacuity assertion instead.
|
|
* The gating body is compared LINE BY LINE with each line whitespace-normalised, not byte-exact.
|
|
Indentation and reflow are free; line boundaries are not, because a newline separates two
|
|
commands — changing `run: |` to `run: >` folds them into one and the whole-text comparison could
|
|
not see it.
|
|
* Gitea's own evaluation of a skipped `needs:` job is not modelled; the guard forbids the job-level
|
|
`if:` that would create one rather than reasoning about it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import re
|
|
from fnmatch import fnmatch
|
|
from pathlib import Path
|
|
|
|
import yaml
|
|
|
|
from scripts.tests import tracked_files
|
|
from scripts.tests.tracked_files import tracked_paths
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
WORKFLOW_DIR = ".gitea/workflows"
|
|
WORKFLOWS = (WORKFLOW_DIR, ("*.yml", "*.yaml"))
|
|
|
|
# The action every image publish in this repo goes through. Matched on the last two path segments
|
|
# for the reason `test_workflow_persist_credentials.py` gives: Gitea accepts a full action URL, and
|
|
# a host is not required to contain a dot, so no hostname heuristic is safe.
|
|
PUBLISH_ACTION = "docker/build-push-action"
|
|
|
|
# Instructions that bring files from the build context into the image.
|
|
COPY_INSTRUCTIONS = frozenset({"ADD", "COPY"})
|
|
|
|
# ---------------------------------------------------------------------------------------------
|
|
# THE PINS. Update these deliberately, in the same commit as the change they describe, and say why.
|
|
# ---------------------------------------------------------------------------------------------
|
|
|
|
# Every `RUN` in a Dockerfile stage that carries the SPA source, keyed `<path>::<stage>`. Whitespace
|
|
# is normalised (see `_normalise`) so a reflow is not a red, but the tokens are exact.
|
|
PINNED_STAGE_COMMANDS: dict[str, tuple[str, ...]] = {
|
|
"docker/Dockerfile::web-build": (
|
|
"RUN npm ci",
|
|
# Lints, typechecks and BUILDS the SPA. It must not TEST it: this stage has neither a git
|
|
# checkout nor the git binary, and naming the specs that cannot run there is the
|
|
# hand-maintained population ersatztv#887 removed.
|
|
"RUN npm run lint && npm run typecheck && npm run build",
|
|
),
|
|
}
|
|
|
|
# The step that runs the suite and gates the image, pinned whole. Its body being exact is what makes
|
|
# "unfiltered", "actually executes", and "cannot be suppressed" true without parsing any of it.
|
|
GATING_WORKFLOW = ".gitea/workflows/docker-build.yml"
|
|
GATING_JOB = "test"
|
|
GATING_STEP_NAME = "Test SPA"
|
|
GATING_STEP_RUN = '"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark web-test\nnpm test -- --run'
|
|
# `docs_only` (ersatztv#416) and the #420 revalidate skip. Both are safe for a DIFFERENT reason and
|
|
# the difference is why this is pinned rather than pattern-matched: docs-only cannot ship an image at
|
|
# all, which is asserted below rather than assumed; the revalidate skip fires only on a tree
|
|
# byte-identical to a head that already carried a green combined status.
|
|
GATING_STEP_IF = "steps.detect.outputs.docs_only != 'true' && steps.revalidate.outputs.skip != 'true'"
|
|
# `npm test` means nothing without the directory it runs in: pointed elsewhere it runs a different
|
|
# package, or none. Part of the pin because it is part of what the step does.
|
|
GATING_STEP_WORKDIR = "web"
|
|
# `shell: bash` is `bash -e`, which is what makes a failing command fail the step. A workflow-level
|
|
# change here alters that for the gating step without touching the step at all.
|
|
GATING_WORKFLOW_SHELL = "bash"
|
|
|
|
# The premise under the `docs_only` half of `GATING_STEP_IF`: the publish step is gated on it too, so
|
|
# the arm that skips the suite ships nothing. Round-1 review found this stated as fact with nothing
|
|
# checking it, one level out from the same defect.
|
|
PUBLISH_STEP_NAME = "Build and push"
|
|
PUBLISH_STEP_IF = "steps.detect.outputs.docs_only != 'true'"
|
|
|
|
# `RUN npm run build` and `RUN npm ci` execute whatever `web/package.json` says they do, and that
|
|
# file is neither a Dockerfile nor a workflow. Cold review re-armed ersatztv#887 through it twice
|
|
# without touching any pinned line: `"build": "vitest run && tsc -b && vite build"` puts the suite
|
|
# back into the gitless stage, and `"prepare"` is run by `npm ci`. So the SCRIPTS THAT MENTION VITEST
|
|
# are pinned too — exactly one, and its body.
|
|
PACKAGE_JSON = "web/package.json"
|
|
# THE WHOLE MAP, not the scripts that mention vitest. Selecting on the literal `vitest` was a
|
|
# SELECTOR where a PIN was available, and cold review re-armed ersatztv#887 through four spellings
|
|
# none of which contain it: `"build": "npm run test -- --run && …"`, `"build": "npm t -- …"`, and the
|
|
# lifecycle hooks `prebuild` (npm runs it before `npm run build`) and `preinstall` (before `npm ci`).
|
|
# Both pinned RUN lines in the gitless stage execute whatever this map says, so the map is pinned.
|
|
PINNED_PACKAGE_SCRIPTS: dict[str, str] = {
|
|
"dev": "vite --host 0.0.0.0",
|
|
"build": "tsc -b && vite build",
|
|
"generate:api": "node scripts/generate-openapi-types.mjs",
|
|
"check:api": "npm run generate:api && git diff --exit-code -- src/api/generated/v1.d.ts",
|
|
"lint": "eslint .",
|
|
"test": "vitest",
|
|
"test:ui-e2e": "playwright test",
|
|
"typecheck": "tsc -b --pretty false",
|
|
"prepare": "cd .. && husky",
|
|
}
|
|
|
|
# `npm test -- --run` collects whatever this config says it does, so `test.exclude`/`include` is the
|
|
# remaining place a filter can be written — and, since this change removes the Dockerfile
|
|
# `--exclude`s, the FIRST place anyone reaching for one would now look. The `test:` block is pinned
|
|
# for the same reason the script map is.
|
|
VITE_CONFIG = "web/vite.config.ts"
|
|
# Vitest resolves its OWN config in preference to `vite.config.*`, so a `web/vitest.config.ts` beside
|
|
# the pinned file silently replaces the block below. MEASURED: adding one with
|
|
# `include: ['nope/**'], passWithNoTests: true` made `npx vitest run` report
|
|
# "No test files found, exiting with code 0" — the gating step green having run NOTHING, and the
|
|
# image published. Pinning one file is worthless while a second can outrank it, so the absence of
|
|
# every higher-precedence spelling is asserted rather than assumed.
|
|
VITEST_CONFIG_PREFIXES = ("vitest.config.", "vitest.workspace.", "vitest.projects.")
|
|
# A vite plugin runs inside the same process as the suite and can shell out from a build hook, so a
|
|
# `buildStart()` calling `npm test` re-arms ersatztv#887 from the plugin side with every other pin
|
|
# matching (measured). The ARRAY is pinned so no plugin joins silently; the plugin BODIES are not,
|
|
# and that is a stated residual — `trackedSourceFiles.ts` is deliberately LAZY for this exact
|
|
# reason, which its own comment now marks as load-bearing for the image build.
|
|
PINNED_VITE_PLUGINS = "plugins: [react(), trackedSourceFilesPlugin()],"
|
|
PINNED_VITE_TEST_BLOCK = """test: {
|
|
environment: 'jsdom',
|
|
environmentOptions: {
|
|
jsdom: {
|
|
url: 'http://localhost/app/'
|
|
}
|
|
},
|
|
exclude: [...configDefaults.exclude, 'e2e/**'],
|
|
setupFiles: './src/setupTests.ts'
|
|
}"""
|
|
|
|
# Substrings that mean "a human should look at this step". NOT a semantic predicate — the pinned step
|
|
# is allowed to contain them and every other step is not.
|
|
SUITE_MENTIONS = ("npm t", "npm run test", "yarn test", "pnpm test", "bun test", "vitest")
|
|
# `npm t` rather than `npm test`: it is npm's own documented alias, it SUBSUMES `npm test` as a
|
|
# substring, and this file enumerates it among the spellings that defeated the parser. Leaving it out
|
|
# meant an unpinned stage running `npm t -- --run` passed the sweep — measured by cold review.
|
|
|
|
|
|
def _normalise(text: str) -> str:
|
|
"""Collapse runs of whitespace so a reflow is not a red, keeping every token exact."""
|
|
return " ".join(text.split())
|
|
|
|
|
|
def _vite_test_block() -> list[str]:
|
|
"""The `test: {` … matching `}` block of `web/vite.config.ts`, normalised per line.
|
|
|
|
Brace matching from a single marker — the same class of syntax handling as splitting a Dockerfile
|
|
on `FROM`: unambiguous, and it interprets nothing. Comment lines are dropped so the block can be
|
|
explained without the explanation being pinned.
|
|
"""
|
|
text = _read(VITE_CONFIG)
|
|
occurrences = text.count("test: {")
|
|
# A decoy `const reviewed = { test: {…} }` copied byte-for-byte from the pin, placed ABOVE
|
|
# `defineConfig`, satisfies a comparison that takes the FIRST match while the real block is
|
|
# narrowed — measured green. The file already requires exactly ONE step named `Test SPA` for the
|
|
# same reason; the reasoning had not been applied here.
|
|
assert occurrences == 1, (
|
|
f"{VITE_CONFIG} contains {occurrences} `test: {{` blocks, not one. The pin compares the "
|
|
f"FIRST, so a second is either a decoy that satisfies it while another block decides, or a "
|
|
f"real second config nobody is checking."
|
|
)
|
|
start = text.index("test: {")
|
|
depth = 0
|
|
for index in range(start, len(text)):
|
|
if text[index] == "{":
|
|
depth += 1
|
|
elif text[index] == "}":
|
|
depth -= 1
|
|
if depth == 0:
|
|
block = text[start : index + 1]
|
|
return [
|
|
_normalise(line)
|
|
for line in block.splitlines()
|
|
if line.strip() and not line.strip().startswith(("//", "*", "/*"))
|
|
]
|
|
raise AssertionError(f"{VITE_CONFIG}: the `test: {{` block is not brace-balanced")
|
|
|
|
|
|
def _normalise_lines(text: str) -> list[str]:
|
|
"""Per-line normalisation, so indentation is free but the LINE BOUNDARIES are not.
|
|
|
|
A newline separates two commands; collapsing it merges them into one. That is not a cosmetic
|
|
difference and the whole-text form cannot see it.
|
|
"""
|
|
return [_normalise(line) for line in text.strip().splitlines() if line.strip()]
|
|
|
|
|
|
def _canonical_action(uses: str) -> str:
|
|
"""The last two path segments of a `uses:` value, without its `@version`.
|
|
|
|
Same normalisation as `test_workflow_persist_credentials._canonical_action`, and deliberately a
|
|
second copy rather than an import: that guard's population is checkout steps, this one's is
|
|
publish steps, and sharing the helper would mean a narrowing there silently narrows here too.
|
|
"""
|
|
ref = uses.strip().strip("'\"").rsplit("@", 1)[0].strip().lower()
|
|
ref = re.sub(r"^https?://", "", ref)
|
|
segments = [seg for seg in ref.split("/") if seg]
|
|
if segments:
|
|
segments[-1] = re.sub(r"\.git$", "", segments[-1])
|
|
return "/".join(segments[-2:]) if len(segments) >= 2 else ref
|
|
|
|
|
|
def dockerfiles() -> list[str]:
|
|
"""Tracked Dockerfiles, by repo-relative path, in every spelling docker recognises.
|
|
|
|
`tracked_files._git_ls_files()` through the MODULE, never a `from … import` binding. The shared
|
|
proof in `test_guard_populations_derive_from_git.py` narrows the index by patching that module
|
|
attribute, and a name bound at import time is not reachable from there — the derivation would go
|
|
on returning every Dockerfile after git stopped tracking them, and the proof would say so.
|
|
"""
|
|
return sorted(
|
|
path
|
|
for path in tracked_files._git_ls_files()
|
|
if path.rpartition("/")[2].startswith("Dockerfile") or path.endswith(".Dockerfile")
|
|
)
|
|
|
|
|
|
def workflow_files() -> list[Path]:
|
|
return tracked_paths(*WORKFLOWS)
|
|
|
|
|
|
def _read(rel: str) -> str:
|
|
return (REPO_ROOT / rel).read_text(encoding="utf-8")
|
|
|
|
|
|
def instructions(text: str) -> list[str]:
|
|
"""The Dockerfile instructions, comments dropped and `\\` continuations joined.
|
|
|
|
This is the whole of the file's syntax handling. It does not interpret what a `RUN` DOES — the
|
|
string is compared against a pin — so the shell traps that defeated three earlier versions
|
|
(heredocs, `#` inside a command, compound punctuation) cannot reach any decision here.
|
|
"""
|
|
joined: list[str] = []
|
|
buffer = ""
|
|
for raw in text.splitlines():
|
|
if raw.lstrip().startswith("#"):
|
|
continue
|
|
stripped = raw.rstrip()
|
|
if stripped.endswith("\\"):
|
|
buffer += stripped[:-1] + " "
|
|
continue
|
|
buffer += stripped
|
|
if buffer.strip():
|
|
joined.append(_normalise(buffer))
|
|
buffer = ""
|
|
if buffer.strip():
|
|
joined.append(_normalise(buffer))
|
|
return joined
|
|
|
|
|
|
def stages(text: str) -> list[tuple[str, list[str]]]:
|
|
"""`(stage name, its instructions)` per build stage, in file order.
|
|
|
|
A stage is delimited by `FROM`, which is the one Dockerfile construct with no shell content and
|
|
therefore nothing to misparse. An unnamed stage is keyed by its index so it is still checkable.
|
|
"""
|
|
found: list[tuple[str, list[str]]] = []
|
|
for line in instructions(text):
|
|
if re.match(r"^FROM\b", line, re.I):
|
|
match = re.search(r"\bAS\s+(\S+)\s*$", line, re.I)
|
|
found.append((match.group(1) if match else f"<unnamed:{len(found)}>", []))
|
|
elif found:
|
|
found[-1][1].append(line)
|
|
return found
|
|
|
|
|
|
def copies_spa_source(command: str) -> bool:
|
|
"""Does this ONE instruction bring the SPA source tree into the image?
|
|
|
|
Every spelling counts, because the negative assertion SKIPS every stage answered False — a miss
|
|
here drops a stage out of the check silently rather than reddening. `ADD` as well as `COPY`; the
|
|
JSON exec form `COPY ["web/", "/dest/"]`, with or without a preceding flag; a source without a
|
|
trailing slash (`COPY web /dest`); and `COPY . …`, which carries the tree without naming it.
|
|
|
|
A `COPY --from=<stage>` is not a context copy: its source is another stage, so `--from=build .`
|
|
would not bring the repo in. Flags are dropped before the source arguments are read.
|
|
"""
|
|
tokens = command.split()
|
|
if not tokens or tokens[0].upper() not in COPY_INSTRUCTIONS:
|
|
return False
|
|
remainder = " ".join(token for token in tokens[1:] if not token.startswith("--")).strip()
|
|
if any(token.startswith("--from=") for token in tokens[1:]):
|
|
# A stage copy is not a CONTEXT copy, so `--from=build .` brings in that stage's `/`, not the
|
|
# repository. But it can still carry the SPA SOURCE across from a stage that has it, and a
|
|
# stage answered False here is UNPINNED and unchecked — measured: `COPY --from=web-build
|
|
# /source/web /source/web` was False, so a suite run in the receiving stage was invisible.
|
|
# The test is a whole `web` path SEGMENT in the source, which distinguishes the source tree
|
|
# from the built artifact this repo actually copies (`/source/ErsatzTV/wwwroot/app/.` has no
|
|
# such segment).
|
|
return any("web" in source.strip("/").split("/") for source in remainder.split()[:-1])
|
|
if remainder.startswith("["):
|
|
try:
|
|
parsed = json.loads(remainder)
|
|
except ValueError:
|
|
# An exec-form array this cannot read is reported as CARRYING the source: that makes the
|
|
# stage pinned rather than skipped, which is the safe direction for an unknown.
|
|
return True
|
|
arguments = [str(item) for item in parsed] if isinstance(parsed, list) else []
|
|
else:
|
|
arguments = remainder.split()
|
|
if len(arguments) < 2:
|
|
return False
|
|
return any(_source_reaches_web(source) for source in arguments[:-1])
|
|
|
|
|
|
def _source_reaches_web(source: str) -> bool:
|
|
"""Could this COPY source bring `web/` in?
|
|
|
|
Exact (`web`, `web/…`), whole-context (`.`, `./`), and GLOB: `COPY web*/ ./` matches `web/` and
|
|
so carries the source, while `webhooks/` does not — measured False for the glob by cold review,
|
|
which left the receiving stage unpinned. `fnmatch` against the literal segment `web` is what
|
|
separates the two without hand-listing patterns.
|
|
"""
|
|
normalised = source.removeprefix("./").lstrip("/").rstrip("/")
|
|
if normalised in {"", ".", "web"} or normalised.startswith("web/"):
|
|
return True
|
|
head = normalised.split("/")[0]
|
|
return any(char in head for char in "*?[") and fnmatch("web", head)
|
|
|
|
|
|
def spa_carrying_stages() -> list[tuple[str, str, list[str]]]:
|
|
"""`(dockerfile, stage name, RUN commands)` for every stage carrying the SPA source."""
|
|
found: list[tuple[str, str, list[str]]] = []
|
|
for rel in dockerfiles():
|
|
for name, lines in stages(_read(rel)):
|
|
if any(copies_spa_source(line) for line in lines):
|
|
found.append((rel, name, [line for line in lines if re.match(r"^RUN\b", line)]))
|
|
return found
|
|
|
|
|
|
def stage_shell_overrides() -> list[str]:
|
|
"""Every `SHELL` instruction inside a pinned stage, which the pin alone cannot see.
|
|
|
|
`SHELL ["/bin/sh", "-c"]` redefines what every later `RUN` in the stage EXECUTES while leaving
|
|
the `RUN` text — and therefore the pin — untouched. Reported rather than modelled: there is no
|
|
`SHELL` in this repo, so the honest move is to refuse the construct here rather than reason about
|
|
what a replacement interpreter would do.
|
|
"""
|
|
overrides: list[str] = []
|
|
for rel in dockerfiles():
|
|
for name, lines in stages(_read(rel)):
|
|
if f"{rel}::{name}" not in PINNED_STAGE_COMMANDS:
|
|
continue
|
|
overrides.extend(f"{rel}::{name}: {line!r}" for line in lines if re.match(r"^SHELL\b", line))
|
|
return overrides
|
|
|
|
|
|
def _jobs(doc: object) -> dict[str, dict]:
|
|
jobs = doc.get("jobs") if isinstance(doc, dict) else None
|
|
if not isinstance(jobs, dict):
|
|
return {}
|
|
return {str(key): value for key, value in jobs.items() if isinstance(value, dict)}
|
|
|
|
|
|
def _steps(job: dict) -> list[dict]:
|
|
steps = job.get("steps")
|
|
return [step for step in steps if isinstance(step, dict)] if isinstance(steps, list) else []
|
|
|
|
|
|
def _needs(job: dict) -> list[str]:
|
|
needs = job.get("needs")
|
|
if isinstance(needs, str):
|
|
return [needs]
|
|
return [str(item) for item in needs] if isinstance(needs, list) else []
|
|
|
|
|
|
def _transitive_needs(job_id: str, jobs: dict[str, dict]) -> set[str]:
|
|
seen: set[str] = set()
|
|
frontier = list(_needs(jobs.get(job_id, {})))
|
|
while frontier:
|
|
current = frontier.pop()
|
|
if current in seen:
|
|
continue
|
|
seen.add(current)
|
|
frontier.extend(_needs(jobs.get(current, {})))
|
|
return seen
|
|
|
|
|
|
def workflow_jobs(rel: str) -> dict[str, dict]:
|
|
return _jobs(yaml.safe_load(_read(rel)))
|
|
|
|
|
|
def find_step(rel: str, job_id: str, name: str) -> dict | None:
|
|
for step in _steps(workflow_jobs(rel).get(job_id, {})):
|
|
if str(step.get("name", "")).strip() == name:
|
|
return step
|
|
return None
|
|
|
|
|
|
def publishing_jobs() -> list[tuple[str, str, str, dict[str, dict]]]:
|
|
"""`(workflow rel path, job id, Dockerfile rel path, that workflow's jobs)` per publishing job.
|
|
|
|
The Dockerfile comes from the step's own `file:` input, defaulting to docker's own default
|
|
(`Dockerfile` at the context root) when the input is absent, so a step that stops naming one is
|
|
still resolved rather than dropped from the population.
|
|
"""
|
|
found: list[tuple[str, str, str, dict[str, dict]]] = []
|
|
for path in workflow_files():
|
|
rel = path.relative_to(REPO_ROOT).as_posix()
|
|
jobs = workflow_jobs(rel)
|
|
for job_id, job in jobs.items():
|
|
for step in _steps(job):
|
|
uses = step.get("uses")
|
|
if not isinstance(uses, str) or _canonical_action(uses) != PUBLISH_ACTION:
|
|
continue
|
|
with_block = step.get("with")
|
|
named = with_block.get("file") if isinstance(with_block, dict) else None
|
|
dockerfile = str(named or "Dockerfile").strip()
|
|
found.append((rel, job_id, re.sub(r"^\./", "", dockerfile), jobs))
|
|
return found
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# NEGATIVE — the gitless stages run exactly what they are pinned to run
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_every_SPA_CARRYING_STAGE_runs_exactly_its_pinned_commands() -> None:
|
|
"""Set equality on command TEXT, not a predicate over it.
|
|
|
|
A suite run put back in ANY spelling — a heredoc, `sh -c`, `npm t`, `timeout npm test`, a
|
|
`#`-suffixed line — is simply not equal to the pin. That is the whole point: three earlier
|
|
versions asked what a command meant and were wrong nine times; this one asks whether it is the
|
|
command that was reviewed.
|
|
"""
|
|
faults: list[str] = []
|
|
seen: set[str] = set()
|
|
for rel, stage, commands in spa_carrying_stages():
|
|
key = f"{rel}::{stage}"
|
|
seen.add(key)
|
|
pinned = PINNED_STAGE_COMMANDS.get(key)
|
|
if pinned is None:
|
|
faults.append(
|
|
f"{key} carries the SPA source but has no entry in `PINNED_STAGE_COMMANDS`, so "
|
|
f"nothing checks what it runs. It runs: {commands}. Add the entry — and if any of "
|
|
f"those commands runs the vitest suite, that is the ersatztv#887 defect: the stage "
|
|
f"has no git checkout and no git binary, so the suite's git-dependent members "
|
|
f"cannot run there, and the red is unreachable on a PR."
|
|
)
|
|
continue
|
|
if tuple(commands) != pinned:
|
|
faults.append(
|
|
f"{key} does not run its pinned commands.\n pinned: {list(pinned)}\n actual: {commands}\n"
|
|
f" If this change is deliberate, update `PINNED_STAGE_COMMANDS` in the same commit "
|
|
f"and say why. If it added a vitest run, do not: this stage cannot run the suite "
|
|
f"(ersatztv#887), and excluding the specs that fail there is the hand-maintained "
|
|
f"population that broke every image build in ersatztv#883."
|
|
)
|
|
for orphan in sorted(set(PINNED_STAGE_COMMANDS) - seen):
|
|
faults.append(
|
|
f"`PINNED_STAGE_COMMANDS` names {orphan}, which no longer exists or no longer carries "
|
|
f"the SPA source. A pin for a stage that is not checked is a sentence nobody is reading."
|
|
)
|
|
assert not faults, "\n\n".join(faults)
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# POSITIVE — the image is gated on the job holding the pinned step
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_every_IMAGE_PUBLISHING_job_is_gated_on_the_job_that_runs_the_suite() -> None:
|
|
"""Without this, deleting `needs:` is silently green."""
|
|
known = set(dockerfiles())
|
|
faults: list[str] = []
|
|
for rel, job_id, dockerfile, jobs in publishing_jobs():
|
|
if dockerfile not in known:
|
|
faults.append(
|
|
f"{rel}: job `{job_id}` builds `{dockerfile}`, which is not a tracked Dockerfile. "
|
|
f"The population here is derived from the git index, so this job's image is "
|
|
f"UNCHECKED — fix the path or widen `dockerfiles()`."
|
|
)
|
|
continue
|
|
if not any(stage_rel == dockerfile for stage_rel, _, _ in spa_carrying_stages()):
|
|
continue
|
|
gates = _transitive_needs(job_id, jobs)
|
|
# `needs:` names a job in the SAME workflow, so the edge only reaches the pinned step when
|
|
# the publishing job lives in `GATING_WORKFLOW`. Matching a bare job id anywhere was a false
|
|
# green: a second workflow publishing this Dockerfile while `needs:`-ing its OWN unrelated
|
|
# job called `test` (`run: echo ok`) satisfied it — measured, and it escaped every
|
|
# workflow-population guard in the repo, 588 tests green.
|
|
if rel != GATING_WORKFLOW or GATING_JOB not in gates:
|
|
faults.append(
|
|
f"{rel}: job `{job_id}` publishes an image built from `{dockerfile}`, which carries "
|
|
f"the SPA source, but does not transitively `needs:` `{GATING_JOB}` IN "
|
|
f"`{GATING_WORKFLOW}` — the job holding the pinned suite step. It needs: "
|
|
f"{sorted(gates)}. A `needs:` edge resolves within its own workflow, so a job named "
|
|
f"`{GATING_JOB}` in a DIFFERENT workflow is a different job and gates nothing. Since "
|
|
f"ersatztv#887 the image build does not run the suite itself, so this edge is the "
|
|
f"ONLY thing between a red suite and a published image."
|
|
)
|
|
assert not faults, "\n".join(faults)
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# EFFECTIVE — the gating step really runs, really fails, and really runs everything
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_the_GATING_STEP_is_exactly_what_was_reviewed() -> None:
|
|
"""Its body, its `if:`, and the absence of `continue-on-error` — all pinned.
|
|
|
|
Pinning the BODY is what makes "unfiltered" and "not suppressed" true without parsing: a
|
|
`--exclude`, a `|| true`, a pipe, a `set +e`, a `true || npm test` that never executes, are each
|
|
a different string from the pin. Three earlier versions tried to recognise those individually and
|
|
missed `true || npm test` and `continue-on-error: ${{ … }}` among others.
|
|
"""
|
|
named = [
|
|
candidate
|
|
for candidate in _steps(workflow_jobs(GATING_WORKFLOW).get(GATING_JOB, {}))
|
|
if str(candidate.get("name", "")).strip() == GATING_STEP_NAME
|
|
]
|
|
assert len(named) == 1, (
|
|
f"{GATING_WORKFLOW}: job `{GATING_JOB}` has {len(named)} steps named {GATING_STEP_NAME!r}, "
|
|
f"not one. That name is what both this assertion and the substring sweep key on, so a "
|
|
f"SECOND step wearing it inherits the exemption and is never checked — measured by cold "
|
|
f"review with a duplicate carrying `continue-on-error: true` and a spec filter."
|
|
)
|
|
step = named[0]
|
|
|
|
faults: list[str] = []
|
|
body = step.get("run")
|
|
# PER LINE, not over the whole body: `_normalise` collapses newlines, so changing `run: |` to
|
|
# `run: >` folds the two lines into one command whose normalised text is byte-identical to the
|
|
# pin. Measured by cold review — the marker script then swallows the suite as its arguments.
|
|
if not isinstance(body, str) or _normalise_lines(body) != _normalise_lines(GATING_STEP_RUN):
|
|
faults.append(
|
|
f"the gating step's `run:` is not the pinned command.\n pinned: {GATING_STEP_RUN!r}\n"
|
|
f" actual: {body!r}\n Anything but this — a spec filter, a `|| true`, a pipe, a "
|
|
f"`set +e`, a suite on the right of a `||` that may never execute — changes whether a "
|
|
f"red suite blocks the image. Update the pin deliberately, in the same commit, with a "
|
|
f"reason."
|
|
)
|
|
|
|
workdir = step.get("working-directory")
|
|
if workdir is None or str(workdir).strip() != GATING_STEP_WORKDIR:
|
|
faults.append(
|
|
f"the gating step's `working-directory:` is {workdir!r}, not the pinned "
|
|
f"{GATING_STEP_WORKDIR!r}. `npm test` run anywhere else runs a different package, or "
|
|
f"none, while the pinned command text is unchanged."
|
|
)
|
|
|
|
shell = ((yaml.safe_load(_read(GATING_WORKFLOW)) or {}).get("defaults") or {}).get("run", {}).get("shell")
|
|
if shell is None or str(shell).strip() != GATING_WORKFLOW_SHELL:
|
|
faults.append(
|
|
f"{GATING_WORKFLOW}'s `defaults.run.shell` is {shell!r}, not the pinned "
|
|
f"{GATING_WORKFLOW_SHELL!r}. `bash` here means `bash -e`, which is what makes a failing "
|
|
f"command fail the step; changing it changes whether a red suite blocks the image, "
|
|
f"without touching the step."
|
|
)
|
|
|
|
if stage_shell_overrides():
|
|
faults.append(
|
|
f"a pinned Dockerfile stage carries a `SHELL` instruction: {stage_shell_overrides()}. "
|
|
f"That redefines what every later `RUN` executes while leaving its text — and therefore "
|
|
f"the pin — unchanged."
|
|
)
|
|
|
|
condition = step.get("if")
|
|
if condition is None or _normalise(str(condition)) != _normalise(GATING_STEP_IF):
|
|
faults.append(
|
|
f"the gating step's `if:` is not the pinned condition.\n pinned: {GATING_STEP_IF!r}\n"
|
|
f" actual: {condition!r}\n A third term here lets the suite not run at all while the "
|
|
f"image still publishes."
|
|
)
|
|
|
|
# ANY spelling, not two literals. `continue-on-error: ${{ … }}` is the idiomatic conditional form
|
|
# and a check comparing against `True`/`"true"` waves it through — a presence test cannot see
|
|
# polarity, and this one is fail-OPEN in exactly the direction that matters.
|
|
job = workflow_jobs(GATING_WORKFLOW).get(GATING_JOB, {})
|
|
for scope, holder in (("step", step), ("job", job)):
|
|
if "continue-on-error" in holder and holder["continue-on-error"] not in (False, "false"):
|
|
faults.append(
|
|
f"the gating {scope} carries `continue-on-error: {holder['continue-on-error']!r}`, "
|
|
f"so a red suite reports success and the `needs:` edge that gates the image blocks "
|
|
f"nothing. Only absent, `false` or `False` is acceptable here."
|
|
)
|
|
|
|
if "shell" in step:
|
|
faults.append(
|
|
f"the gating step carries its own `shell: {step['shell']!r}`, which overrides the pinned "
|
|
f"workflow default. The default is pinned because `bash` means `bash -e`; a step-level "
|
|
f"override moves that decision somewhere nothing checks."
|
|
)
|
|
if "defaults" in job:
|
|
faults.append(
|
|
f"the job `{GATING_JOB}` carries `defaults: {job['defaults']!r}`, which overrides the "
|
|
f"pinned workflow default for every step in it, including the gating one."
|
|
)
|
|
|
|
if "if" in job:
|
|
faults.append(
|
|
f"the job `{GATING_JOB}` carries a job-level `if:` ({job['if']!r}). None did when "
|
|
f"ersatztv#887 made this `needs:` edge the only gate, and a job that does not run cannot "
|
|
f"gate anything — Gitea reports it `skipped`, and what that does to a dependent job is "
|
|
f"not something this repo relies on."
|
|
)
|
|
assert not faults, "\n".join(faults)
|
|
|
|
|
|
def test_the_npm_SCRIPT_MAP_is_exactly_what_was_reviewed() -> None:
|
|
"""The whole map, because a pinned `RUN npm …` executes whatever it says.
|
|
|
|
An earlier version pinned only the scripts whose body contained `vitest`, which is a SELECTOR —
|
|
the category this file elsewhere calls the worst-behaved, because going short is silent. It went
|
|
short four ways, each re-arming ersatztv#887 in the gitless stage with every other pin matching:
|
|
`npm run test`, `npm t`, and the `prebuild`/`preinstall` LIFECYCLE HOOKS, which npm runs for
|
|
`npm run build` and `npm ci` without anything naming them. Pinning the map removes the category:
|
|
a script that does not exist cannot be a hook, and one that changes is not equal.
|
|
"""
|
|
scripts = json.loads(_read(PACKAGE_JSON)).get("scripts", {})
|
|
assert scripts == PINNED_PACKAGE_SCRIPTS, (
|
|
f"`{PACKAGE_JSON}`'s scripts are not the pinned map.\n pinned: {PINNED_PACKAGE_SCRIPTS}\n"
|
|
f" actual: {scripts}\n Both pinned `RUN npm …` lines in the gitless `web-build` stage "
|
|
f"execute whatever this says, INCLUDING npm's `pre*`/`post*` lifecycle hooks, which nothing "
|
|
f"in the Dockerfile names. A suite run added here reaches that stage with no Dockerfile or "
|
|
f"workflow line changed — ersatztv#887 through a file nobody would think to check against "
|
|
f"this guard. Update the pin deliberately, in the same commit, with a reason."
|
|
)
|
|
|
|
|
|
def test_the_VITE_PLUGIN_LIST_is_exactly_what_was_reviewed() -> None:
|
|
"""A plugin can run the suite from a build hook, so the list of them is pinned.
|
|
|
|
Vite plugins execute in the same process as `vite build` and `vitest`, and a `buildStart()` that
|
|
shells out to `npm test` puts the suite back into the gitless stage with every command pin
|
|
intact — measured, both as an inline plugin and inside an existing one. The list is pinned so a
|
|
new plugin cannot arrive unnoticed; what each plugin DOES is a residual, and the mitigation
|
|
there is that `trackedSourceFilesPlugin` is deliberately lazy.
|
|
"""
|
|
text = _read(VITE_CONFIG)
|
|
assert text.count(PINNED_VITE_PLUGINS) == 1, (
|
|
f"`{VITE_CONFIG}`'s plugin list is not the pinned one ({PINNED_VITE_PLUGINS!r}). A vite "
|
|
f"plugin runs in the same process as the suite and can shell out from a build hook, so a "
|
|
f"new one is a decision to state. Update the pin deliberately."
|
|
)
|
|
|
|
|
|
def test_NO_OTHER_vitest_CONFIG_outranks_the_pinned_one() -> None:
|
|
"""A pinned config is worthless if a second file can take precedence over it.
|
|
|
|
Vitest prefers `vitest.config.*` (and `vitest.workspace.*`/`vitest.projects.*`) to
|
|
`vite.config.*`. MEASURED: dropping a `web/vitest.config.ts` with `include: ['nope/**']` and
|
|
`passWithNoTests: true` beside the pinned file made the suite collect zero specs and exit 0 — the
|
|
gating step green having run nothing at all, which is worse than the filtered run ersatztv#887
|
|
removed. The construct is refused rather than modelled: there is no such file today, so the
|
|
honest move is to assert none appears.
|
|
"""
|
|
intruders = sorted(
|
|
path
|
|
for path in tracked_files._git_ls_files()
|
|
if path.startswith("web/") and path.rpartition("/")[2].startswith(VITEST_CONFIG_PREFIXES)
|
|
)
|
|
assert not intruders, (
|
|
f"these files outrank the pinned `{VITE_CONFIG}` in vitest's config resolution: {intruders}. "
|
|
f"Whatever they say about `include`/`exclude` decides which specs the gating run collects, "
|
|
f"and the pin on `{VITE_CONFIG}` cannot see it — a suite that collects nothing still exits 0. "
|
|
f"Either fold the settings into `{VITE_CONFIG}` and its pin, or pin this file too."
|
|
)
|
|
|
|
|
|
def test_the_VITE_TEST_BLOCK_is_exactly_what_was_reviewed() -> None:
|
|
"""`npm test -- --run` collects what this config says; "unfiltered" is otherwise unverified.
|
|
|
|
`test.exclude` is where the Dockerfile's `--exclude`s would naturally be re-created now that they
|
|
are gone, and a filter there leaves every other pin matching while the gating run reports green
|
|
over a narrowed suite. Cold review measured three: adding the three git-dependent specs to
|
|
`exclude`, narrowing `include` to `src/**`, and `include: ['nope/**']` with `passWithNoTests`.
|
|
"""
|
|
block = _vite_test_block()
|
|
assert block == _normalise_lines(PINNED_VITE_TEST_BLOCK), (
|
|
f"`{VITE_CONFIG}`'s `test:` block is not the pinned one.\n"
|
|
f" pinned: {_normalise_lines(PINNED_VITE_TEST_BLOCK)}\n actual: {block}\n"
|
|
f" This block decides WHICH specs `npm test -- --run` collects, so a filter here narrows "
|
|
f"the gating suite with the step's own command unchanged — the same defect ersatztv#887 "
|
|
f"removed from the Dockerfile, relocated. Update the pin deliberately."
|
|
)
|
|
|
|
|
|
def test_the_DOCS_ONLY_arm_cannot_publish_an_image() -> None:
|
|
"""The premise under half of `GATING_STEP_IF`, asserted rather than assumed.
|
|
|
|
The gating step is allowed to skip when `docs_only` is true. That is only safe because the
|
|
publish step skips on the same condition, so the arm ships nothing. Round-1 review found this
|
|
stated as fact in the prose with nothing checking it — the same shape as the defect one level in.
|
|
"""
|
|
step = find_step(GATING_WORKFLOW, "build", PUBLISH_STEP_NAME)
|
|
assert step is not None, f"{GATING_WORKFLOW}: job `build` has no step named {PUBLISH_STEP_NAME!r}."
|
|
condition = step.get("if")
|
|
assert condition is not None and _normalise(str(condition)) == _normalise(PUBLISH_STEP_IF), (
|
|
f"the publish step's `if:` is {condition!r}, not the pinned {PUBLISH_STEP_IF!r}. The gating "
|
|
f"step is permitted to skip on `docs_only`, and that is safe ONLY because this step skips "
|
|
f"on it too. Without this, a docs-only push to `main` skips the SPA suite and publishes an "
|
|
f"image anyway."
|
|
)
|
|
|
|
|
|
def test_only_the_PINNED_step_mentions_the_suite() -> None:
|
|
"""A substring sweep, deliberately not a semantic predicate.
|
|
|
|
Every earlier version tried to decide whether a command RUNS the suite and was wrong nine times.
|
|
This asks the much weaker question — does any OTHER step's shell body mention it at all — whose
|
|
failure mode is a false red asking a human to look, never a false green.
|
|
"""
|
|
strays: list[str] = []
|
|
for path in workflow_files():
|
|
rel = path.relative_to(REPO_ROOT).as_posix()
|
|
for job_id, job in workflow_jobs(rel).items():
|
|
for step in _steps(job):
|
|
body = step.get("run")
|
|
if not isinstance(body, str):
|
|
continue
|
|
if (
|
|
rel == GATING_WORKFLOW
|
|
and job_id == GATING_JOB
|
|
and str(step.get("name", "")).strip() == GATING_STEP_NAME
|
|
):
|
|
continue
|
|
hit = [mention for mention in SUITE_MENTIONS if mention in body]
|
|
if hit:
|
|
strays.append(f"{rel} :: job `{job_id}` step {str(step.get('name', '?'))!r} mentions {hit}")
|
|
assert not strays, (
|
|
"these steps mention the SPA suite and are not the pinned gating step:\n "
|
|
+ "\n ".join(strays)
|
|
+ "\n\nThis is a SUBSTRING sweep, not a claim that they run it — look, and either move the "
|
|
"run into the pinned step or record why this one is not a second, unpinned gate."
|
|
)
|
|
|
|
|
|
def test_no_DOCKERFILE_outside_a_pinned_stage_mentions_the_suite() -> None:
|
|
"""The same weak sweep over Dockerfiles, covering stages that carry no SPA source.
|
|
|
|
A stage without `web/` has no suite to run, so `PINNED_STAGE_COMMANDS` does not cover it. This
|
|
reports a mention there anyway rather than leaving the class unexamined.
|
|
"""
|
|
strays: list[str] = []
|
|
for rel in dockerfiles():
|
|
for name, lines in stages(_read(rel)):
|
|
if f"{rel}::{name}" in PINNED_STAGE_COMMANDS:
|
|
continue
|
|
for line in lines:
|
|
hit = [mention for mention in SUITE_MENTIONS if mention in line]
|
|
if hit:
|
|
strays.append(f"{rel} :: stage `{name}`: {line!r} mentions {hit}")
|
|
assert not strays, "these unpinned Dockerfile stages mention the SPA suite:\n " + "\n ".join(strays)
|
|
|
|
|
|
# ------------------------------------------------------------------------------------------------
|
|
# Anti-vacuity and negative controls
|
|
# ------------------------------------------------------------------------------------------------
|
|
|
|
|
|
def test_no_run_BODY_builds_or_pushes_an_image() -> None:
|
|
"""`publishing_jobs()` keys on the ACTION, which is a selector; assert nothing publishes around it.
|
|
|
|
A job doing `docker build -f ./docker/Dockerfile … && docker push …` in a shell body is not in
|
|
that population, so the positive invariant is silently not applied to it — and anti-vacuity does
|
|
not notice, because the real `build` job is still there. Measured MISSED by cold review after
|
|
this assertion was dropped in the rewrite; it is restored rather than reasoned away.
|
|
"""
|
|
stray: list[str] = []
|
|
for path in workflow_files():
|
|
rel = path.relative_to(REPO_ROOT).as_posix()
|
|
for job_id, job in workflow_jobs(rel).items():
|
|
for step in _steps(job):
|
|
body = step.get("run")
|
|
if not isinstance(body, str):
|
|
continue
|
|
for line in body.splitlines():
|
|
if line.lstrip().startswith("#"):
|
|
continue
|
|
if re.search(r"\bdocker\s+(?:push|build|buildx\s+build)\b", line):
|
|
stray.append(f"{rel} :: job `{job_id}`: {line.strip()!r}")
|
|
assert not stray, (
|
|
f"image build/push outside `{PUBLISH_ACTION}`:\n " + "\n ".join(stray) + "\n\n"
|
|
"`publishing_jobs()` derives from that action, so these jobs are NOT checked for the suite "
|
|
"gate — extend it before adding one."
|
|
)
|
|
|
|
|
|
def test_the_populations_are_NOT_empty() -> None:
|
|
"""Every assertion above passes over an empty set, which is how a completeness guard dies."""
|
|
assert dockerfiles(), "`git ls-files` reported no Dockerfile — the derivation is broken, not the repo."
|
|
assert workflow_files(), f"`git ls-files` reported no {WORKFLOW_DIR}/*.yml — the derivation is broken."
|
|
|
|
carrying = spa_carrying_stages()
|
|
assert carrying, (
|
|
f"no stage in {dockerfiles()} was read as carrying the SPA source, so the negative assertion "
|
|
"checked nothing and the positive one skipped every job. `copies_spa_source` or `stages` "
|
|
"stopped matching."
|
|
)
|
|
assert PINNED_STAGE_COMMANDS, "the pin table is empty; the negative assertion asserts nothing."
|
|
|
|
published = publishing_jobs()
|
|
assert published, (
|
|
f"no job in {WORKFLOW_DIR} uses `{PUBLISH_ACTION}`, so the gate assertion ran over an empty "
|
|
"population. Either the repo stopped publishing images or `_canonical_action` stopped "
|
|
"recognising the action."
|
|
)
|
|
assert any(dockerfile in {rel for rel, _, _ in carrying} for _, _, dockerfile, _ in published), (
|
|
"no publishing job builds an SPA-carrying Dockerfile, so the gate assertion is vacuous."
|
|
)
|
|
|
|
|
|
def test_the_STAGE_SPLIT_finds_the_real_stages() -> None:
|
|
"""`stages()` is the only Dockerfile syntax this file interprets; pin what it returns.
|
|
|
|
A split that stopped recognising `FROM … AS name` would put every instruction in one bucket, the
|
|
pin key would never match, and `test_every_SPA_CARRYING_STAGE_runs_exactly_its_pinned_commands`
|
|
would report an orphan pin — loud, but for the wrong reason. Pinned so the cause is legible.
|
|
"""
|
|
names = [name for name, _ in stages(_read("docker/Dockerfile"))]
|
|
assert names[:3] == ["dotnet-runtime", "web-build", "runtime-base"], names
|
|
assert [stage for _, stage, _ in spa_carrying_stages()] == ["web-build"], spa_carrying_stages()
|
|
|
|
|
|
def test_the_SPA_CARRYING_predicate_reads_every_COPY_SPELLING() -> None:
|
|
"""Pinned both ways: every stage answered False is SKIPPED by the negative assertion.
|
|
|
|
The synthetic spellings were MEASURED False by cold review before this was widened; a new stage
|
|
using any of them would have gone unpinned and unchecked.
|
|
"""
|
|
for spelling in (
|
|
'COPY ["web/", "/source/web/"]',
|
|
'COPY --chown=node:node ["web/", "/source/web/"]',
|
|
"COPY web /source/web",
|
|
"ADD web/ /source/web/",
|
|
"COPY web/. ./web/",
|
|
"COPY . /source/",
|
|
"COPY ./ /source/",
|
|
"COPY --chown=node:node web/ /source/web/",
|
|
"COPY web/package*.json ./web/",
|
|
# A GLOB that matches `web/`. Measured False before `_source_reaches_web`, which left a
|
|
# stage copying the source this way unpinned and unchecked.
|
|
"COPY web*/ ./",
|
|
"COPY * /source/",
|
|
):
|
|
assert copies_spa_source(spelling), spelling
|
|
|
|
# A STAGE copy can still carry the SPA source across. Measured False before this clause: the
|
|
# receiving stage was then unpinned, so a suite run in it was invisible.
|
|
assert copies_spa_source("COPY --from=web-build /source/web /source/web")
|
|
assert copies_spa_source("COPY --from=builder /app/web ./web")
|
|
|
|
for spelling in (
|
|
"COPY design-system/. ./design-system/",
|
|
"COPY --from=web-build /source/ErsatzTV/wwwroot/app/. ./app/",
|
|
"COPY webhooks/. ./webhooks/", # `web` must be a whole path segment
|
|
"RUN npm ci",
|
|
"COPY web/", # a single argument is not a copy
|
|
):
|
|
assert not copies_spa_source(spelling), spelling
|
|
|
|
|
|
def test_the_INSTRUCTION_JOIN_survives_what_defeated_the_PARSER() -> None:
|
|
"""The shapes that broke three earlier versions must not break the pin comparison.
|
|
|
|
None of these needs interpreting — they only need to arrive at the comparison intact, so that a
|
|
stage running them is NOT EQUAL to its pin. Each was a measured false green when this file tried
|
|
to decide what the command meant.
|
|
"""
|
|
heredoc = "RUN <<EOF\nnpm test -- --run\nEOF\n"
|
|
assert "RUN <<EOF" in instructions(heredoc)[0], instructions(heredoc)
|
|
assert instructions("RUN echo hi # npm test -- --run\n") == ["RUN echo hi # npm test -- --run"]
|
|
assert instructions("RUN a=(1 2); npm test\n") == ["RUN a=(1 2); npm test"]
|
|
assert instructions("RUN npm run lint && \\\n npm test\n") == ["RUN npm run lint && npm test"]
|
|
# A comment LINE is still dropped, which is what lets the Dockerfile explain the rule.
|
|
assert instructions("# RUN npm test -- --run\nRUN npm ci\n") == ["RUN npm ci"]
|
|
|
|
|
|
def test_the_PIN_rejects_every_spelling_the_PARSER_missed() -> None:
|
|
"""The nine shapes that defeated the predicate, run against the pin instead.
|
|
|
|
Not one of them is equal to `RUN npm run lint && npm run typecheck && npm run build`, which is
|
|
the entire argument for replacing the mechanism: the pin does not need to recognise a spelling to
|
|
reject it.
|
|
"""
|
|
pinned = PINNED_STAGE_COMMANDS["docker/Dockerfile::web-build"]
|
|
for spelling in (
|
|
"RUN <<EOF",
|
|
"RUN echo build#1 && npm test -- --run",
|
|
"RUN (npm run lint); npm test -- --run",
|
|
"RUN npm t -- --run",
|
|
"RUN ./node_modules/.bin/vitest run",
|
|
"RUN pnpm vitest run",
|
|
"RUN timeout 900 npm test -- --run",
|
|
"RUN su node -c 'npm test -- --run'",
|
|
"RUN sh -c 'npm test -- --run'",
|
|
"RUN npm run lint && npm run typecheck && npm test -- --run && npm run build",
|
|
):
|
|
assert spelling not in pinned, spelling
|
|
assert (pinned[0], spelling) != pinned, spelling
|