"""The SPA vitest suite never runs inside an image build, and the image is gated on a step that does run it, unfiltered and non-advisory (ersatztv#887). WHAT THIS IS PROTECTING. `docker/Dockerfile`'s `web-build` stage is gitless twice over. It copies only `web/` and `design-system/`, so it holds no `.git` — a property of the STAGE, not of the build context, which is the repository root (`context: .`) and whose `.dockerignore` does not exclude `.git`: the directory is there to be copied and simply is not. And `node:22-bookworm-slim` ships no git binary, which is why a `COPY .git` would not help either. Members of the SPA suite need one or the other — `pageSizeCallSites.guard.test.ts` and `completeAnnotations.guard.test.ts` derive their file population from `git ls-files` and refuse to fall back to a directory walk (`testing.guard-derives-population-from-source`), and `trackedSourceFiles.realgit.test.ts` builds a real temporary repository. Running the suite there anyway therefore requires a list of the members that cannot run, and that list is a population nothing derives: ersatztv#883 added a third member without updating a hand-written pair of `--exclude`s, and every image build failed from that commit on. The red was structurally unreachable on a PR — `Build & push image (amd64)` carries `if: github.event_name != 'pull_request'` — so it landed on `main` and on the `v*` tag path, where a release cut fails at the image build and `:latest` stops being republished. ## WHY THIS GUARD PINS TEXT INSTEAD OF PARSING IT A predicate asking "does this command RUN the suite, and can it FAIL?" of arbitrary shell text was wrong nine times, and every one of the nine was the same mechanism: * heredoc bodies — skipped as data, but BuildKit EXECUTES `RUN <` folds them into one and the whole-text comparison could not see it. * Gitea's own evaluation of a skipped `needs:` job is not modelled; the guard forbids the job-level `if:` that would create one rather than reasoning about it. """ from __future__ import annotations import json import re from fnmatch import fnmatch from pathlib import Path import yaml from scripts.tests import tracked_files from scripts.tests.tracked_files import tracked_paths REPO_ROOT = Path(__file__).resolve().parents[2] WORKFLOW_DIR = ".gitea/workflows" WORKFLOWS = (WORKFLOW_DIR, ("*.yml", "*.yaml")) # The action every image publish in this repo goes through. Matched on the last two path segments # for the reason `test_workflow_persist_credentials.py` gives: Gitea accepts a full action URL, and # a host is not required to contain a dot, so no hostname heuristic is safe. PUBLISH_ACTION = "docker/build-push-action" # Instructions that bring files from the build context into the image. COPY_INSTRUCTIONS = frozenset({"ADD", "COPY"}) # --------------------------------------------------------------------------------------------- # THE PINS. Update these deliberately, in the same commit as the change they describe, and say why. # --------------------------------------------------------------------------------------------- # Every `RUN` in a Dockerfile stage that carries the SPA source, keyed `::`. Whitespace # is normalised (see `_normalise`) so a reflow is not a red, but the tokens are exact. PINNED_STAGE_COMMANDS: dict[str, tuple[str, ...]] = { "docker/Dockerfile::web-build": ( "RUN npm ci", # Lints, typechecks and BUILDS the SPA. It must not TEST it: this stage has neither a git # checkout nor the git binary, and naming the specs that cannot run there is the # hand-maintained population ersatztv#887 removed. "RUN npm run lint && npm run typecheck && npm run build", ), } # The step that runs the suite and gates the image, pinned whole. Its body being exact is what makes # "unfiltered", "actually executes", and "cannot be suppressed" true without parsing any of it. GATING_WORKFLOW = ".gitea/workflows/docker-build.yml" GATING_JOB = "test" GATING_STEP_NAME = "Test SPA" GATING_STEP_RUN = '"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark web-test\nnpm test -- --run' # `docs_only` (ersatztv#416) and the #420 revalidate skip. Both are safe for a DIFFERENT reason and # the difference is why this is pinned rather than pattern-matched: docs-only cannot ship an image at # all, which is asserted below rather than assumed; the revalidate skip fires only on a tree # byte-identical to a head that already carried a green combined status. GATING_STEP_IF = "steps.detect.outputs.docs_only != 'true' && steps.revalidate.outputs.skip != 'true'" # `npm test` means nothing without the directory it runs in: pointed elsewhere it runs a different # package, or none. Part of the pin because it is part of what the step does. GATING_STEP_WORKDIR = "web" # `shell: bash` is `bash -e`, which is what makes a failing command fail the step. A workflow-level # change here alters that for the gating step without touching the step at all. GATING_WORKFLOW_SHELL = "bash" # The premise under the `docs_only` half of `GATING_STEP_IF`: the publish step is gated on it too, so # the arm that skips the suite ships nothing. It is CHECKED rather than stated as fact — an # unchecked premise is one level out from the same defect. PUBLISH_STEP_NAME = "Build and push" PUBLISH_STEP_IF = "steps.detect.outputs.docs_only != 'true'" # `RUN npm run build` and `RUN npm ci` execute whatever `web/package.json` says they do, and that # file is neither a Dockerfile nor a workflow. ersatztv#887 was RE-ARMED through it twice # without touching any pinned line: `"build": "vitest run && tsc -b && vite build"` puts the suite # back into the gitless stage, and `"prepare"` is run by `npm ci`. So the SCRIPTS THAT MENTION VITEST # are pinned too — exactly one, and its body. PACKAGE_JSON = "web/package.json" # THE WHOLE MAP, not the scripts that mention vitest. Selecting on the literal `vitest` was a # SELECTOR where a PIN was available, and ersatztv#887 was RE-ARMED through four spellings # none of which contain it: `"build": "npm run test -- --run && …"`, `"build": "npm t -- …"`, and the # lifecycle hooks `prebuild` (npm runs it before `npm run build`) and `preinstall` (before `npm ci`). # Both pinned RUN lines in the gitless stage execute whatever this map says, so the map is pinned. PINNED_PACKAGE_SCRIPTS: dict[str, str] = { "dev": "vite --host 0.0.0.0", "build": "tsc -b && vite build", "generate:api": "node scripts/generate-openapi-types.mjs", "check:api": "npm run generate:api && git diff --exit-code -- src/api/generated/v1.d.ts", "lint": "eslint .", "test": "vitest", "test:ui-e2e": "playwright test", "typecheck": "tsc -b --pretty false", "prepare": "cd .. && husky", } # `npm test -- --run` collects whatever this config says, and `vite build` in the gitless stage # loads it too — so the WHOLE FILE is pinned, not a block within it. Three rounds each closed one # SPELLING of a marker match (`test: {`, then `test: {`, then `test : {` and `"test": {`), and two # more routes never touched the marker at all (`].concat([evil])` after the pinned array, # `...moreTest` after the pinned block — `defineConfig` is the identity function, so a later spread # REPLACES what the pin matched). That is `every-blocker-was-one-mechanism-so-delete-it` at the # count where it says withdraw: the mechanism was partial matching, and it is gone. # # STATED COST: any edit that changes a LINE'S TOKEN SEQUENCE reddens, a comment's words included. # Whitespace and blank lines are free — `_normalise_lines` collapses them, which is what lets a # reflow through. That tolerance is inert for this file's current content (no template literal and # no ASI-sensitive token: every `return`/`=>`/backtick in it sits inside a `//` comment) and would # stop being inert if one were introduced. It is 48 lines, nothing generates it, and it is the same # move already made for `web/package.json`'s script map — a PIN where a partial match was defended. VITE_CONFIG = "web/vite.config.ts" PINNED_VITE_CONFIG = """import { fileURLToPath, URL } from 'node:url'; import react from '@vitejs/plugin-react'; import { configDefaults, defineConfig } from 'vitest/config'; import { trackedSourceFilesPlugin } from './vite-plugins/trackedSourceFiles'; const aspNetHost = 'http://localhost:8409'; export default defineConfig({ base: '/app/', plugins: [react(), trackedSourceFilesPlugin()], resolve: { alias: { '@': fileURLToPath(new URL('./src', import.meta.url)), '@design-system': fileURLToPath(new URL('../design-system', import.meta.url)) } }, build: { outDir: '../ErsatzTV/wwwroot/app', emptyOutDir: true }, server: { port: 5173, strictPort: false, proxy: { '/api': aspNetHost, '/iptv': aspNetHost, '/artwork': aspNetHost } }, test: { environment: 'jsdom', environmentOptions: { jsdom: { url: 'http://localhost/app/' } }, // `e2e/` holds the Playwright UI-E2E specs (ersatztv#445), which drive a REAL browser against a // running instance. Vitest's default `include` glob (`**/*.{test,spec}.*`) would otherwise pick // them up and run them under jsdom, where `@playwright/test`'s `test()` has no runner and the // whole file dies. Spread the defaults rather than replacing `exclude` outright — a bare // `exclude: ['e2e/**']` would drop `**/node_modules/**` and make vitest crawl deps. // NOTE: excluding (not narrowing `include` to `src/**`) is deliberate — `scripts/` holds a real // vitest test too (`generate-openapi-types.test.mjs`), which an `src/**`-only include would // silently stop running. exclude: [...configDefaults.exclude, 'e2e/**'], setupFiles: './src/setupTests.ts' } }); """ # Every config filename that can outrank the pinned one, for EITHER tool. Read out of the pinned # tarballs rather than remembered: vite 8.1.3's `DEFAULT_CONFIG_FILES` is # `["vite.config.js", "vite.config.mjs", "vite.config.ts", …]` — so a `vite.config.js` beats the # pinned `.ts` for `vite build`, which is what the gitless stage runs. Vitest 4.1.9 resolves # `vitest.config.*` before `vite.config.*`. MEASURED: a `web/vite.config.js` whose plugin shells out # to the suite ran it in the gitless stage with all 1411 tests green. OUTRANKING_CONFIG_PREFIXES = ("vitest.config.", "vitest.workspace.", "vitest.projects.", "vite.config.") # Substrings that mean "a human should look at this step". NOT a semantic predicate — the pinned step # is allowed to contain them and every other step is not. SUITE_MENTIONS = ("npm t", "npm run test", "yarn test", "pnpm test", "bun test", "vitest") # `npm t` rather than `npm test`: it is npm's own documented alias, it SUBSUMES `npm test` as a # substring, and this file enumerates it among the spellings that defeated the parser. Leaving it out # meant an unpinned stage running `npm t -- --run` passed the sweep — MEASURED. def _normalise(text: str) -> str: """Collapse runs of whitespace so a reflow is not a red, keeping every token exact.""" return " ".join(text.split()) def _normalise_lines(text: str) -> list[str]: """Per-line normalisation, so indentation is free but the LINE BOUNDARIES are not. A newline separates two commands; collapsing it merges them into one. That is not a cosmetic difference and the whole-text form cannot see it. """ return [_normalise(line) for line in text.strip().splitlines() if line.strip()] def _canonical_action(uses: str) -> str: """The last two path segments of a `uses:` value, without its `@version`. Same normalisation as `test_workflow_persist_credentials._canonical_action`, and deliberately a second copy rather than an import: that guard's population is checkout steps, this one's is publish steps, and sharing the helper would mean a narrowing there silently narrows here too. """ ref = uses.strip().strip("'\"").rsplit("@", 1)[0].strip().lower() ref = re.sub(r"^https?://", "", ref) segments = [seg for seg in ref.split("/") if seg] if segments: segments[-1] = re.sub(r"\.git$", "", segments[-1]) return "/".join(segments[-2:]) if len(segments) >= 2 else ref def dockerfiles() -> list[str]: """Tracked Dockerfiles, by repo-relative path, in every spelling docker recognises. `tracked_files._git_ls_files()` through the MODULE, never a `from … import` binding. The shared proof in `test_guard_populations_derive_from_git.py` narrows the index by patching that module attribute, and a name bound at import time is not reachable from there — the derivation would go on returning every Dockerfile after git stopped tracking them, and the proof would say so. """ return sorted( path for path in tracked_files._git_ls_files() if path.rpartition("/")[2].startswith("Dockerfile") or path.endswith(".Dockerfile") ) def workflow_files() -> list[Path]: return tracked_paths(*WORKFLOWS) def _read(rel: str) -> str: return (REPO_ROOT / rel).read_text(encoding="utf-8") def instructions(text: str) -> list[str]: """The Dockerfile instructions, comments dropped and `\\` continuations joined. This is the whole of the file's syntax handling. It does not interpret what a `RUN` DOES — the string is compared against a pin — so the shell traps that defeated the parser (heredocs, `#` inside a command, compound punctuation) cannot reach any decision here. """ joined: list[str] = [] buffer = "" for raw in text.splitlines(): if raw.lstrip().startswith("#"): continue stripped = raw.rstrip() if stripped.endswith("\\"): buffer += stripped[:-1] + " " continue buffer += stripped if buffer.strip(): joined.append(_normalise(buffer)) buffer = "" if buffer.strip(): joined.append(_normalise(buffer)) return joined def stages(text: str) -> list[tuple[str, list[str]]]: """`(stage name, its instructions)` per build stage, in file order. A stage is delimited by `FROM`, which is the one Dockerfile construct with no shell content and therefore nothing to misparse. An unnamed stage is keyed by its index so it is still checkable. """ found: list[tuple[str, list[str]]] = [] for line in instructions(text): if re.match(r"^FROM\b", line, re.I): match = re.search(r"\bAS\s+(\S+)\s*$", line, re.I) found.append((match.group(1) if match else f"", [])) elif found: found[-1][1].append(line) return found def copies_spa_source(command: str) -> bool: """Does this ONE instruction bring the SPA source tree into the image? Every spelling counts, because the negative assertion SKIPS every stage answered False — a miss here drops a stage out of the check silently rather than reddening. `ADD` as well as `COPY`; the JSON exec form `COPY ["web/", "/dest/"]`, with or without a preceding flag; a source without a trailing slash (`COPY web /dest`); and `COPY . …`, which carries the tree without naming it. A `COPY --from=` is not a context copy: its source is another stage, so `--from=build .` would not bring the repo in. Flags are dropped before the source arguments are read. """ tokens = command.split() if not tokens or tokens[0].upper() not in COPY_INSTRUCTIONS: return False remainder = " ".join(token for token in tokens[1:] if not token.startswith("--")).strip() if remainder.startswith("["): # Parse the exec form BEFORE the `--from=` branch: splitting `["/source/web", "/dest"]` on # whitespace leaves a trailing comma on the first token, so the segment test never matched # and the stage went unpinned. Measured False. try: parsed = json.loads(remainder) except ValueError: return True remainder = " ".join(str(item) for item in parsed) if isinstance(parsed, list) else "" if any(token.startswith("--from=") for token in tokens[1:]): # A stage copy is not a CONTEXT copy, so `--from=build .` brings in that stage's `/`, not the # repository. But it can still carry the SPA SOURCE across from a stage that has it, and a # stage answered False here is UNPINNED and unchecked — measured: `COPY --from=web-build # /source/web /source/web` was False, so a suite run in the receiving stage was invisible. # The test is a whole `web` path SEGMENT in the source, which distinguishes the source tree # from the built artifact this repo actually copies (`/source/ErsatzTV/wwwroot/app/.` has no # such segment). return any("web" in source.strip("/").split("/") for source in remainder.split()[:-1]) # `remainder` is already a plain space-joined string by here — the exec form was parsed above, # before the `--from=` branch. A second `startswith("[")` test here is dead # code: measured by poisoning it with a `raise`, which left every test green. arguments = remainder.split() if len(arguments) < 2: return False return any(_source_reaches_web(source) for source in arguments[:-1]) def _source_reaches_web(source: str) -> bool: """Could this COPY source bring `web/` in? Exact (`web`, `web/…`), whole-context (`.`, `./`), and GLOB: `COPY web*/ ./` matches `web/` and so carries the source, while `webhooks/` does not — MEASURED False for the glob, which left the receiving stage unpinned. `fnmatch` against the literal segment `web` is what separates the two without hand-listing patterns. """ normalised = source.removeprefix("./").lstrip("/").rstrip("/") if normalised in {"", ".", "web"} or normalised.startswith("web/"): return True head = normalised.split("/")[0] return any(char in head for char in "*?[") and fnmatch("web", head) def spa_carrying_stages() -> list[tuple[str, str, list[str]]]: """`(dockerfile, stage name, RUN commands)` for every stage carrying the SPA source.""" found: list[tuple[str, str, list[str]]] = [] for rel in dockerfiles(): for name, lines in stages(_read(rel)): if any(copies_spa_source(line) for line in lines): found.append((rel, name, [line for line in lines if re.match(r"^RUN\b", line)])) return found def stage_shell_overrides() -> list[str]: """Every `SHELL` instruction inside a pinned stage, which the pin alone cannot see. `SHELL ["/bin/sh", "-c"]` redefines what every later `RUN` in the stage EXECUTES while leaving the `RUN` text — and therefore the pin — untouched. Reported rather than modelled: there is no `SHELL` in this repo, so the honest move is to refuse the construct here rather than reason about what a replacement interpreter would do. """ overrides: list[str] = [] for rel in dockerfiles(): for name, lines in stages(_read(rel)): if f"{rel}::{name}" not in PINNED_STAGE_COMMANDS: continue overrides.extend(f"{rel}::{name}: {line!r}" for line in lines if re.match(r"^SHELL\b", line)) return overrides def _jobs(doc: object) -> dict[str, dict]: jobs = doc.get("jobs") if isinstance(doc, dict) else None if not isinstance(jobs, dict): return {} return {str(key): value for key, value in jobs.items() if isinstance(value, dict)} def _steps(job: dict) -> list[dict]: steps = job.get("steps") return [step for step in steps if isinstance(step, dict)] if isinstance(steps, list) else [] def _needs(job: dict) -> list[str]: needs = job.get("needs") if isinstance(needs, str): return [needs] return [str(item) for item in needs] if isinstance(needs, list) else [] def _transitive_needs(job_id: str, jobs: dict[str, dict]) -> set[str]: seen: set[str] = set() frontier = list(_needs(jobs.get(job_id, {}))) while frontier: current = frontier.pop() if current in seen: continue seen.add(current) frontier.extend(_needs(jobs.get(current, {}))) return seen def workflow_jobs(rel: str) -> dict[str, dict]: return _jobs(yaml.safe_load(_read(rel))) def find_step(rel: str, job_id: str, name: str) -> dict | None: for step in _steps(workflow_jobs(rel).get(job_id, {})): if str(step.get("name", "")).strip() == name: return step return None def publishing_jobs() -> list[tuple[str, str, str, dict[str, dict]]]: """`(workflow rel path, job id, Dockerfile rel path, that workflow's jobs)` per publishing job. The Dockerfile comes from the step's own `file:` input, defaulting to docker's own default (`Dockerfile` at the context root) when the input is absent, so a step that stops naming one is still resolved rather than dropped from the population. """ found: list[tuple[str, str, str, dict[str, dict]]] = [] for path in workflow_files(): rel = path.relative_to(REPO_ROOT).as_posix() jobs = workflow_jobs(rel) for job_id, job in jobs.items(): for step in _steps(job): uses = step.get("uses") if not isinstance(uses, str) or _canonical_action(uses) != PUBLISH_ACTION: continue with_block = step.get("with") named = with_block.get("file") if isinstance(with_block, dict) else None dockerfile = str(named or "Dockerfile").strip() found.append((rel, job_id, re.sub(r"^\./", "", dockerfile), jobs)) return found # ------------------------------------------------------------------------------------------------ # NEGATIVE — the gitless stages run exactly what they are pinned to run # ------------------------------------------------------------------------------------------------ def test_every_SPA_CARRYING_STAGE_runs_exactly_its_pinned_commands() -> None: """Set equality on command TEXT, not a predicate over it. A suite run put back in ANY spelling — a heredoc, `sh -c`, `npm t`, `timeout npm test`, a `#`-suffixed line — is simply not equal to the pin. That is the whole point: three earlier versions asked what a command meant and were wrong nine times; this one asks whether it is the command that was reviewed. """ faults: list[str] = [] seen: set[str] = set() for rel, stage, commands in spa_carrying_stages(): key = f"{rel}::{stage}" seen.add(key) pinned = PINNED_STAGE_COMMANDS.get(key) if pinned is None: faults.append( f"{key} carries the SPA source but has no entry in `PINNED_STAGE_COMMANDS`, so " f"nothing checks what it runs. It runs: {commands}. Add the entry — and if any of " f"those commands runs the vitest suite, that is the ersatztv#887 defect: the stage " f"has no git checkout and no git binary, so the suite's git-dependent members " f"cannot run there, and the red is unreachable on a PR." ) continue if tuple(commands) != pinned: faults.append( f"{key} does not run its pinned commands.\n pinned: {list(pinned)}\n actual: {commands}\n" f" If this change is deliberate, update `PINNED_STAGE_COMMANDS` in the same commit " f"and say why. If it added a vitest run, do not: this stage cannot run the suite " f"(ersatztv#887), and excluding the specs that fail there is the hand-maintained " f"population that broke every image build in ersatztv#883." ) for orphan in sorted(set(PINNED_STAGE_COMMANDS) - seen): faults.append( f"`PINNED_STAGE_COMMANDS` names {orphan}, which no longer exists or no longer carries " f"the SPA source. A pin for a stage that is not checked is a sentence nobody is reading." ) assert not faults, "\n\n".join(faults) # ------------------------------------------------------------------------------------------------ # POSITIVE — the image is gated on the job holding the pinned step # ------------------------------------------------------------------------------------------------ def test_every_IMAGE_PUBLISHING_job_is_gated_on_the_job_that_runs_the_suite() -> None: """Without this, deleting `needs:` is silently green.""" known = set(dockerfiles()) faults: list[str] = [] for rel, job_id, dockerfile, jobs in publishing_jobs(): if dockerfile not in known: faults.append( f"{rel}: job `{job_id}` builds `{dockerfile}`, which is not a tracked Dockerfile. " f"The population here is derived from the git index, so this job's image is " f"UNCHECKED — fix the path or widen `dockerfiles()`." ) continue if not any(stage_rel == dockerfile for stage_rel, _, _ in spa_carrying_stages()): continue gates = _transitive_needs(job_id, jobs) # `needs:` names a job in the SAME workflow, so the edge only reaches the pinned step when # the publishing job lives in `GATING_WORKFLOW`. Matching a bare job id anywhere was a false # green: a second workflow publishing this Dockerfile while `needs:`-ing its OWN unrelated # job called `test` (`run: echo ok`) satisfied it — measured, and it escaped every # workflow-population guard in the repo, 588 tests green. if rel != GATING_WORKFLOW or GATING_JOB not in gates: faults.append( f"{rel}: job `{job_id}` publishes an image built from `{dockerfile}`, which carries " f"the SPA source, but does not transitively `needs:` `{GATING_JOB}` IN " f"`{GATING_WORKFLOW}` — the job holding the pinned suite step. It needs: " f"{sorted(gates)}. A `needs:` edge resolves within its own workflow, so a job named " f"`{GATING_JOB}` in a DIFFERENT workflow is a different job and gates nothing. Since " f"ersatztv#887 the image build does not run the suite itself, so this edge is the " f"ONLY thing between a red suite and a published image." ) assert not faults, "\n".join(faults) # ------------------------------------------------------------------------------------------------ # EFFECTIVE — the gating step really runs, really fails, and really runs everything # ------------------------------------------------------------------------------------------------ def test_the_GATING_STEP_is_exactly_what_was_reviewed() -> None: """Its body, its `if:`, and the absence of `continue-on-error` — all pinned. Pinning the BODY is what makes "unfiltered" and "not suppressed" true without parsing: a `--exclude`, a `|| true`, a pipe, a `set +e`, a `true || npm test` that never executes, are each a different string from the pin. Recognising those individually was tried three times and missed `true || npm test` and `continue-on-error: ${{ … }}` among others. """ named = [ candidate for candidate in _steps(workflow_jobs(GATING_WORKFLOW).get(GATING_JOB, {})) if str(candidate.get("name", "")).strip() == GATING_STEP_NAME ] assert len(named) == 1, ( f"{GATING_WORKFLOW}: job `{GATING_JOB}` has {len(named)} steps named {GATING_STEP_NAME!r}, " f"not one. That name is what both this assertion and the substring sweep key on, so a " f"SECOND step wearing it inherits the exemption and is never checked — measured by cold " f"review with a duplicate carrying `continue-on-error: true` and a spec filter." ) step = named[0] faults: list[str] = [] body = step.get("run") # PER LINE, not over the whole body: `_normalise` collapses newlines, so changing `run: |` to # `run: >` folds the two lines into one command whose normalised text is byte-identical to the # pin. MEASURED — the marker script then swallows the suite as its arguments. if not isinstance(body, str) or _normalise_lines(body) != _normalise_lines(GATING_STEP_RUN): faults.append( f"the gating step's `run:` is not the pinned command.\n pinned: {GATING_STEP_RUN!r}\n" f" actual: {body!r}\n Anything but this — a spec filter, a `|| true`, a pipe, a " f"`set +e`, a suite on the right of a `||` that may never execute — changes whether a " f"red suite blocks the image. Update the pin deliberately, in the same commit, with a " f"reason." ) workdir = step.get("working-directory") if workdir is None or str(workdir).strip() != GATING_STEP_WORKDIR: faults.append( f"the gating step's `working-directory:` is {workdir!r}, not the pinned " f"{GATING_STEP_WORKDIR!r}. `npm test` run anywhere else runs a different package, or " f"none, while the pinned command text is unchanged." ) shell = ((yaml.safe_load(_read(GATING_WORKFLOW)) or {}).get("defaults") or {}).get("run", {}).get("shell") if shell is None or str(shell).strip() != GATING_WORKFLOW_SHELL: faults.append( f"{GATING_WORKFLOW}'s `defaults.run.shell` is {shell!r}, not the pinned " f"{GATING_WORKFLOW_SHELL!r}. `bash` here means `bash -e`, which is what makes a failing " f"command fail the step; changing it changes whether a red suite blocks the image, " f"without touching the step." ) if stage_shell_overrides(): faults.append( f"a pinned Dockerfile stage carries a `SHELL` instruction: {stage_shell_overrides()}. " f"That redefines what every later `RUN` executes while leaving its text — and therefore " f"the pin — unchanged." ) condition = step.get("if") if condition is None or _normalise(str(condition)) != _normalise(GATING_STEP_IF): faults.append( f"the gating step's `if:` is not the pinned condition.\n pinned: {GATING_STEP_IF!r}\n" f" actual: {condition!r}\n A third term here lets the suite not run at all while the " f"image still publishes." ) # ANY spelling, not two literals. `continue-on-error: ${{ … }}` is the idiomatic conditional form # and a check comparing against `True`/`"true"` waves it through — a presence test cannot see # polarity, and this one is fail-OPEN in exactly the direction that matters. job = workflow_jobs(GATING_WORKFLOW).get(GATING_JOB, {}) for scope, holder in (("step", step), ("job", job)): if "continue-on-error" in holder and holder["continue-on-error"] not in (False, "false"): faults.append( f"the gating {scope} carries `continue-on-error: {holder['continue-on-error']!r}`, " f"so a red suite reports success and the `needs:` edge that gates the image blocks " f"nothing. Only absent, `false` or `False` is acceptable here." ) if "shell" in step: faults.append( f"the gating step carries its own `shell: {step['shell']!r}`, which overrides the pinned " f"workflow default. The default is pinned because `bash` means `bash -e`; a step-level " f"override moves that decision somewhere nothing checks." ) if "defaults" in job: faults.append( f"the job `{GATING_JOB}` carries `defaults: {job['defaults']!r}`, which overrides the " f"pinned workflow default for every step in it, including the gating one." ) if "if" in job: faults.append( f"the job `{GATING_JOB}` carries a job-level `if:` ({job['if']!r}). None did when " f"ersatztv#887 made this `needs:` edge the only gate, and a job that does not run cannot " f"gate anything — Gitea reports it `skipped`, and what that does to a dependent job is " f"not something this repo relies on." ) assert not faults, "\n".join(faults) def test_the_npm_SCRIPT_MAP_is_exactly_what_was_reviewed() -> None: """The whole map, because a pinned `RUN npm …` executes whatever it says. Pinning only the scripts whose body contains `vitest` is a SELECTOR — the category this file elsewhere calls the worst-behaved, because going short is silent. It goes short four ways, each re-arming ersatztv#887 in the gitless stage with every other pin matching: `npm run test`, `npm t`, and the `prebuild`/`preinstall` LIFECYCLE HOOKS, which npm runs for `npm run build` and `npm ci` without anything naming them. Pinning the map removes the category: a script that does not exist cannot be a hook, and one that changes is not equal. """ scripts = json.loads(_read(PACKAGE_JSON)).get("scripts", {}) assert scripts == PINNED_PACKAGE_SCRIPTS, ( f"`{PACKAGE_JSON}`'s scripts are not the pinned map.\n pinned: {PINNED_PACKAGE_SCRIPTS}\n" f" actual: {scripts}\n Both pinned `RUN npm …` lines in the gitless `web-build` stage " f"execute whatever this says, INCLUDING npm's `pre*`/`post*` lifecycle hooks, which nothing " f"in the Dockerfile names. A suite run added here reaches that stage with no Dockerfile or " f"workflow line changed — ersatztv#887 through a file nobody would think to check against " f"this guard. Update the pin deliberately, in the same commit, with a reason." ) def test_the_VITE_CONFIG_is_exactly_what_was_reviewed() -> None: """The whole file, because every partial match of it was defeated. `test.exclude`/`include` decide which specs the gating run collects, and `plugins:` can shell out to the suite from a build hook — both inside a file the Dockerfile never mentions. Pinning a BLOCK of it was defeated seven measured ways: a decoy copy above `defineConfig` with the real member respelled `test: {`, `test : {` or `"test": {`; the same for `plugins:`; and two that never touched the marker — `[…].concat([evil])` and a trailing `...moreTest` spread, which replaces the pinned object because `defineConfig` is identity in both vite and vitest. Comparing the file entire has no marker to respell and nothing after the span. WHAT THE COMPARISON CANNOT SEE, measured: `_normalise_lines` drops blank lines and collapses whitespace WITHIN a line, so a reflow, an indentation change and a change to the spacing inside a STRING LITERAL are all invisible. The first two carry no meaning; the third could, and does not here — no string in this file has consequential internal spacing. Line ORDER and any token change are caught (both measured). Stated because "pinned whole" invites the reader to assume byte equality, and it is not. """ actual = _normalise_lines(_read(VITE_CONFIG)) pinned = _normalise_lines(PINNED_VITE_CONFIG) assert actual == pinned, ( f"`{VITE_CONFIG}` is not the pinned file. First difference:\n" + "\n".join( f" line {index + 1}:\n pinned: {pinned[index] if index < len(pinned) else ''}\n" f" actual: {actual[index] if index < len(actual) else ''}" for index in range(max(len(actual), len(pinned))) if index >= min(len(actual), len(pinned)) or actual[index] != pinned[index] ) + "\n\n This file decides which specs the gating run collects and what `vite build` loads " "in the gitless stage. Any edit that changes a line's TOKENS reddens — a comment's words " "included, whitespace and blank lines not — deliberately, because every " "partial match of it was defeated. Update `PINNED_VITE_CONFIG` in the same commit, with a " "reason." ) def test_NO_OTHER_vitest_CONFIG_outranks_the_pinned_one() -> None: """A pinned config is worthless if a second file can take precedence over it. Vitest prefers `vitest.config.*` (and `vitest.workspace.*`/`vitest.projects.*`) to `vite.config.*`. MEASURED: dropping a `web/vitest.config.ts` with `include: ['nope/**']` and `passWithNoTests: true` beside the pinned file made the suite collect zero specs and exit 0 — the gating step green having run nothing at all, which is worse than the filtered run ersatztv#887 removed. The construct is refused rather than modelled: there is no such file today, so the honest move is to assert none appears. """ intruders = sorted( path for path in tracked_files._git_ls_files() # DIRECT children of `web/` only: vite resolves its config from the project ROOT, so a # `web/e2e/vite.config.ts` outranks nothing — and `web/e2e/` is a real directory. Reddening # on it would be fail-noisy with a message that is simply false. if path.startswith("web/") and "/" not in path[len("web/") :] and path.rpartition("/")[2].startswith(OUTRANKING_CONFIG_PREFIXES) and path != VITE_CONFIG ) assert not intruders, ( f"these files outrank the pinned `{VITE_CONFIG}` in vite's or vitest's config resolution: " f"{intruders}. Whatever they say decides what the gating run COLLECTS and what `vite build` " f"loads in the gitless stage, and the pin on `{VITE_CONFIG}` cannot see either — a suite " f"that collects nothing still exits 0, and a plugin loaded from a `.js` config still runs. " f"Either fold the settings into `{VITE_CONFIG}` and its pin, or pin this file too." ) def test_the_DOCS_ONLY_arm_cannot_publish_an_image() -> None: """The premise under half of `GATING_STEP_IF`, asserted rather than assumed. The gating step is allowed to skip when `docs_only` is true. That is only safe because the publish step skips on the same condition, so the arm ships nothing. Stating that in prose with nothing checking it is the same shape as the defect one level in. """ step = find_step(GATING_WORKFLOW, "build", PUBLISH_STEP_NAME) assert step is not None, f"{GATING_WORKFLOW}: job `build` has no step named {PUBLISH_STEP_NAME!r}." condition = step.get("if") assert condition is not None and _normalise(str(condition)) == _normalise(PUBLISH_STEP_IF), ( f"the publish step's `if:` is {condition!r}, not the pinned {PUBLISH_STEP_IF!r}. The gating " f"step is permitted to skip on `docs_only`, and that is safe ONLY because this step skips " f"on it too. Without this, a docs-only push to `main` skips the SPA suite and publishes an " f"image anyway." ) def test_only_the_PINNED_step_mentions_the_suite() -> None: """A substring sweep, deliberately not a semantic predicate. Deciding whether a command RUNS the suite was wrong nine times. This asks the much weaker question — does any OTHER step's shell body mention it at all — whose failure mode is a false red asking a human to look, never a false green. """ strays: list[str] = [] for path in workflow_files(): rel = path.relative_to(REPO_ROOT).as_posix() for job_id, job in workflow_jobs(rel).items(): for step in _steps(job): body = step.get("run") if not isinstance(body, str): continue if ( rel == GATING_WORKFLOW and job_id == GATING_JOB and str(step.get("name", "")).strip() == GATING_STEP_NAME ): continue hit = [mention for mention in SUITE_MENTIONS if mention in body] if hit: strays.append(f"{rel} :: job `{job_id}` step {str(step.get('name', '?'))!r} mentions {hit}") assert not strays, ( "these steps mention the SPA suite and are not the pinned gating step:\n " + "\n ".join(strays) + "\n\nThis is a SUBSTRING sweep, not a claim that they run it — look, and either move the " "run into the pinned step or record why this one is not a second, unpinned gate." ) def test_no_DOCKERFILE_outside_a_pinned_stage_mentions_the_suite() -> None: """The same weak sweep over Dockerfiles, covering stages that carry no SPA source. A stage without `web/` has no suite to run, so `PINNED_STAGE_COMMANDS` does not cover it. This reports a mention there anyway rather than leaving the class unexamined. """ strays: list[str] = [] for rel in dockerfiles(): for name, lines in stages(_read(rel)): if f"{rel}::{name}" in PINNED_STAGE_COMMANDS: continue for line in lines: hit = [mention for mention in SUITE_MENTIONS if mention in line] if hit: strays.append(f"{rel} :: stage `{name}`: {line!r} mentions {hit}") assert not strays, "these unpinned Dockerfile stages mention the SPA suite:\n " + "\n ".join(strays) # ------------------------------------------------------------------------------------------------ # Anti-vacuity and negative controls # ------------------------------------------------------------------------------------------------ def test_no_run_BODY_builds_or_pushes_an_image() -> None: """`publishing_jobs()` keys on the ACTION, which is a selector; assert nothing publishes around it. A job doing `docker build -f ./docker/Dockerfile … && docker push …` in a shell body is not in that population, so the positive invariant is silently not applied to it — and anti-vacuity does not notice, because the real `build` job is still there. MEASURED MISSED without this assertion, which is why it is asserted rather than reasoned away. """ stray: list[str] = [] for path in workflow_files(): rel = path.relative_to(REPO_ROOT).as_posix() for job_id, job in workflow_jobs(rel).items(): for step in _steps(job): body = step.get("run") if not isinstance(body, str): continue for line in body.splitlines(): if line.lstrip().startswith("#"): continue if re.search(r"\bdocker\s+(?:push|build|buildx\s+build)\b", line): stray.append(f"{rel} :: job `{job_id}`: {line.strip()!r}") assert not stray, ( f"image build/push outside `{PUBLISH_ACTION}`:\n " + "\n ".join(stray) + "\n\n" "`publishing_jobs()` derives from that action, so these jobs are NOT checked for the suite " "gate — extend it before adding one." ) def test_the_populations_are_NOT_empty() -> None: """Every assertion above passes over an empty set, which is how a completeness guard dies.""" assert dockerfiles(), "`git ls-files` reported no Dockerfile — the derivation is broken, not the repo." assert workflow_files(), f"`git ls-files` reported no {WORKFLOW_DIR}/*.yml — the derivation is broken." carrying = spa_carrying_stages() assert carrying, ( f"no stage in {dockerfiles()} was read as carrying the SPA source, so the negative assertion " "checked nothing and the positive one skipped every job. `copies_spa_source` or `stages` " "stopped matching." ) assert PINNED_STAGE_COMMANDS, "the pin table is empty; the negative assertion asserts nothing." published = publishing_jobs() assert published, ( f"no job in {WORKFLOW_DIR} uses `{PUBLISH_ACTION}`, so the gate assertion ran over an empty " "population. Either the repo stopped publishing images or `_canonical_action` stopped " "recognising the action." ) assert any(dockerfile in {rel for rel, _, _ in carrying} for _, _, dockerfile, _ in published), ( "no publishing job builds an SPA-carrying Dockerfile, so the gate assertion is vacuous." ) def test_the_STAGE_SPLIT_finds_the_real_stages() -> None: """`stages()` is the only Dockerfile syntax this file interprets; pin what it returns. A split that stopped recognising `FROM … AS name` would put every instruction in one bucket, the pin key would never match, and `test_every_SPA_CARRYING_STAGE_runs_exactly_its_pinned_commands` would report an orphan pin — loud, but for the wrong reason. Pinned so the cause is legible. """ names = [name for name, _ in stages(_read("docker/Dockerfile"))] assert names[:3] == ["dotnet-runtime", "web-build", "runtime-base"], names assert [stage for _, stage, _ in spa_carrying_stages()] == ["web-build"], spa_carrying_stages() def test_the_SPA_CARRYING_predicate_reads_every_COPY_SPELLING() -> None: """Pinned both ways: every stage answered False is SKIPPED by the negative assertion. The synthetic spellings were MEASURED False before the predicate was widened; a new stage using any of them would have gone unpinned and unchecked. """ for spelling in ( 'COPY ["web/", "/source/web/"]', 'COPY --chown=node:node ["web/", "/source/web/"]', "COPY web /source/web", "ADD web/ /source/web/", "COPY web/. ./web/", "COPY . /source/", "COPY ./ /source/", "COPY --chown=node:node web/ /source/web/", "COPY web/package*.json ./web/", # A GLOB that matches `web/`. Measured False before `_source_reaches_web`, which left a # stage copying the source this way unpinned and unchecked. "COPY web*/ ./", "COPY * /source/", ): assert copies_spa_source(spelling), spelling # A STAGE copy can still carry the SPA source across. Measured False before this clause: the # receiving stage was then unpinned, so a suite run in it was invisible. assert copies_spa_source("COPY --from=web-build /source/web /source/web") assert copies_spa_source("COPY --from=builder /app/web ./web") # The EXEC form of the same thing. Measured False before the parse order was fixed — splitting # `["/source/web", "/dest"]` on whitespace left a trailing comma on the first token, so the # segment test never matched. Pinned here because only the development battery witnessed it, and # that battery is explicitly not standing. assert copies_spa_source('COPY --from=web-build ["/source/web", "/dest"]') assert copies_spa_source('COPY --from=web-build ["/source/web","/dest"]') assert not copies_spa_source('COPY --from=web-build ["/source/ErsatzTV/wwwroot/app/.", "/dest"]') for spelling in ( "COPY design-system/. ./design-system/", "COPY --from=web-build /source/ErsatzTV/wwwroot/app/. ./app/", "COPY webhooks/. ./webhooks/", # `web` must be a whole path segment "RUN npm ci", "COPY web/", # a single argument is not a copy ): assert not copies_spa_source(spelling), spelling def test_the_INSTRUCTION_JOIN_survives_what_defeated_the_PARSER() -> None: """The shapes that broke the parser must not break the pin comparison. None of these needs interpreting — they only need to arrive at the comparison intact, so that a stage running them is NOT EQUAL to its pin. Each was a measured false green when this file tried to decide what the command meant. """ heredoc = "RUN < None: """The nine shapes that defeated the predicate, run against the pin instead. Not one of them is equal to `RUN npm run lint && npm run typecheck && npm run build`, which is the entire argument for replacing the mechanism: the pin does not need to recognise a spelling to reject it. """ pinned = PINNED_STAGE_COMMANDS["docker/Dockerfile::web-build"] for spelling in ( "RUN <