timothy/ersatztv
1
0
Fork 0
Code Issues 51 Pull Requests 5 Actions Packages Projects Releases 9 Wiki Activity

fix(887): the pin's real limit, measured — and three absolutes it does not support

Browse Source 
A fourth cold review attacked the pin itself. BLOCKER empty, the mechanism upheld, and
every finding was prose claiming more than I had measured — plus one one-token gap that
was live.

THE ABSOLUTES, refuted by execution and now corrected in all three places they appeared
(guard docstring, `docs/guard-inventory.md`, the decision record's `rule:`):

  * "A pin cannot produce a false green." True only in the trivial reading. A pin is
    immune to a different SPELLING of the command — the entire class that defeated the
    parser nine times — and is NOT immune to the same text MEANING something else. Two
    mutants re-armed ersatztv#887 through `web/package.json` alone: `RUN npm run build`
    executes whatever that file says, so `"build": "vitest run && …"` puts the suite back
    into the gitless stage with every pin still matching, and `"prepare"` does it via
    `npm ci`. Now pinned: exactly one script may mention vitest, and its body is fixed.
  * Residual (1), "a stage that does not carry the SPA source is unpinned — correct,
    since without `web/` there is no suite there". False. The boundary is what
    `copies_spa_source` RECOGNISES, which is narrower than "has the suite available".
    Restated, with the case still outside it named: a stage copy that RENAMES the tree.
  * The substring sweep's "never a false green". Its reported failures are false reds;
    what it fails to REPORT is not. `SUITE_MENTIONS` is a hand-written SELECTOR — a third
    category beside population and pin, and the worst-behaved, because a population going
    short is caught by an equality and a stale pin reddens loudly, while a selector going
    short is silent. It was short by exactly one entry: `npm t`, npm's own alias, which
    this guard already names among the spellings that defeated the parser. A stage
    running `npm t -- --run` escaped it. Fixed, and the category is now named.

ALSO CLOSED: `run: |` -> `run: >` folded the two-line body into one command whose
whitespace-normalised text was byte-identical to the pin, so the marker script swallowed
the suite as its arguments — the body is now compared LINE BY LINE, since a newline
separates two commands. A SECOND step named `Test SPA` inherited the exemption both the
pin lookup and the sweep key on; exactly one is now required. And `COPY web*/` — a glob
that matches `web/` — was read as not carrying the source, leaving the receiving stage
unpinned.

Battery 45 -> 51, 0 missed. The remaining meaning-change route, an `ENV` rewriting `PATH`
so a pinned `RUN` resolves a different `npm`, is not modelled and is recorded as a
residual rather than implied away.

Refs: #887
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019T79beF1Ufid3dXju4yqkF
This commit is contained in:
timothy
2026-08-30 13:37:58 +02:00
co-authored by Claude Opus 5
parent b53c3bab47
commit 566cabea6d
4 changed files with 113 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/tests/test_image_build_delegates_the_spa_suite.py
+107 -20
View File
@@ -40,8 +40,18 @@ Nine defects from one mechanism is `process.enumerate-workaround-behaviors-befor
to remove it rather than patch a tenth time, and the mechanism is removed here. **Nothing in this
file decides what a command MEANS.** The commands that may run in the risky places are PINNED as
text; anything else — a heredoc, a wrapper, a new spelling, a rewrite — is simply not equal to the
pin and reddens. A pin cannot have a false green, only a false red, and a false red is a human
reading a diff they should have read anyway.
pin and reddens.
WHAT A PIN IS AND IS NOT IMMUNE TO, stated precisely because an earlier version of this paragraph
claimed the absolute "a pin cannot have a false green" and cold review refuted it by execution. A
pin is immune to the class that defeated the parser: a DIFFERENT SPELLING of the command. It is not
immune to the same text MEANING something else, and there are five such routes. Four are closed —
`working-directory`, `defaults.run.shell`, a stage `SHELL`, and `web/package.json`'s scripts, since
`RUN npm run build` executes whatever that file says — and each is pinned or refused below. The
fifth, an `ENV` rewriting `PATH` so a pinned `RUN` resolves a different `npm`, is not modelled and
is stated as a residual rather than implied away. So: the pin removes the spelling problem entirely
and shrinks the meaning problem to an enumerated list, which is a much better place to be than a
predicate that was wrong nine times — but it is not "cannot be wrong".
The cost is stated rather than hidden: a legitimate edit to those command lines reddens this guard
and its author must update the pin. That is the same deliberate review trigger `KNOWN_GATE_*` uses,
@@ -65,13 +75,24 @@ red). Only the second is safe to write by hand.
WHAT IS NOT COVERED, stated rather than implied:
* A Dockerfile stage that does NOT carry the SPA source is unpinned. It could run the suite and
this guard would not care — correctly, since without `web/` there is no suite there to run. The
decision rests on `copies_spa_source`, whose false-NEGATIVE direction is the one that would drop
a stage out of the check; it is pinned in both directions by its own test.
* A step OTHER than the pinned one could run the suite. `test_only_the_PINNED_step_mentions_the_suite`
reports any `run:` body containing `npm test` or `vitest` as a substring — a substring, not a
semantic predicate, so its failure mode is a false red asking a human to look.
* A Dockerfile stage that does not carry the SPA source is unpinned. The boundary is what
`copies_spa_source` recognises — a context copy of `web/` or `.`, a matching GLOB, or a stage
copy whose source names `web` — and NOT the wider "has the suite available to run", which is
what an earlier version of this sentence claimed. A stage copy that RENAMES the tree on the way
in (`COPY --from=web-build /source/web /elsewhere`) has no `web` segment in its source and is
not recognised. The false-NEGATIVE direction is the one that drops a stage out of the check, so
it is pinned in both directions by its own test.
* A step or stage OTHER than a pinned one could run the suite, and the substring sweep is what
reports it. `SUITE_MENTIONS` is a hand-written SELECTOR — a third category the population/pin
split does not name, and the one that behaves worst: a population going short is caught by an
equality, a pin going stale goes loudly red, but a SELECTOR going short is silent. It was short
by exactly one entry (`npm t`, npm's own alias, which this file names among the spellings that
defeated the parser) until cold review measured a stage escaping through it. `PUBLISH_ACTION` is
the same category and is covered by the anti-vacuity assertion instead.
* The gating body is compared LINE BY LINE with each line whitespace-normalised, not byte-exact.
Indentation and reflow are free; line boundaries are not, because a newline separates two
commands — changing `run: |` to `run: >` folds them into one and the whole-text comparison could
not see it.
* Gitea's own evaluation of a skipped `needs:` job is not modelled; the guard forbids the job-level
`if:` that would create one rather than reasoning about it.
"""
@@ -80,6 +101,7 @@ from __future__ import annotations
import json
import re
from fnmatch import fnmatch
from pathlib import Path
import yaml
@@ -139,9 +161,20 @@ GATING_WORKFLOW_SHELL = "bash"
PUBLISH_STEP_NAME = "Build and push"
PUBLISH_STEP_IF = "steps.detect.outputs.docs_only != 'true'"
# `RUN npm run build` and `RUN npm ci` execute whatever `web/package.json` says they do, and that
# file is neither a Dockerfile nor a workflow. Cold review re-armed ersatztv#887 through it twice
# without touching any pinned line: `"build": "vitest run && tsc -b && vite build"` puts the suite
# back into the gitless stage, and `"prepare"` is run by `npm ci`. So the SCRIPTS THAT MENTION VITEST
# are pinned too — exactly one, and its body.
PACKAGE_JSON = "web/package.json"
PINNED_VITEST_SCRIPTS: dict[str, str] = {"test": "vitest"}
# Substrings that mean "a human should look at this step". NOT a semantic predicate — the pinned step
# is allowed to contain them and every other step is not.
SUITE_MENTIONS = ("npm test", "npm run test", "vitest")
SUITE_MENTIONS = ("npm t", "npm run test", "vitest")
# `npm t` rather than `npm test`: it is npm's own documented alias, it SUBSUMES `npm test` as a
# substring, and this file enumerates it among the spellings that defeated the parser. Leaving it out
# meant an unpinned stage running `npm t -- --run` passed the sweep — measured by cold review.
def _normalise(text: str) -> str:
@@ -149,6 +182,15 @@ def _normalise(text: str) -> str:
return " ".join(text.split())
def _normalise_lines(text: str) -> list[str]:
"""Per-line normalisation, so indentation is free but the LINE BOUNDARIES are not.
A newline separates two commands; collapsing it merges them into one. That is not a cosmetic
difference and the whole-text form cannot see it.
"""
return [_normalise(line) for line in text.strip().splitlines() if line.strip()]
def _canonical_action(uses: str) -> str:
"""The last two path segments of a `uses:` value, without its `@version`.
@@ -264,11 +306,22 @@ def copies_spa_source(command: str) -> bool:
arguments = remainder.split()
if len(arguments) < 2:
return False
for source in arguments[:-1]:
normalised = source.removeprefix("./").rstrip("/")
if normalised in {"", ".", "web"} or normalised.startswith("web/"):
return True
return False
return any(_source_reaches_web(source) for source in arguments[:-1])
def _source_reaches_web(source: str) -> bool:
"""Could this COPY source bring `web/` in?
Exact (`web`, `web/…`), whole-context (`.`, `./`), and GLOB: `COPY web*/ ./` matches `web/` and
so carries the source, while `webhooks/` does not — measured False for the glob by cold review,
which left the receiving stage unpinned. `fnmatch` against the literal segment `web` is what
separates the two without hand-listing patterns.
"""
normalised = source.removeprefix("./").rstrip("/")
if normalised in {"", ".", "web"} or normalised.startswith("web/"):
return True
head = normalised.split("/")[0]
return any(char in head for char in "*?[") and fnmatch("web", head)
def spa_carrying_stages() -> list[tuple[str, str, list[str]]]:
@@ -451,15 +504,25 @@ def test_the_GATING_STEP_is_exactly_what_was_reviewed() -> None:
a different string from the pin. Three earlier versions tried to recognise those individually and
missed `true || npm test` and `continue-on-error: ${{ … }}` among others.
"""
step = find_step(GATING_WORKFLOW, GATING_JOB, GATING_STEP_NAME)
assert step is not None, (
f"{GATING_WORKFLOW}: job `{GATING_JOB}` has no step named {GATING_STEP_NAME!r}. That step is "
f"the only thing running the SPA suite, and the image is gated on the job holding it."
named = [
candidate
for candidate in _steps(workflow_jobs(GATING_WORKFLOW).get(GATING_JOB, {}))
if str(candidate.get("name", "")).strip() == GATING_STEP_NAME
]
assert len(named) == 1, (
f"{GATING_WORKFLOW}: job `{GATING_JOB}` has {len(named)} steps named {GATING_STEP_NAME!r}, "
f"not one. That name is what both this assertion and the substring sweep key on, so a "
f"SECOND step wearing it inherits the exemption and is never checked — measured by cold "
f"review with a duplicate carrying `continue-on-error: true` and a spec filter."
)
step = named[0]
faults: list[str] = []
body = step.get("run")
if not isinstance(body, str) or _normalise(body) != _normalise(GATING_STEP_RUN):
# PER LINE, not over the whole body: `_normalise` collapses newlines, so changing `run: |` to
# `run: >` folds the two lines into one command whose normalised text is byte-identical to the
# pin. Measured by cold review — the marker script then swallows the suite as its arguments.
if not isinstance(body, str) or _normalise_lines(body) != _normalise_lines(GATING_STEP_RUN):
faults.append(
f"the gating step's `run:` is not the pinned command.\n pinned: {GATING_STEP_RUN!r}\n"
f" actual: {body!r}\n Anything but this — a spec filter, a `|| true`, a pipe, a "
@@ -522,6 +585,26 @@ def test_the_GATING_STEP_is_exactly_what_was_reviewed() -> None:
assert not faults, "\n".join(faults)
def test_only_the_PINNED_npm_SCRIPTS_run_vitest() -> None:
"""A pinned `RUN` executes whatever `web/package.json` says; pin that too.
The Dockerfile stage runs `npm ci` and `npm run build`. Neither line changes when a script body
does, so `"build": "vitest run && …"` or a `"prepare"` that runs vitest re-arms ersatztv#887 with
every pin still matching — measured by cold review, twice. A substring test on `vitest`, not a
predicate: it decides which scripts must be pinned, and going short is caught by the equality
below rather than by the substring.
"""
scripts = json.loads(_read(PACKAGE_JSON)).get("scripts", {})
mentioning = {name: body for name, body in scripts.items() if isinstance(body, str) and "vitest" in body}
assert mentioning == PINNED_VITEST_SCRIPTS, (
f"the `{PACKAGE_JSON}` scripts mentioning vitest are {mentioning}, not the pinned "
f"{PINNED_VITEST_SCRIPTS}. Every `RUN npm …` in a pinned Dockerfile stage executes whatever "
f"these say, so a suite run added here reaches the gitless stage with no pinned line "
f"changed — which is ersatztv#887 re-armed through a file nobody would think to check "
f"against this guard. If the change is deliberate, update the pin and say why."
)
def test_the_DOCS_ONLY_arm_cannot_publish_an_image() -> None:
"""The premise under half of `GATING_STEP_IF`, asserted rather than assumed.
@@ -647,6 +730,10 @@ def test_the_SPA_CARRYING_predicate_reads_every_COPY_SPELLING() -> None:
"COPY ./ /source/",
"COPY --chown=node:node web/ /source/web/",
"COPY web/package*.json ./web/",
# A GLOB that matches `web/`. Measured False before `_source_reaches_web`, which left a
# stage copying the source this way unpinned and unchecked.
"COPY web*/ ./",
"COPY * /source/",
):
assert copies_spa_source(spelling), spelling