Round three found the one half of the new mechanism with no relevance gate. `verify_claim`'s GREEN path read exactly two things — the run exited 0, and something PASSED — and both are satisfied by a proof that never touches the mutated file at all. Reproduced before fixing: retargeting the shipped GREEN entry's proof from `test_check_doc_narrative.py` to `test_bom_guard_detection.py` changed nothing, and the entry still reported verified. The RED direction never had this hole, because a proof that ignores the mutation stays green and is refused as "the clause is not load-bearing". So a GREEN entry now declares a `reach_replacement` and its `reach_expect`: a SECOND mutation of the SAME clause, required to REDDEN the same proof, executed through `verify_mutation` so its red is read through the diagnostic gate rather than on exit status. The shipped entry declares `path = p` — dropping the `b/` stripping every scanned diff header goes through — and the run then scans NOTHING, which is what the declared diagnostic reads. The same retarget now fails, naming the reach verdict. The gate runs LAST of the three: run first it would refuse before the status and vacuity gates were read and neither could be witnessed failing alone (#685), and the sandbox is reset between a claim's two proof runs for the reason it is reset between mutations. It has its own disarm proof, and the two synthetic claim sandboxes are now real git repositories so `reset_sandbox` has a baseline; `_lib_with` shares the baseline registry, since a copied module's own starts empty. Also from that round: - The record no longer counts the mutation-outcome claims in the pinned proposal-3 scan. A third of the same shape sits in the same result set (`test_a_verdict_BEYOND_A_SHORT_PAGE_is_still_found`), and which side of the line a sentence falls on is a judgement, so an exact count is a figure the next reader re-derives differently — the failure this record is about. - The calibration paragraph no longer restates the post-review-verdict outcome as a dated witnessing. It points at the `CLAIMS` entry that executes it, which is the form the rewritten shell comment beside it demands. - The comment in `check-doc-narrative.py` claimed a universal ("reddens no test") while one file is executed. It now names that file, so the quote binds an outcome no wider than what is checked. - Proposal 4 from the issue is dispositioned explicitly: rejected as a rule here, on the issue's own argument that an exhortation does not fire at the moment of least slack. - `docs/README.md`'s task-signal parenthetical now names the `CLAIMS` population; the file was owned by another slot when this branch started. Cost re-measured 2026-09-05, three baseline/branch pairs: the `CLAIMS` half adds 31.7-43.6%, up from the 12.7-16.6% measured before the gate existed. The old figure is retired rather than scaled — growing the population invalidates the measurement that described it. Refs #881 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QqCpYFsKgnAnx6jVwrKiV
676 lines
88 KiB
Markdown
676 lines
88 KiB
Markdown
# Guard inventory (ersatztv#774 / #775)
|
||
|
||
Every executable guard **file** in this repo, what it blocks, and whether it ships a proof it can go
|
||
red. `scripts/tests/test_guard_inventory.py` derives the population from the **git index** and the
|
||
workflow/hook call sites and asserts **set equality** against the `Guard` column, so a new guard
|
||
cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.
|
||
The index rather than a filesystem walk since ersatztv#806 — the practical consequence is that a new
|
||
guard joins the population when it is **staged**, not when the file appears. Nothing local runs
|
||
these checks at all: `.husky/pre-commit` runs lint-staged, the decisions guard, the root-PNG check
|
||
and `dotnet format`, and `grep -rn pytest .husky/` returns nothing. The runner is
|
||
`pr-checks.yml::script-tests`, `on: pull_request`, so the red arrives in CI — plus
|
||
`docker-build.yml:723` on the release path, which re-runs two of these files
|
||
(`test_ci_dropped_step_guard.py` and `test_ci_release_path_scan_job.py`) as a `needs:` of `build`.
|
||
|
||
**Read `docs/decisions/records/testing/guard-derives-population-from-source.md` and
|
||
`…/guard-ships-with-mutation-proof.md` before editing a guard or adding a row.**
|
||
|
||
## Columns
|
||
|
||
- **Kind** — `GUARD` (it can block a commit, a push, a tool call, a merge or a CI job), `TOOLING`
|
||
(it does work but asserts nothing; listed so its absence from the guard set is a recorded
|
||
decision, not an oversight), or `PROOF` (a `scripts/tests/` file whose job is to prove another
|
||
guard). `PROOF` exists to stop a regress: once test files entered the population, every mutation
|
||
proof became a row wanting a proof of its own. A `scripts/tests/` file that enforces a repo
|
||
invariant with no separate guard behind it is a `GUARD`, graded normally, and may cite a mutation
|
||
case in its own file.
|
||
- **Proof** — `MUTATION`: a clause-level mutation **was executed** and this named test **was
|
||
witnessed red**. The test either performs the disarm itself (a monkeypatch, a deselection, a
|
||
removed marker) or a recorded incident shows it going red when the clause went away.
|
||
`BEHAVIOUR-ONLY`: a real test drives the guard through its real entry point with good and bad
|
||
input, but no one has demonstrated that removing the clause reddens it. `NONE`.
|
||
- **Proof ref** — `file.py::function`, verified to exist by the inventory test.
|
||
|
||
The distinction between `MUTATION` and `BEHAVIOUR-ONLY` is the whole point of #775 and is not a
|
||
grading curve. A behavioural test proves the guard reacts to its input; only a mutation proves the
|
||
guard is *load-bearing*. #685 shipped two guards on one condition where deleting either left the
|
||
suite green, and every behavioural test passed throughout.
|
||
|
||
**The definition above is the second one.** The first read "a named test disarms this guard's clause
|
||
and asserts red", and three rows were graded `MUTATION` against it that do not disarm anything —
|
||
they feed the real script an input only that clause rejects (empty stdin, a short page, a full first
|
||
page). The objection to that is right twice over: those tests are the same
|
||
species as ones graded `BEHAVIOUR-ONLY` eight rows away, so the column was being applied as a curve
|
||
on the very day it was introduced. They are regraded. The surviving criterion is *witnessed*, not
|
||
*plausible* — "removing this clause would surely redden that test" is an argument, and this table
|
||
exists because arguments of that shape have been wrong here six times.
|
||
|
||
## Scope limit, stated rather than implied
|
||
|
||
The `## Inventory` table covers guard **files**, discovered by reading `.claude/hooks/*.sh`,
|
||
`.husky/*` and `scripts/tests/test_*.py` **out of the git index**, plus every `scripts/…` path
|
||
referenced by a workflow or a hook. Guard **jobs** are covered separately by
|
||
`## Workflow-job guards` below. The classes below are outside BOTH populations, and are enumerated
|
||
rather than summarised: an unlisted class reads as covered, and guards this inventory itself shipped
|
||
with have sat in these gaps. Treat anything not named here as unassessed, not as included:
|
||
|
||
1. ~~**Guards inline in workflow YAML**~~ — **CLOSED by ersatztv#786**, and kept here rather than
|
||
deleted because the entry named the shape and the shape is worth keeping visible. `pr-checks.yml`'s
|
||
`ci-image-pin` — which stated an invariant it did not check (#774) — sat in this gap. Workflow jobs
|
||
now declare `env.CI_JOB_ROLE`, and `scripts/tests/test_workflow_job_guards.py` asserts set equality
|
||
between the guard-declaring jobs and `## Workflow-job guards` in both directions. The judgement
|
||
#774 said the filesystem cannot supply is still a human one; what changed is that it is now
|
||
RECORDED, next to the job, and a job that acquires no marker fails the suite. **Two residuals**:
|
||
the check proves each job was classified, never that the classification is CORRECT; and an
|
||
`inline` assertion cannot be proven to work without running the job, so those rows carry
|
||
`Proof: NONE` honestly instead of borrowing a neighbouring file's proof.
|
||
2. **C# and TypeScript guards** — seven files, none with a row:
|
||
`ErsatzTV.Tests/Integration/SearchIndexMutationCoverageTests.cs` (ersatztv#824 — derives the
|
||
`ISearchIndex` population from the declaring assembly and asserts set equality against a
|
||
hand-written covered set),
|
||
`ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `web/src/api/pageSizeCallSites.guard.test.ts`,
|
||
`web/src/api/completeRequest.guard.test.ts`, `web/src/api/completeAnnotations.guard.test.ts`
|
||
(added by ersatztv#820 — the second time this hand-written list has had to be extended by hand),
|
||
and the pair
|
||
`web/vite-plugins/trackedSourceFiles.test.ts` + its `.realgit.test.ts` sibling, which together
|
||
prove the population `pageSizeCallSites.guard.test.ts` derives (split because the `realgit` half
|
||
needs the git BINARY and its thirteen injected-runner siblings do not, so the prerequisite is
|
||
confined to one file — it is NOT the Docker exclusion it was originally written for, which
|
||
ersatztv#887 removed along with the whole in-image suite run). Note what that costs: this
|
||
list is a HAND-WRITTEN mirror of a population nothing derives, so it goes stale silently and CI
|
||
stays green — #807 added the third entry, and nothing mechanical caught that
|
||
`pageSizeCallSites.guard.test.ts` had become the only one named.
|
||
Since #819 that guard derives its file population from the git index like the Python ones, which
|
||
makes it read as covered by `test_guard_populations_derive_from_git.py`. It is not: that register
|
||
detects Python modules IMPORTING `scripts/tests/tracked_files.py`, and
|
||
`test_every_index_derived_module_is_registered`'s docstring names this gap ("a module deriving a
|
||
file population some other way — shelling out to `git ls-files` itself, or going back to
|
||
`Path.rglob` — is invisible to it"). So the TS guard's derivation is re-checked by review, not
|
||
mechanically.
|
||
3. **Mentions counted as call sites.** The `scripts/…` scrape matches any occurrence, including
|
||
inside a comment or an `::error::` string. `scripts/update-openapi.sh` is named in a
|
||
`pr-checks.yml` error message, so removing the step that runs it would leave its row intact.
|
||
4. **Nested and non-lowercase paths** beyond `scripts/tests/` — a guard under
|
||
`scripts/scripted-schedules/`, or with an uppercase name, is invisible to the scrape.
|
||
5. **Non-`.sh` hooks** — the hook pattern is `*.sh` only. (`.yaml` workflows are no longer in this
|
||
gap: the caller scan matches `*.yml` and `*.yaml` since #806, because Gitea accepts both.)
|
||
6. **Transitive calls** — a script invoked only by another script, rather than by a workflow or
|
||
hook, is not discovered. Being outside this population is NOT an exemption from proof:
|
||
`testing.verification-code-needs-its-own-proof` (#796) puts such a script in the ordinary branch
|
||
whenever a proof test can drive it hermetically. A row here is not the route — the population is
|
||
derived from the call sites, so one is rejected as a phantom — but the manifest can still name
|
||
the checker, as `target`, with its proof test as `guard`. `scripts/mcp_smoke.py` is the worked
|
||
example: no row of its own, proven by `scripts/tests/test_mcp_smoke.py`, which carries the row.
|
||
That row is `GUARD` citing itself while its declared `target` is a different file — an ESTABLISHED
|
||
shape here, not a new one, and deliberately not counted: `test_review_verdict_vocabulary.py`
|
||
targets `scripts/check-review-verdict.sh` and `test_mutation_harness.py` targets
|
||
`mutation_harness_lib.py`. Worth noting only because the self-citation carve-out in
|
||
`test_every_proof_ref_points_at_a_row_marked_PROOF` is worded for a checker guarding a repo
|
||
invariant, whereas these guard another file.
|
||
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
|
||
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
|
||
population and hold no rows. They are not guards (they assert nothing on their own), but the
|
||
middle two ARE what `test_mutation_harness.py` is made of, so gutting either would take that
|
||
guard with it. What catches that is the guard's own row: its declared mutation targets
|
||
`mutation_harness_lib.py`, and its proof test refuses to run if the clause it names has moved.
|
||
`tracked_files.py` has no such backstop — it is load-bearing for every module that imports it
|
||
(#806), and an edit to it is covered only by those modules' own proofs. Recorded rather than
|
||
force-fitted: a row for a library would need a `Kind` the vocabulary does not have.
|
||
8. **Nested workflow directories** — the workflow scope is direct children of `.gitea/workflows`, so
|
||
a tracked `.gitea/workflows/nested/x.yml` is invisible to the caller scan and to
|
||
`test_ci_image_pin_population.py`. Left as scope rather than widened: whether Gitea executes
|
||
nested workflow files was not verified here, and widening on an unverified premise risks a
|
||
permanent red on a correct tree, which is how a correct guard gets deleted.
|
||
|
||
Hook **wiring** is checked (`test_every_hook_file_is_actually_WIRED` reads `.claude/settings.json`
|
||
and the husky hooks with full-line comments stripped), so a hook file whose registration is deleted
|
||
fails rather than keeping a row that reads as coverage. Its limit, stated because the check reads
|
||
stronger than it is: **it is a substring test for the basename, not a parse of the invocation.**
|
||
`: # .claude/hooks/decisions-guard.sh disabled` still reads as wired, and conversely a hook invoked
|
||
through a wrapper or a constructed path reads as unwired. It catches deletion, which is the common
|
||
case; it does not catch deliberate disablement. The check does not extend to the `scripts/` half at
|
||
all.
|
||
|
||
## File populations and where they come from (ersatztv#806)
|
||
|
||
Every guard here whose members are FILES derives them from the **git index**, never a filesystem
|
||
walk. The disk is not an authoritative source: it reports build output and editor droppings and
|
||
differs per machine, so a guard derived from it asserts a different population in CI than on the
|
||
laptop of the person it is meant to stop. `scripts/tests/tracked_files.py` is the single derivation
|
||
and carries the full rationale; `scripts/tests/test_guard_populations_derive_from_git.py` proves it,
|
||
in both directions: removing EVERY member of each registered derivation from the index one at a time
|
||
and requiring it to disappear while still on disk, and watching for a directory LISTING issued while
|
||
the derivation runs (reading files stays allowed). Removal alone is blind to a source that
|
||
contributes only untracked members — an `rglob` reaching `.husky/_/` adds and never removes — and
|
||
any check phrased as "an untracked file must not enter" is itself machine-dependent, because the
|
||
untracked file has to exist. Watching for the call needs no arranged state.
|
||
|
||
That second check is a regression guard against the accidental shapes, not a boundary: what is
|
||
observed is any call that goes **through one of the spies**, whenever it happens — the spy records
|
||
into a list that outlives the patch, so a reference captured during the window and invoked after it
|
||
still counts. Whether the call goes through a spy is what decides, not when. The instance list — what
|
||
never reaches a spy at all — lives in the check's own docstring and is deliberately not restated
|
||
here, because a second copy of it drifted from the first within one commit.
|
||
|
||
One deliberate exception to "from the index" sits in the same file: the registration check lists
|
||
`scripts/tests/test_*.py` from disk on purpose, because it is a superset check over what pytest
|
||
collects — an untracked stray there makes it MORE demanding, never blind, whereas using the index
|
||
would let an unstaged new guard escape registration.
|
||
|
||
Its own limit, stated because a check described as complete stops being re-examined: it finds
|
||
derivations by PARSING each `test_*.py` for an import of the shared helper, so a module that derives
|
||
a file population some other way is invisible to it, and no mechanical check can close that (#774
|
||
reached the same conclusion about detecting filter-shaped guards by token).
|
||
|
||
The audit #806 asked for, recorded whichever way it came out, because "we looked and left it" and
|
||
"we never looked" are indistinguishable a year later:
|
||
|
||
| Guard | Population | Completeness claim over tracked files? | Outcome |
|
||
| --- | --- | --- | --- |
|
||
| `test_guard_inventory.py` | `.claude/hooks/*.sh`, `.husky/*`, `scripts/tests/test_*.py`, workflow/hook callers | **yes** — set equality against this table | converted to the index; `.husky/_/` had been excluded only because `_` is a directory, so the obvious "make it recursive" edit would have reintroduced #778's defect here |
|
||
| `test_hook_fire_log.py` | `.claude/hooks/*.sh` | **yes** — every hook must be instrumented | converted; an untracked scratch `.sh` used to demand instrumentation and redden the suite on that checkout alone |
|
||
| `test_ci_image_pin_population.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — "docker-build is the ONLY workflow pinning the toolchain image" | converted, and `*.yaml` added: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible while the test read as covering all of them |
|
||
| `test_workflow_job_guards.py` | `.gitea/workflows/*.y*ml` | **yes** — every tracked workflow's jobs must each declare a `CI_JOB_ROLE`, and the guard-declaring ones must match `## Workflow-job guards` both ways | index-derived from the start (ersatztv#786); registered in `test_guard_populations_derive_from_git.py` so the shared proof covers it |
|
||
| `test_pr_changed_files.py` | `.gitea/workflows/*.y*ml` | **yes** — "no OTHER workflow writes the review-verdict status", and since ersatztv#748 "every tracked workflow declares a `permissions:` block", with NO exemption list (the one `ci-image.yml` briefly needed was deleted when #744 landed, per #835) | converted. Not on #806's list: an untracked `.yaml` dropped in `.gitea/workflows/` reddened two guards while absent from the index — the issue's list of files to assess was a starting point, not the population |
|
||
| `test_workflow_persist_credentials.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — every `actions/checkout` in every workflow must drop the persisted credential | derived from the index from the start (#835). Shipped with NO exemption list, which is why it waited for #744: the one non-compliant checkout would otherwise have bought a permanent entry that keeps passing after its reason expires |
|
||
| `test_image_build_delegates_the_spa_suite.py` | DERIVED: tracked Dockerfiles and workflows from the git index; the stages within them that carry the SPA source. PINNED (hand-written, and the distinction is the whole design): the commands each such stage runs, and the gating `Test SPA` step's `run:` body and `if:` | **yes on the derived populations, by pin rather than by predicate** — every SPA-carrying stage must have a pin and match it; every image-publishing job whose Dockerfile has such a stage must transitively `needs:` the gating job | ersatztv#887. **THE PARSER WAS WITHDRAWN, and that is the finding worth carrying.** Three versions of this guard asked "does this command RUN the suite, and can it FAIL?" of arbitrary shell text, and that predicate was wrong NINE times across three cold-review rounds, always the same mechanism: heredoc bodies skipped as data when BuildKit EXECUTES `RUN <<EOF` (and the opener matched inside quotes, blinding the scan over the last 303 lines of `docker-build.yml` — wrong in BOTH directions at once); `shlex.shlex` not clearing `commenters` the way `shlex.split` does, so `#` truncated mid-word including the live `${#reports[@]}` idiom; compound punctuation (`);`) welding two commands; `npm t`, `./node_modules/.bin/vitest`, `pnpm vitest`, `yarn vitest`, `node …/vitest.mjs`, `timeout …`, `su -c …`, `if npm test; then` all invisible; `true || npm test` counted as the gating run while never executing; and `continue-on-error: ${{ … }}` passing a check written against two literals. Nine defects from one mechanism is `process.enumerate-workaround-behaviors-before-deleting`'s signal to remove it, so **nothing here decides what a command MEANS any more** — the risky command lines are compared as TEXT against a pin, and every one of those spellings is simply unequal to it. A pin is immune to the class that defeated the parser — a DIFFERENT SPELLING of the command — and is NOT immune to the same text meaning something else; an earlier version of this row claimed the absolute "a pin cannot produce a false green" and cold review refuted it by execution. SIXTEEN meaning-change routes have been IDENTIFIED. **This is the ONE place that count lives** — it briefly existed in three artifacts with two different values, and re-syncing three copies is how a number stays wrong (`state-the-invariant-not-the-measurement`). It is a running total rather than a proof: written as five, six, seven, eight and eleven, and wrong at every count so far. Handled: `working-directory` and the step's own `shell:`; the workflow `defaults.run.shell` and the job-level `defaults` that overrides it; a stage `SHELL`, refused; `web/package.json`'s script map, pinned WHOLE after selecting on the literal `vitest` missed `npm run test`, `npm t` and the `prebuild`/`preinstall` LIFECYCLE HOOKS npm runs for `npm run build` and `npm ci` — four one-line edits that each re-armed ersatztv#887 with every other pin matching; `web/vite.config.ts`, PINNED WHOLE — it decides what the suite COLLECTS and is where a filter would now naturally be written now that the Dockerfile's are gone; and the ABSENCE of a `vitest.config.*`/`vitest.workspace.*`/`vitest.projects.*`, which vitest resolves in PREFERENCE to `vite.config.ts` — measured: one with `include: ['nope/**']` made the gating run collect ZERO specs and exit 0, green having run nothing, which is worse than the filtered run #887 removed. Three more came from round six, all measured green before the fix: a DECOY first `test: {` block (the comparison took the first match while a later one decided), a vite PLUGIN whose `buildStart()` shells out to the suite (the plugin ARRAY is pinned now; the bodies are a residual), and a SECOND WORKFLOW publishing this Dockerfile while `needs:`-ing its own unrelated job called `test` — that one escaped every workflow-population guard in the repo, 588 tests green, and is why the `needs:` resolution is now bound to the gating WORKFLOW rather than to a bare job id. Round SEVEN found three more, two of them introduced by round six's own fixes: **`web/vite.config.js` outranks the pinned `.ts`** for `vite build` — read from vite 8.1.3's own `DEFAULT_CONFIG_FILES`, which lists `.js` and `.mjs` BEFORE `.ts` — so a plugin there ran the suite in the gitless stage with all 1411 tests green, and the rule "pinning one config is worthless while a second can outrank it" had been applied to the `vitest.config.*` family and not to this one; the decoy fix counted ONE SPELLING of the marker, so writing the real block `test: {` with two spaces defeated it; and the plugin pin added in that same commit shipped the identical decoy hole it was written next door to fix — `each-fix-round-introduces-an-adjacent-defect` inside a single commit. **Round EIGHT then defeated the repaired matcher four more ways (`test : {`, `"test": {`, and the same two for `plugins:`), plus two that never touched the marker at all: `[…].concat([evil])` after the pinned array, and a trailing `...moreTest` spread — `defineConfig` is the identity function in both vite and vitest, so a later spread REPLACES what the pin matched.** Three consecutive rounds each closing one spelling of one mechanism is `every-blocker-was-one-mechanism-so-delete-it` at the count where it says withdraw, so the partial match was withdrawn: **`web/vite.config.ts` is pinned WHOLE.** 48 lines, nothing generates it, no marker to respell and nothing after the span. Stated cost: any edit to that file reddens, a comment included. That single assertion catches all seven routes, and it deleted a bracket walk, a block extractor and two uniqueness assertions with them — including a claim, made in a commit message and in this row, that the two pins "share one bracket walk" and could not drift, which was FALSE when written: the block extractor had its own inline copy and never called the shared one. The gating job's `container:` image is pinned by `test_ci_image_pin_population.py` and CITED rather than re-asserted. What is NOT modelled, stated as a list rather than as an "only": an `ENV` in a pinned stage that changes what a pinned `RUN` resolves or executes (`PATH`, `npm_config_script_shell`); the BODIES of the pinned vite plugins — there are TWO, and only `trackedSourceFilesPlugin`'s laziness is a mitigation; `react()`'s body is third-party and unmitigated; a dependency's own install script, reached through `npm ci` and `web/package-lock.json`, which is a supply-chain concern wider than this guard; and a publish through some action other than `docker/build-push-action` — `PUBLISH_ACTION` is a SELECTOR like `SUITE_MENTIONS`, and an earlier version of this row wrongly claimed anti-vacuity covered it (anti-vacuity proves the selector is non-empty, which the real `build` job guarantees; it cannot prove completeness — measured with a `redhat-actions/buildah-build` job). **THE SHAPE ALL OF THESE SHARE, and the thing to check when adding a pin: a pin assumes it is pinning the artifact that still DECIDES.** Every route found so far is authority moving where the pin is not looking — to another FILE (`vitest.config.*` outranks `vite.config.ts`), another OCCURRENCE in the same file (a decoy first `test: {`), another WORKFLOW (a `needs:` edge naming a job called `test` that is not this one), or a HOOK the pinned command invokes (a vite plugin, an npm lifecycle script). The pin removes the spelling problem outright and reduces the meaning problem to an enumerated list. THE POPULATION/PIN SPLIT IS LOAD BEARING: a population decides what is CHECKED, so a hand-written one goes silently short; a pin decides what is EXPECTED, so a stale one goes loudly red — only the second is safe to write by hand. **A DEVELOPMENT BATTERY of 73 mutants applies cleanly with 0 missed** — and it is called that deliberately, because it is not in the repo, nothing re-derives it, and a later independent battery of 17 found 9 misses against an earlier head. It is evidence of reach at a point in time, not a coverage measurement, each caught by the assertion intended for it, including all nine spellings above; the battery is the author's plus two disjoint cold-review batteries. Twenty-eight of the 73 attack what the pin CANNOT see by construction — a change that alters what the pinned text MEANS rather than the text itself: the gating step's `working-directory` moved, `defaults.run.shell` changed from `bash` (which is `bash -e`, and is what makes a failing command fail the step) to `sh`, a `SHELL` instruction redefining every later `RUN` in a pinned stage, and a NEW stage taking the SPA source across with `COPY --from=web-build /source/web` and running the suite there. The last was MEASURED escaping the pin — a stage copy was read as not carrying the source, so the receiving stage was unpinned and unchecked. All four are now covered; the first three by pinning the workdir and the shell and refusing `SHELL` outright. GRADING, stated rather than implied: exactly ONE of the 73 is DECLARED in `mutation_manifest.py` and re-executed every suite; the other 72 were witnessed during development and are NOT standing — the same footing `pageSizeCallSites.guard.test.ts` states for its nine. RESIDUALS, corrected after cold review measured the first two of them WRONG rather than merely short: (1) a stage not carrying the SPA source is unpinned, and the boundary is what `copies_spa_source` RECOGNISES — a context copy of `web/` or `.`, a matching glob, or a stage copy whose source names `web` — not the wider "has the suite available to run", which is what this entry used to claim. What remains outside it, MEASURED after two previous attempts at this sentence each named a shape that was in fact covered: a stage copy whose source is an ANCESTOR of the tree (`COPY --from=web-build /source /src` brings `/source/web` along without naming it) or names it only as `/source/.`. The substring sweep is the backstop for those; (1b) an `ENV` altering `PATH` changes which `npm` a pinned `RUN` resolves without changing its text — `SHELL` is refused outright because it redefines the interpreter, `ENV` is not modelled; (2) the substring sweep over other steps and stages rests on `SUITE_MENTIONS`, a hand-written SELECTOR — a THIRD category beside population and pin, and the worst-behaved: a population going short is caught by an equality, a pin going stale reddens loudly, but a selector going short is SILENT. It was short by one entry (`npm t`, npm's own alias, which this guard names among the spellings that defeated the parser) until cold review measured a stage escaping through it. The sweep's reported failures are false reds; what it fails to report is not; (2b) the gating body is compared line-by-line with each line whitespace-normalised, NOT byte-exact — reflow is free, line boundaries are not, because `run: |` -> `run: >` folds two commands into one; (3) Gitea's treatment of a SKIPPED `needs:` job is not modelled — the guard forbids the job-level `if:` that would create one rather than reasoning about it; (4) the pins are hand-written and a legitimate edit reddens this guard, which is the intended review trigger and the stated cost |
|
||
| `web/src/api/pageSizeCallSites.guard.test.ts` | `import.meta.glob` over `web/src/**/*.{ts,tsx,mts,cts}`, INTERSECTED with the git index | **yes** — an unregistered discovered site fails | converted under #819. The population is the glob INTERSECTED with the git index, and the walk/index divergence is asserted in BOTH directions WITHIN a shared scope: a key the index does not carry is dropped, and a tracked, on-disk, in-scope path that never reached the scanner FAILS rather than shrinking the population silently (that direction catches a dotfile, which the glob cannot match, and a disk/index spelling divergence, which `core.ignorecase` and NFD/NFC normalisation make permanent). The SCOPE itself cancels out of that equality — both sides call one `isInScopeSourcePath` — so narrowing it moves both sets together and no assertion comparing the two sides can see it. That is closed separately, by re-deriving the predicate from its two components over the whole tracked index (`the shared scope predicate is EXACTLY its two documented components`). Two mechanisms carry that. FIRST, the scope predicate is proved by a CLOSED-FORM restatement — it may share no helper, at any depth, with the predicate it checks, since anything shared sits on both sides of the comparison and cancels. FOUR earlier attempts were each measured going green while removing real files: a table of example paths (four of `src/`'s eight directories, missed a 23-file narrowing), a decomposition delegating its filename half to `isScannableSourceFileName` (11 files), one still sharing a `basename` helper (15 files, and blind to a planted call site), and one still sharing the tracked-file array every comparison was derived from (8 files, also blind). SECOND — because that last one showed a restatement cannot police the population it reads — the plugin runs a separate `git ls-files --others` query and the guard requires every in-scope walked path to appear in `tracked ∪ others`; narrowing `tracked` cannot suppress that, since it adds nothing to `others`. SIX residuals, NOT a claim of closure, each with its direction MEASURED by planting a real `pageSize` call site rather than reasoned about. FAIL-NOISY (they redden a checkout; they cannot hide a call site): (2) a scope term matching no tracked path today survives until the day it first matches one; (4) the WIRING in `listSourceFiles`, indistinguishable on a clean checkout where the walk set and the index set agree on every in-scope key (they are not literally equal — the index also carries `.css`, which the glob never yields), so no assertion COMPARING THE TWO POPULATIONS can tell which is passed; (5) `ABSENT_FROM_DISK` emptied. BLIND — each hid a planted `pageSize: 100` with the whole suite green: (1) a COORDINATED edit of the scope predicate and BOTH closed-form restatements, which is three sites and not two, since a second restatement guards `expectedTrackedSources` (measured: the two-site edit reddens, the three-site edit does not) — tolerated because a three-site policy edit is review-visible in a way a one-line slip is not; and (3) a misdescription by the plugin's THREE outputs — not only a mispartition of `tracked`/`others`, which preserves the union any consumer compares, but a FATTENED `absentFromDisk`, which subtracts real files from the hole-detection direction. The fattening is now caught wherever it overlaps a file the walk found (a walked path is on disk by construction, so it can never be legitimately absent); restricted to paths the walk cannot see anyway — a dotfile, a case-divergent name — it stays invisible from the consuming side and is answerable only by testing the derivation directly, and the real-git tests that do so catch an UNCONDITIONAL misdescription, not one keyed on a path pattern. Neither blind residual is tolerated for being harmless. A sixth is specific to WATCH mode and is stale in BOTH directions, so it belongs to neither group: the virtual module has no backing file, so the index is read once per dev-server lifetime while the glob refreshes. A file CREATED mid-session reddens the cross-check misleadingly (noisy); a file already UNTRACKED when the watcher started keeps that classification when staged mid-session, so its call sites are never scanned — MEASURED green across both phases while `npx vitest run` on the same tree reports `UNREGISTERED`, i.e. blind, and a window that did not exist before #819 because the population was then the walk. A watch-mode green is therefore not authoritative for this guard; restarting the watcher clears the noisy case and opens the blind one, so the remedy is `npm test -- --run`. Invalidating the module from `configureServer` was implemented, measured and REJECTED: it fixes the created-mid-session red and additionally blinds the create-then-stage sequence, while the already-untracked-then-staged sequence is blind either way, since `git add` fires no watcher event in either design. The index is read by `web/vite-plugins/trackedSourceFiles.ts` in Vite's own Node context and handed to the app project as the virtual module `virtual:etv-tracked-source-files`. That is what reaches the index without admitting `@types/node` to `tsconfig.app.json` — the obstacle that deferred this, since wiring those types in was tried and reverted (it leaked Node's `setTimeout` into the app project and broke three unrelated tests). The plugin throws rather than falling back to the unfiltered walk when git fails, reports zero files, or cannot learn the Vite root. Residual, stated: a file tracked but DELETED in the working tree is subtracted deliberately, since an unstaged deletion is a normal developer state and a guard red on one gets ignored (#806); it is distinguished from the hole cases above by an on-disk existence check, not conflated with them |
|
||
| `web/src/api/completeAnnotations.guard.test.ts` | TWO derived populations: the `Complete<…>` annotations across `web/src/**/*.{ts,tsx,mts,cts}` INTERSECTED with the git index, and the droppable SCHEMAS parsed out of the generated `src/api/generated/v1.d.ts` | **partial, and the split is stated** — set equality holds for the SCHEMA population (a new optional member fails until dispositioned) and every tracked in-scope path must be supplied by the glob; there is NO closed-form restatement of the scope predicate, so a coordinated edit to it is caught by five named path pins AND a 0.95 population-ratio floor, not by a completeness proof | ersatztv#820. Turns `test_optional_request_members.py`'s `COVERED` disposition — worded "the builder is annotated `Complete<T>`", a claim about ANOTHER LANGUAGE'S source that nothing checked — into a check, and makes `docs/spa-conventions.md` §4b's prohibitions executable as asserted ABSENCES. It found one live defect: `playouts.ts` declared two request types as hand-written mirrors SHADOWING generated schemas of the same name, so their `Complete<>` was checking a local copy rather than the contract. **NARROWED after four BLOCKED review rounds, and the narrowing is the point.** It originally also derived the write WRAPPERS and required the annotation on the wrapper parameter. Every blocker across those rounds came out of that one mechanism — the obligation attached to the wrong population; reachability mistaken for protection (`Complete<T>` is shallow, so a wrapper annotation never reached a nested schema); body discovery keyed on a parameter NAME, then on parameter-versus-local; and finally an `export function` -> `export const` refactor that removed real protection while both the AST scan and its supposedly independent regex cross-check stayed blind TOGETHER, because both keyed on the same token. Five defects from one mechanism is the `process.enumerate-workaround-behaviors-before-deleting` signal to remove it rather than patch a sixth time, and ~250 lines went with it. WHAT IS NOT COVERED, listed because the earlier version of this row stated residuals in the direction that was actually covered: (1) the obligation is per-SCHEMA, not per-SITE and not per-WRAPPER — deleting the `Complete<>` from an API wrapper stays green as long as some production file still names that schema, so the wrapper half of §4b rests on review; (2) it is a TOKEN-PRESENCE check, so a dead `export type X = Complete<Y>` that nothing uses discharges the obligation as well as a live builder does — it catches deletion, which is the failure actually observed in #807, not substitution; (3) the PHANTOM direction needs a fresh literal in a contextually typed position and is not checked at all (sites-in-code, #777); (4) the test-file exclusion covers `*.test.*`, `*.spec.*` and the ONE setup file `vite.config.ts` names, whose path is pinned so a rename reddens — a SECOND `setupFiles` entry is loaded by vitest every run and WOULD discharge the obligation; that needs a three-site coordinated edit and is tolerated on the terms the sibling guard states for its own; (5) a MUST-NOT-ANNOTATE violation inside a `*.guard.test.ts` is not seen, since that file class is excluded to keep `completeRequest.guard.test.ts`'s synthetic `Complete<{…}>` fixtures out of the resolver. The disposition VALUES are cross-checked against the authoritative Python table by `scripts/tests/test_complete_annotation_dispositions.py` — without it, flipping one row from `ANNOTATED` to `CREATE` silently retired the requirement, which cold review demonstrated. Nine mutations were witnessed by hand across development, NOT re-executed per suite, so this guard claims no standing `MUTATION` grade — the same footing as `pageSizeCallSites.guard.test.ts`. (Its Python cross-check `test_complete_annotation_dispositions.py` DOES carry a declared harness-executed mutation and is graded accordingly.) A SIXTH residual, shared with its sibling: every population here derives from `trackedSources.tracked`, so a narrowing inside `web/vite-plugins/trackedSourceFiles.ts` cancels out of every comparison — this guard never reads the plugin's separate `others` query, and relies on `pageSizeCallSites.guard.test.ts` and `trackedSourceFiles.test.ts` policing the plugin. A coordinated scope edit is caught by five named path pins AND a 0.95 population-ratio floor whose denominator is computed by a different expression, not by a completeness proof |
|
||
| `test_ci_release_path_scan_job.py` | `.gitea/workflows/*.y*ml` + `scripts/**` | **no** — a fixture assembling a tmp harness, asserted about behaviour not membership | takes its file LIST from the index anyway, for hermeticity not completeness: `shutil.copytree` copied whatever was on disk, including untracked files and `scripts/__pycache__`, into a tree whose behaviour the probes then measure. Content still comes from the working tree. The copy is not a git repo, so the two files this step RUNS may not use the helper — see the fixture docstring |
|
||
| `test_ci_dropped_step_guard.py` | the parsed workflow document (its OWN subject only — the release-path `scan` job runs it in a non-git copy, so it may not derive from the index) | **no filesystem population at all** | unchanged by #806; the SCOPE residual it left open — `MARKED_JOBS` as a hand-written mirror of the required contexts on `main` — was closed by #787, which derives it from `.gitea/required-status-contexts.json` and reconciles that snapshot against the server in `scripts/check-required-contexts.sh`. The cross-workflow half of that claim lives in `test_ci_status_context_uniqueness.py`, which CAN use the index |
|
||
|
||
`test_remote_state_inventory.py` fixed its own population under #778 and kept a private copy of the
|
||
derivation; #806 folded it onto the shared one, so that module is covered by the proof above like
|
||
the rest. Across the whole change, every module that derived a file population its own way now goes
|
||
through `tracked_files.py` — one implementation of the rule instead of one per module. The
|
||
registered derivations are listed in `DERIVATIONS` in
|
||
`scripts/tests/test_guard_populations_derive_from_git.py`; this page deliberately keeps no count
|
||
of them.
|
||
|
||
**Still on filesystem walks, deliberately out of scope:** the decisions corpus
|
||
(`scripts/decisions_lib.py`'s `active_files()`, and the suites over it). Its members are `docs/`
|
||
Markdown with no generated-file pressure and a different lifecycle, and folding it in here would
|
||
have been the reflex this milestone argues against — a change with no defect behind it. It is
|
||
recorded as unexamined rather than as cleared.
|
||
|
||
## Inventory
|
||
|
||
| Guard | Blocks | Kind | Proof | Proof ref |
|
||
| --- | --- | --- | --- | --- |
|
||
| `.claude/hooks/decisions-guard.sh` | a commit | GUARD | NONE | — |
|
||
| `.claude/hooks/design-sync-reminder.sh` | the first Stop after a UI change (one-shot, then allows) | GUARD | NONE | — |
|
||
| `.claude/hooks/posttooluse-worktree-marker.sh` | nothing (writes the marker the worktree guard reads) | GUARD | MUTATION | `test_worktree_ownership_guard.py::test_MUTATION_a_marker_hook_that_stops_WRITING_makes_the_guard_go_quiet` |
|
||
| `.claude/hooks/prepush-clean-worktree-check.sh` | a push with uncommitted changes in the pushed set | GUARD | NONE | — |
|
||
| `.claude/hooks/prepush-donewhen.sh` | a direct push to `main` with unticked Done-when boxes | GUARD | NONE | — |
|
||
| `.claude/hooks/prepush-rebase-check.sh` | a push from a branch behind `origin/main` | GUARD | BEHAVIOUR-ONLY | `test_prepush_rebase_check_tag_exemption.py::test_zero_ref_lines_does_not_exempt` |
|
||
| `.claude/hooks/pretooluse-agent-model.sh` | an Agent dispatch naming no model (asks) | GUARD | NONE | — |
|
||
| `.claude/hooks/pretooluse-agent-ram.sh` | an Agent dispatch under 10% free RAM | GUARD | NONE | — |
|
||
| `.claude/hooks/pretooluse-bash-guard.sh` | a Bash call setting `ETV_UPDATE_GOLDENS` | GUARD | NONE | — |
|
||
| `.claude/hooks/pretooluse-bom-guard.sh` | a commit/push carrying a BOM in a touched `.cs` | GUARD | MUTATION | `test_bom_guard_detection.py::test_DISARMING_the_BOM_comparison_stops_detection` |
|
||
| `.claude/hooks/pretooluse-merge-consent.sh` | a PR merge without derived consent | GUARD | BEHAVIOUR-ONLY | `test_merge_consent_exemption.py::test_protected_path_on_a_LATER_page_is_still_seen` |
|
||
| `.claude/hooks/pretooluse-nav-guard.sh` | a browser navigate to a streaming URL | GUARD | NONE | — |
|
||
| `.claude/hooks/pretooluse-worktree-guard.sh` | a commit/merge in a foreign worktree | GUARD | MUTATION | `test_worktree_ownership_guard.py::test_MUTATION_disarming_the_guards_MARKER_READ_stops_the_deny` |
|
||
| `.husky/commit-msg` | a commit with no `Co-Authored-By` trailer | GUARD | NONE | — |
|
||
| `.husky/pre-commit` | a commit failing lint-staged, decisions, root-PNG or format | GUARD | NONE | — |
|
||
| `.husky/pre-push` | a push failing any pre-push hook or the SPA gate | GUARD | MUTATION | `test_prepush_unsets_git_env.py::test_MUTATION_DELETING_the_unset_lets_drift_through_silently` |
|
||
| `scripts/build_decisions_catalog.py` | the `decisions-guard` job, on a stale catalog | GUARD | MUTATION | `test_build_catalog_check_path.py::test_MUTATION_disarming_the_stale_comparison_stops_detection` |
|
||
| `scripts/check-doc-narrative.py` | nothing, by design (advisory `::warning::` only, exits 0 on every path (the error/degradation arms are defensive and unproven — see the record) — `docs.no-session-narrative` says a string predicate over prose may not be load-bearing) | TOOLING | NONE | — |
|
||
| `scripts/check-kickoff-guard.sh` | the `decisions-guard` job, on a revived #237 reference | GUARD | NONE | — |
|
||
| `scripts/check-required-contexts.sh` | the merge-consent hook, when `main`'s live required status checks no longer match `.gitea/required-status-contexts.json` (the snapshot `test_ci_dropped_step_guard.py` scopes itself to) | GUARD | MUTATION | `test_check_required_contexts.py::test_MUTATION_disarming_the_set_comparison_stops_every_drift_report` |
|
||
| `scripts/check-review-verdict.sh` | the merge-consent hook's verdict classification | GUARD | BEHAVIOUR-ONLY | `test_check_review_verdict.py::test_falseopen_token_must_be_a_whole_word` |
|
||
| `scripts/ci-detect-already-validated.sh` | nothing directly (feeds the skip gate) | GUARD | NONE | — |
|
||
| `scripts/ci-detect-docs-only.sh` | nothing directly (feeds the skip gate), but its `--depth` decides whether a COMPLETE checkout is grafted shallow — which is what stamped every `:latest` image `InformationalVersion 0.0.0-<sha>` (#836) | GUARD | MUTATION | `test_docs_only_detector_clone_depth.py::test_the_push_arm_leaves_a_COMPLETE_clone_complete` |
|
||
| `scripts/ci-peak-anon.sh` | nothing (samples container memory) | TOOLING | NONE | — |
|
||
| `scripts/ci-prove-ban-detects.sh` | the release path, if the delimiter ban is disarmed | GUARD | NONE | — |
|
||
| `scripts/ci-step-ran.sh` | the two required contexts, on a dropped step | GUARD | MUTATION | `test_ci_dropped_step_guard.py::test_dropping_ANY_single_step_FAILS_the_guard` |
|
||
| `scripts/ci-toolchain-image-resolves.sh` | the `toolchain-preflight` job, when the pinned CI toolchain image has been deleted from the registry | GUARD | MUTATION | `test_ci_toolchain_image_resolves.py::test_MUTATION_a_deleted_tag_is_reported_as_a_failure` |
|
||
| `scripts/decisions_validate.py` | the `decisions-guard` job, on a lifecycle fault | GUARD | MUTATION | `test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan` |
|
||
| `scripts/e2e-functional.sh` | the Functional E2E job, on a failed HTTP contract assertion | GUARD | NONE | — |
|
||
| `scripts/e2e-local.sh` | nothing (boots a local instance) | TOOLING | NONE | — |
|
||
| `scripts/e2e-ui.sh` | nothing (drives the Playwright flows) | TOOLING | NONE | — |
|
||
| `scripts/hook-fire-log.sh` | nothing (records that each hook fired, and reports it) | TOOLING | NONE | — |
|
||
| `scripts/jq-preflight.sh` | the `script-tests` job, on a jq version change | GUARD | BEHAVIOUR-ONLY | `test_jq_preflight.py::test_below_the_floor_is_LOUD` |
|
||
| `scripts/post-review-verdict.sh` | a verdict the gate will not inherit being reported as posted | GUARD | MUTATION | `test_post_review_verdict.py::test_a_verdict_posted_by_an_ALLOWLISTED_account_is_accepted` |
|
||
| `scripts/pr-changed-files.sh` | the verdict exemption, on an incomplete enumeration | GUARD | BEHAVIOUR-ONLY | `test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration` |
|
||
| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` |
|
||
| `scripts/update-openapi.sh` | nothing (regenerates the spec) | TOOLING | NONE | — |
|
||
| `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_build_catalog_check_path.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_check_required_contexts.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_check_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_check_doc_narrative.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_ci_dropped_step_guard.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_complete_annotation_dispositions.py` | the `script-tests` job, when the SPA guard's `DISPOSITIONS` table disagrees with `test_optional_request_members.py` about a schema both rule on | GUARD | MUTATION | `test_complete_annotation_dispositions.py::test_the_two_dispositions_AGREE` |
|
||
| `scripts/tests/test_ci_image_paths_pin_agreement.py` | the `script-tests` job, when `ci-image.yml`'s `on.push.paths` and `ci-image-pin`'s `git log` pathspec stop naming the same image sources — the live comparison is `test_the_publish_paths_and_the_pin_pathspec_name_the_same_sources`; the ref below is its mutation proof | GUARD | MUTATION | `test_ci_image_paths_pin_agreement.py::test_a_diverging_list_is_DETECTED` |
|
||
| `scripts/tests/test_ci_image_pin_population.py` | the `script-tests` job, when a container job loses its pin | GUARD | MUTATION | `test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED` |
|
||
| `scripts/tests/test_ci_release_path_scan_job.py` | the `script-tests` job, on a weakened release-path scan job | GUARD | NONE | — |
|
||
| `scripts/tests/test_ci_status_context_uniqueness.py` | the `script-tests` job, when two CI jobs synthesize the same status-check context (which branch protection cannot tell apart) | GUARD | MUTATION | `test_ci_status_context_uniqueness.py::test_no_two_jobs_synthesize_the_SAME_status_context` |
|
||
| `scripts/tests/test_ci_toolchain_image_resolves.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_decisions_lib.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_decisions_validate.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_docs_only_detector_clone_depth.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_guard_inventory.py` | the `script-tests` job, on an unclassified guard or a stale proof ref | GUARD | MUTATION | `test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist` |
|
||
| `scripts/tests/test_guard_populations_derive_from_git.py` | the `script-tests` job, on a guard whose file population admits a file git does not track | GUARD | MUTATION | `test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file` |
|
||
| `scripts/tests/test_hook_fire_log.py` | the `script-tests` job, on a hook that stops reporting that it fired, whose reporting changes what the harness sees, or that resolves the SOURCED sink from `$CLAUDE_PROJECT_DIR` instead of its own location | GUARD | MUTATION | `test_hook_fire_log.py::test_a_LATER_reassignment_the_regex_cannot_see_is_DETECTED` |
|
||
| `scripts/tests/test_image_build_delegates_the_spa_suite.py` | the `script-tests` job, on ANY change to the commands an SPA-carrying Dockerfile stage runs, on ANY change to the gating `Test SPA` step (its `run:` body, its `if:`, or a `continue-on-error` in any spelling), on the publish step losing its `docs_only` gate, on an image-publishing job that stops being gated on the job holding that step, or on any other step mentioning the suite | GUARD | MUTATION | `test_image_build_delegates_the_spa_suite.py::test_every_SPA_CARRYING_STAGE_runs_exactly_its_pinned_commands` |
|
||
| `scripts/tests/test_jq_preflight.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_mcp_smoke.py` | the `script-tests` job, when the MCP smoke checker accepts a server it should refuse — an impostor identity, a reply carrying only an id, a missing expected tool, or an `initialize` pre-answered before the request (the `tools/list` twin is held by two mechanisms jointly and is covered behaviourally, not by the declared clause) | GUARD | MUTATION | `test_mcp_smoke.py::test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE` |
|
||
| `scripts/tests/test_merge_consent_base_change.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_merge_consent_exemption.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_merge_consent_head_change.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_merge_consent_required_check.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_migration_equivalence.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names, or when a DECLARED PROSE CLAIM in `CLAIMS` stops producing the outcome its sentence asserts (either direction) | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
|
||
| `scripts/tests/test_post_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_prepush_unsets_git_env.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_pr_changed_files.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_prepush_rebase_check_tag_exemption.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_optional_request_members.py` | the `script-tests` job, on an OpenAPI request schema that can silently drop a member with no stated disposition | GUARD | MUTATION | `test_optional_request_members.py::test_every_droppable_request_schema_has_a_stated_disposition` |
|
||
| `scripts/tests/test_prove_fix.py` | the `script-tests` job | PROOF | NONE | — |
|
||
| `scripts/tests/test_review_verdict_vocabulary.py` | the `script-tests` job, when the H10 verdict vocabulary stops being shared — a word added to the one declaration that fails to reach the write side or the read side | GUARD | MUTATION | `test_review_verdict_vocabulary.py::test_a_word_added_to_the_shared_source_reaches_BOTH_sides` |
|
||
| `scripts/tests/test_remote_state_inventory.py` | the `script-tests` job, on an executable that talks to a remote service with no row in `docs/remote-state-inventory.md` | GUARD | MUTATION | `test_remote_state_inventory.py::test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file` |
|
||
| `scripts/tests/test_workflow_persist_credentials.py` | the `script-tests` job, when an `actions/checkout` step no longer sets `persist-credentials: false` | GUARD | MUTATION | `test_workflow_persist_credentials.py::test_every_actions_checkout_DROPS_the_persisted_credential` |
|
||
| `scripts/tests/test_workflow_job_guards.py` | the `script-tests` job, on a workflow JOB that is a guard with no row in the workflow-job table below, or a job with no `CI_JOB_ROLE` at all | GUARD | MUTATION | `test_workflow_job_guards.py::test_the_inventory_covers_exactly_the_guard_JOBS_that_exist` |
|
||
| `scripts/tests/test_worktree_ownership_guard.py` | the `script-tests` job | PROOF | NONE | — |
|
||
|
||
## Workflow-job guards (ersatztv#786)
|
||
|
||
The `## Inventory` table above covers guard **files**. A guard can also be a workflow **job** — its
|
||
assertion written inline in YAML, or delegated to a script that already holds a file row. Those jobs
|
||
were outside every population here until #786; `pr-checks.yml:ci-image-pin`, which stated an
|
||
invariant it did not check (#774), was sitting in exactly that gap.
|
||
|
||
**"Which jobs are guards" is a judgement the filesystem cannot supply**, so it is declared where the
|
||
job is: every job in every tracked workflow carries `env.CI_JOB_ROLE`, one of `guard`, `report-only` or
|
||
`none`. `report-only` is the third value and needs its own sentence, because "guard vs none" does not
|
||
imply it: a `report-only` job produces a **verdict it cannot enforce** — every check step carries
|
||
`continue-on-error: true`, so it reports and never fails. `docs-reminder` is the only one.
|
||
Note the limit: nothing detects a `guard` job whose checks are all `continue-on-error`, because the
|
||
checker compares the marker against the ROW, never against the job's ability to fail.
|
||
|
||
`scripts/tests/test_workflow_job_guards.py` derives the population from the parsed workflows
|
||
in the **git index** and asserts set equality against this table in both directions — a new guard job
|
||
acquires a row before the suite goes green, and a row naming a job that stopped being a guard is
|
||
reported too.
|
||
|
||
**The line is what the job PRODUCES, not whether a step can fail.** A `guard` job's output is a
|
||
**verdict** — it exists to pass or fail on some condition. A `none` job's output is an **artifact**:
|
||
an image, a set of PRs. A `none` job can still go red, but that is an ERROR in producing the thing,
|
||
not a finding about the repo. `docker-build.yml::build` publishes an image and runs a smoke test on
|
||
it, and is `none`: the smoke test failing means the build did not work, not that an invariant was
|
||
violated.
|
||
|
||
Most jobs are therefore guards, and that is the correct outcome rather than a sign the marker
|
||
distinguishes nothing: CI here is overwhelmingly checks, and a rule that excluded them would have to
|
||
draw a line no one could restate. **Which jobs are `none` is read off the markers, not restated
|
||
here** — a list in prose is a second copy of the workflow that rots on the next job added, and the
|
||
table below is the only enumeration that a check keeps honest.
|
||
|
||
**A rejected alternative, recorded so it is not re-adopted.** Drawing the line at "a guard enforces
|
||
an invariant about the REPOSITORY, so a job exercising the PRODUCT is `none`" reads as more
|
||
principled and is wrong: it puts `test` and `migrations` outside the table, and those are the two
|
||
REQUIRED status contexts on `main` — precisely the jobs where a failure to fire is fail-OPEN against
|
||
branch protection, and precisely the gap #786 exists to close. `test` also runs the C# and
|
||
TypeScript structural guards named in scope-limit item 2, so "exercises the product" was never a
|
||
clean description of it.
|
||
|
||
`Assertion` says where the logic actually lives, because that determines what a proof could even
|
||
look like: an **inline** assertion cannot be unit-tested without running the job, so most carry
|
||
`NONE` and say so, rather than borrowing credibility from a neighbouring file's proof. Two inline
|
||
rows do cite a pytest — `ci-image-pin` and `set-verdict-status` — and each names which PART of the
|
||
job it covers, because a proof reference that covers a fraction must not read as covering the job.
|
||
|
||
| Job | Blocks | Kind | Assertion | Proof |
|
||
| --- | --- | --- | --- | --- |
|
||
| `dependency-scan.yml::scan` | a weekly/dispatch run, when `dotnet list package --vulnerable` reports a vulnerable direct or transitive package | GUARD | inline — `grep -q "has the following vulnerable packages"` over the restore output | NONE |
|
||
| `docker-build.yml::toolchain-preflight` | a merge, through the COMBINED commit status the consent gate reads (#598). **Not** the other jobs: nothing `needs:` it and it is not a required context, so it reports rather than gates. It exists because a job that CONSUMED the vanished image could not run to say so (#772) | GUARD | `scripts/ci-toolchain-image-resolves.sh` | that script's row above |
|
||
| `docker-build.yml::test` | a merge — the REQUIRED context `Build ErsatzTV Image / Build & test (.NET) (pull_request)` | GUARD | the `dotnet test` / `npm test` suites, which include the C# and TypeScript structural guards named in scope-limit item 2 above | `scripts/ci-step-ran.sh` covers a dropped step — every consequential step `mark`s and the final step `assert`s, which is the mechanism `test_ci_dropped_step_guard.py` enforces on this job; the suites carry their own assertions |
|
||
| `docker-build.yml::migrations` | a merge — the REQUIRED context `Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request)` | GUARD | `dotnet ef migrations has-pending-model-changes` plus apply-to-fresh-DB, per provider | `scripts/ci-step-ran.sh` covers a dropped step — same `mark`/`assert` mechanism, enforced on this job by `test_ci_dropped_step_guard.py` |
|
||
| `docker-build.yml::functional-e2e` | a PR or a push to `main`, on a broken HTTP contract or UI flow | GUARD | `scripts/e2e-functional.sh`, which holds the HTTP contract assertions and is the only `GUARD` of the three. `scripts/e2e-ui.sh` DRIVES the Playwright specs (the UI assertions live in those specs, not in the script) and `scripts/e2e-local.sh` only BOOTS the instance they run against; both are `TOOLING` above, and naming either as the assertion would credit the coverage to a script that does not provide it | `scripts/e2e-functional.sh`'s row above (`GUARD`, `NONE`); no dropped-step guard — see the decision table below |
|
||
| `docker-build.yml::scan` | a merge and the release path, when a banned expression delimiter appears in the `run:` body of a delimiter-ban job (`DELIMITER_BAN_JOBS` = the required contexts **plus `build`**, which is wider than the marked set). It carries no `if:`, so it runs on PRs too and a red blocks through the combined status | GUARD | the ban assertion is `test_ci_dropped_step_guard.py::test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body` — note that file is graded `PROOF` above for its `ci-step-ran.sh` role, and also carries this GUARD-shaped assertion; `test_ci_release_path_scan_job.py` (`GUARD`) protects the job that runs it, and `scripts/ci-prove-ban-detects.sh` proves the ban still detects | `scripts/ci-prove-ban-detects.sh`'s row above — which itself carries `NONE`, as do the two pytest files, so this row claims no proof at all |
|
||
| `docker-build.yml::api-docs` | a PR whose committed OpenAPI spec, endpoint index or SPA client types are stale | GUARD | inline — `git diff --exit-code` after re-running `scripts/update-openapi.sh` | NONE |
|
||
| `docker-build.yml::format` | a PR whose changed `.cs` files do not conform to `.editorconfig` | GUARD | inline — `dotnet format --verify-no-changes` | NONE |
|
||
| `pr-checks.yml::ci-image-pin` | a PR whose `ersatztv-ci` pin is not the 7-char sha of the last commit to touch `docker/ci` | GUARD | inline | `test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED` covers the POPULATION half; `test_ci_image_paths_pin_agreement.py::test_a_diverging_list_is_DETECTED` covers the PATHSPEC half, bounded: it compares the pathspec this job's `git log` assignment WRITES against `ci-image.yml`'s publish trigger, and does not establish that the staleness comparison consumes that variable. Neither test runs the git-history comparison itself (that `expected` really is the last `docker/ci` commit), which still has no test |
|
||
| `pr-checks.yml::docs-reminder` | **nothing.** Both of its check steps — and the `Set up Python` between them — carry `continue-on-error: true`, so no finding it makes can fail the job. (`Checkout` does not, so the job can still go red on infrastructure — never on a finding.) | REPORT-ONLY | inline + `scripts/check-doc-narrative.py` | n/a — it cannot go red |
|
||
| `pr-checks.yml::decisions-guard` | a PR whose decision records fail lifecycle validation, whose catalog is stale, or whose kickoff file drifted | GUARD | `scripts/decisions_validate.py`, `scripts/build_decisions_catalog.py`, `scripts/check-kickoff-guard.sh` | those scripts' rows above |
|
||
| `pr-checks.yml::prove-fix` | a PR whose `Proves:` trailer names a test that passes without the fix | GUARD | inline + `scripts/prove-fix.sh` | that script's row above |
|
||
| `pr-checks.yml::script-tests` | a PR failing ruff or the `scripts/tests` suite — this job is the RUNNER for every `scripts/tests/` row above | GUARD | inline (the ruff population guards) + pytest | the suite it runs |
|
||
| `review-verdict.yml::set-verdict-status` | a MERGE, by withholding the branch-protection-required `review-verdict/h10` status | GUARD | inline | `scripts/tests/test_pr_changed_files.py`, which EXECUTES the shipped `run:` body: it guards the changed-file derivation, the exemption classification, the timeline fence and — since #849 — the post-write verification state machine, each `test_MUTATION_…` there disarming through `_run_classify(mutate=…)` the shipped clause it names, bound by a count assertion — every clause with a predecessor is mutated back to it and the rest are counterfactual, which that file's own section header enumerates, and the six clauses with no reachable fixture are enumerated in that record rather than counted. It does NOT cover the runner's step wiring, which no local test can reach |
|
||
|
||
### The four jobs with no dropped-step guard, decided per job (ersatztv#786)
|
||
|
||
`scripts/ci-step-ran.sh` turns a step the runner silently DROPPED into a red job. Its scope is
|
||
derived from `.gitea/required-status-contexts.json` (#787), so today it is enforced on `test` and
|
||
`migrations` — the two required Actions contexts, where a drop is fail-OPEN against branch
|
||
protection. `toolchain-preflight` and `scan` carry the `mark`/`assert` protocol voluntarily without
|
||
being in the enforced scope. That leaves the jobs below, which #786 listed as unexplained;
|
||
each is recorded here rather than given a marker reflexively.
|
||
|
||
| Job | Decision |
|
||
| --- | --- |
|
||
| `functional-e2e` | **No guard.** Not a required context, and it runs only on PRs and pushes to `main`. A dropped step there cannot send a required check green; it costs coverage on a run a human is already reading. |
|
||
| `api-docs` | **No guard.** Not a required context. Its single assertion is the LAST step, and a dropped `git diff --exit-code` leaves the regeneration steps' output uncommitted — visible in the job log rather than silently green. |
|
||
| `format` | **No guard.** Not a required context, and the job is three steps: a drop is the difference between a formatting red and no formatting check, on a PR whose other gates still run. |
|
||
| `build` | **Partial, and the residual is genuine.** Its delimiter class is covered — the `scan` job re-runs the ban test on the release path and is a `needs:` of `build` (#767). Per-step markers would additionally catch a drop caused by something else, and need a bucket modelling the publish-ref `if:` that the guard's always/gated buckets do not express. Stated at `.gitea/workflows/docker-build.yml`'s "Smoke + IPTV E2E" step. |
|
||
|
||
Widening `ci-step-ran.sh` beyond the required contexts is deliberately not done here:
|
||
`test_ci_dropped_step_guard.py` argues the ban is scoped to where a drop is CONSEQUENTIAL, and a
|
||
marker protocol applied everywhere costs a maintained bucket per job for jobs whose failure mode is
|
||
"a human reads a shorter log".
|
||
|
||
|
||
## The `MUTATION` column is executed, not asserted
|
||
|
||
Every row graded `MUTATION` carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`.
|
||
`scripts/tests/test_mutation_harness.py` applies each one to an isolated copy of this repository and
|
||
requires that row's own named test to go red; the manifest and the `MUTATION` rows are compared for
|
||
set equality in both directions, so the grade and the mutation cannot drift apart. Adding a row
|
||
graded `MUTATION` without declaring its clause fails the suite. Full contract and its limits:
|
||
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
|
||
|
||
**That file also executes a SECOND population, which is not this table and must not be read off it.**
|
||
`CLAIMS` in the same manifest binds declared PROSE claims — a sentence in a code comment, a test
|
||
docstring or a decision record asserting that some mutation reddens, or does not redden, a named
|
||
test — to the file and the sentence that make them (ersatztv#881). Those entries carry a `site` and
|
||
a verbatim `quote` instead of a guard, so a rewritten sentence reports as a retarget; and they may
|
||
declare a GREEN outcome, which no row here can — one that also declares a REACH mutation of the same
|
||
clause required to redden the same proof, because a green from a proof that never touches the mutated
|
||
file is indistinguishable from the green the sentence claims. A guard's row says nothing about
|
||
whether a claim about that guard has been declared, and the two populations are derived differently
|
||
on purpose.
|
||
|
||
One thing that column still does not say: it does not say the declared clause is the ONLY thing the
|
||
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
|
||
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
|
||
behaviour, which catches the recorded proof going stale but not much else.
|
||
|
||
**No row is graded `DETECTOR` (measured 2026-08-30, and asserted every run — see below), and the
|
||
one that was is the worked example of the grade doing its job.** `instrumentation_faults` held it: the arms a stripped hook trips several of at once meant
|
||
no single-arm disarm reddened its proof, so the entry carried the finest mutation that survived
|
||
(`if not _SOURCES_SINK.search(text):` -> `if False:`) as data, re-run every suite and required to
|
||
KEEP surviving. ersatztv#891 gave the guard byte-identity of the hook preamble, which decides alone —
|
||
so the entry was regraded to `CLAUSE` and the survivor dropped. NOTE THE DIRECTION, because it is
|
||
not the one the contract anticipates: the contract regrades when a survivor STARTS REDDENING, and
|
||
this survivor could no longer redden AT ALL — byte-identity necessarily matches the line it tested.
|
||
A mutation required to keep surviving that is guaranteed to survive is a tautology, not a canary, so
|
||
it had to go rather than be carried on. Because the set is now empty, TWO things are unexercised:
|
||
the survivor runner `test_every_SURVIVING_clause_mutation_still_does`, and the `unevidenced` arm of
|
||
`test_every_entry_declares_a_known_granularity_and_DETECTOR_entries_CARRY_their_survivor`. Both
|
||
self-arm the moment an entry is graded `DETECTOR`. `verify_mutation`'s "still PASSED" verdict path is
|
||
NOT one of them, though it looks like it belongs: it is not `DETECTOR`-gated, and
|
||
`test_an_INERT_mutation_is_REPORTED_rather_than_passed` drives it every run. Recorded here rather
|
||
than implied to be covered, and asserted by `test_the_DETECTOR_survivor_set_is_empty_ON_PURPOSE` so
|
||
the empty set stays a decision rather than an unread skip.
|
||
|
||
## What the numbers say
|
||
|
||
47 guards, 6 tooling scripts, 23 proof files. **27 guards carry a mutation proof; 5 are
|
||
behaviour-only; 15 have none.** These figures are asserted against the table by
|
||
`test_the_summary_counts_match_the_table`, because a hand-maintained summary of a table is a second
|
||
copy of it — the duplication family this change argues against, and one that has already drifted
|
||
here.
|
||
|
||
**One guard was WITHDRAWN rather than shipped**, and it is worth a line here because the inventory
|
||
is where a future session will look for it. A `test_review_verdict_vocabulary_parity.py` asserted
|
||
set equality between the H10 verdict vocabulary's write side and read side by extracting both from
|
||
shell source with regexes. Six cold-review rounds each found another shell construction that either
|
||
escaped it or made it red on a correct tree. It was deleted rather than patched a seventh time: the
|
||
`testing.guard-derives-population-from-source` record says a weak detector is itself the
|
||
symptom-keyed mistake, and this was the empirical demonstration. The underlying duplication is real
|
||
and is now UNMITIGATED — tracked in #788, which fixes it by removing the second copy.
|
||
|
||
**What that file asserted, and where each part went** — enumerated rather than summarised, because a
|
||
summarised removal silently drops an invariant and claims otherwise, which is
|
||
`process.enumerate-workaround-behaviors-before-deleting` failing on its own removal:
|
||
|
||
| Invariant it asserted | Where it is now |
|
||
|---|---|
|
||
| Bidirectional write/read vocabulary parity | **LOST.** The reason for the withdrawal; #788 |
|
||
| No READ-side word is both positive and negative | **LOST** as a universal property; #788 |
|
||
| No WRITE-side word is both success and failure | **LOST** as a universal property; #788 |
|
||
| Strict extractor consumes every case arm | Retired with the extractor; #788 |
|
||
| `POS_RE`/`NEG_RE` each assigned exactly once | Retired with the extractor; #788 |
|
||
| Both extractors find non-empty sets (anti-vacuity) | Retired with the extractor; #788 |
|
||
| A wrong write source makes the extractors disagree | Retired with the extractor; #788 |
|
||
|
||
Two guards were ADDED in its place. They are **not** replacements for rows 2 and 3 and the table
|
||
above deliberately does not list them as such: relabelling a lost invariant as a narrower surviving
|
||
one is the exact failure this table exists to prevent — and it would be worst committed by the
|
||
table itself:
|
||
|
||
| Added | What it pins |
|
||
|---|---|
|
||
| `test_check_review_verdict.py::test_each_verdict_word_retains_its_established_polarity` | the five established tokens still READ with their established polarity |
|
||
| `test_post_review_verdict.py::test_each_verdict_word_posts_its_established_polarity` | the five established tokens still POST with their established polarity |
|
||
|
||
**Polarity is strictly weaker than disjointness, and the gap is demonstrable.** Put `MERGEABLE` in
|
||
BOTH `case` arms of `post-review-verdict.sh`: the success arm wins, every polarity assertion stays
|
||
green, and the withdrawn disjointness test failed. What the two added guards do catch is the
|
||
dangerous direction — a token that a reviewer means as `BLOCKED` silently reading or posting as
|
||
approval, which writes a green `review-verdict/h10`. A token added to one script and not the other
|
||
is untested by either, which is the whole of #788.
|
||
|
||
**A guard was WITHDRAWN from `test_hook_fire_log.py` after four iterations**, recorded here because
|
||
this is where a future session will look for it. A `test_no_redirection_in_the_sink_PRECEDES_its_stderr_redirect`
|
||
generalised the `exec … 2>/dev/null` rule to any command by pattern-matching shell source. Each
|
||
iteration fixed a false positive or false negative the previous one introduced — `>/dev/null 2>&1`
|
||
flagged wrongly, then operators preceded by a digit (`exec 0<`) missed entirely, then multi-command
|
||
lines flagged wrongly — and cold review then constructed more of both (`[[ "$x" < "$y" ]] 2>/dev/null`,
|
||
a backslash continuation, a `>` inside a quoted string). Deleted rather than patched a fifth time,
|
||
on the same reasoning as the vocabulary-parity withdrawal above.
|
||
|
||
**`test_hook_fire_log.py` proves more clauses than its row claims, and this file no longer tries to
|
||
enumerate them.** Every revision of the hand-written list was wrong in a new way: a count that
|
||
drifted from the proofs, an arm credited with proofs it does not exercise, an unclaimed list missing
|
||
members, and a scope sentence that did not partition the file. Successive review rounds fixed one and
|
||
introduced the next — the paragraph written to REPLACE the list then misattributed a machine check
|
||
and re-committed the scope error it had itself just recorded. Same shape as the two withdrawals
|
||
above, withdrawn on the same reasoning: this file now says deliberately LESS about that guard than it
|
||
could, and what remains is only what nothing beside the code already says.
|
||
|
||
Read the artifacts instead. The row's `Proof ref` and `scripts/tests/mutation_manifest.py` are held
|
||
against each other and EXECUTED by `test_mutation_harness.py`, so a stale ref there is a red rather
|
||
than a reading error. The arms are explained in `scripts/tests/test_hook_fire_log.py` itself, in the
|
||
comments through `instrumentation_faults`; the list partway down that function enumerates the arms
|
||
byte-identity does NOT subsume.
|
||
|
||
One thing no artifact records, so it is written here: **the `Blocks` column is not machine-checked at
|
||
all.** `inventory_rows()` parses that cell into a discard variable and nothing else reads it, so a
|
||
reader is the only check. Read it as the DEFECT MECHANISM, not the symptom: a self-disabled hook and
|
||
a hook reporting under another hook's name both end in "stops reporting that it fired", but neither
|
||
is the mechanism that column names (losing the preamble), so neither is claimed by it.
|
||
|
||
**The SELF-DISABLEMENT arm is narrow, and that is recorded rather than implied.** It matches
|
||
`^\s*(export\s+)?ETV_HOOK_FIRE_DISABLE=1`, so it sees the bare and `export`ed forms. Measured
|
||
2026-08-30, four other spellings disable reporting and pass clean: `ETV_HOOK_FIRE_DISABLE="1"`,
|
||
`…='1'`, `: ${ETV_HOOK_FIRE_DISABLE:=1}` and `readonly ETV_HOOK_FIRE_DISABLE=1`. No hook uses any of
|
||
them — no tracked hook mentions `ETV_HOOK_FIRE_DISABLE` at all. It is left narrow deliberately, but
|
||
NOT because widening is impossible: the whole-artifact pin already shipped by ROOT — the clause that
|
||
a hook must source the sink from its OWN tree (ersatztv#891) — transfers verbatim: collect
|
||
the non-comment lines mentioning the variable and require the list EMPTY, which refuses all four
|
||
spellings and anything not yet imagined. That is a whole-artifact pin, not a fifth alternation, and
|
||
it is the shape ersatztv#891 withdrew a lexical predicate in favour of. It is not done here only
|
||
because ersatztv#891 was already large and the hole is recorded; whoever needs the net should add
|
||
that check rather than another `|`-arm.
|
||
|
||
ROOT carries a NEGATIVE CONTROL because its assertion is an ABSENCE — the decoy's marker must not
|
||
appear — which is also exactly what an inert fixture produces. (That is the distinguishing property,
|
||
not "the only clause with a control": transparency's `test_the_stripper_actually_strips` and
|
||
coverage's `test_the_population_is_not_empty` are anti-vacuity arguments of the same family.) The
|
||
control runs the same decoy with the hook copied into it — making the decoy genuinely `$repo_root` —
|
||
and requires the marker to appear.
|
||
|
||
ROOT is also proved in BOTH LAUNCH CONSTRUCTS, because they resolve differently and only one of them
|
||
is reachable. The Claude harness starts a hook by ABSOLUTE path, where `$CLAUDE_PROJECT_DIR` names
|
||
the same tree by construction; husky starts `./.claude/hooks/…` by a path RELATIVE to the pushed
|
||
tree, so self-location resolves through the CWD and the two roots can genuinely disagree. The husky
|
||
half (`test_the_HUSKY_RELATIVE_launch_resolves_to_the_pushed_tree`, with its own control) derives
|
||
its population from the `.husky/*` wiring and is guarded against an empty one by
|
||
`test_the_husky_launched_population_is_not_empty` — a `@parametrize` over nothing does not fail, it
|
||
collects one SKIPPED placeholder that is indistinguishable from coverage in a full run.
|
||
|
||
**`test_the_suite_does_not_write_to_the_PRODUCTION_log` is a structural check, and deliberately
|
||
reads nothing under `$HOME`.** It asserts that the environment a module sees at IMPORT time already
|
||
resolves away from `$HOME/.cache/ersatztv/hook-fire/`, and that a hook driven with that snapshot
|
||
lands its records in the isolated directory instead. `scripts/tests/conftest.py` supplies the
|
||
property in two layers, and they fail differently: `pytest_configure` sets `ETV_HOOK_FIRE_LOG_DIR`
|
||
before collection, which is what reaches import-time snapshots and the module- and session-scoped
|
||
fixtures no autouse function fixture can cover; and a `Popen` wrapper fails any launch that does
|
||
not CARRY the isolated directory — the route left when an environment is built from scratch rather
|
||
than derived from `os.environ`. That rule is deliberately "carries an isolated dir" and NOT "is not
|
||
the production one": the sink's default has two branches (`$HOME/.cache/...`, and `/tmp/.cache/...`
|
||
when `HOME` is unset or empty), so a single-production-path comparison models one and waves the
|
||
other through — of the 83 launches here whose environment is built from scratch, 81 carry no `HOME`
|
||
either and so land in the branch such a comparison cannot see. A from-scratch environment must
|
||
therefore carry the variable. No count of those sites is kept here — it is a hand-written population
|
||
nothing derives, and the commit introducing the rule already falsified its own by adding sites in
|
||
the tests that prove it. The guard enumerates them: a missing one is a red, not a stale sentence.
|
||
Proofs: `test_REMOVING_the_pre_collection_isolation_LETS_an_IMPORT_TIME_snapshot_LEAK` runs both
|
||
arms as nested pytest sessions under a fake `HOME`, and
|
||
`test_a_launch_NOT_CARRYING_the_isolated_log_dir_FAILS_THE_LAUNCH` drives the wrapper over both
|
||
branches, an explicitly-wrong value, and a RELATIVE value made to land on a shared log by the
|
||
child's `cwd` — that last one is also why the guard anchors a
|
||
relative path at the launch's `cwd=` rather than at its own — each with a negative control.
|
||
`test_the_resolver_agrees_with_the_SINKS_OWN_expansion` runs the Python
|
||
resolver and `etv_hook_fire_log_dir` over the same environments every run, because a resolver that
|
||
disagrees with the sink fails OPEN — it clears a launch the sink then points at the real log.
|
||
|
||
**Two earlier shapes are recorded because both read as correct and neither is** (ersatztv#809,
|
||
ersatztv#822). The guard used to snapshot `st_mtime_ns` across the real log directory and require it
|
||
unchanged; that oracle is state every concurrent Claude Code session on the machine writes, so an
|
||
unrelated session firing a hook inside the window failed the suite with `a test run modified the
|
||
production hook-fire log` — an accusation about the suite when the writer was another process.
|
||
Observed on three separate branches, green on every immediate re-run, and corroborated by a reviewer
|
||
who found a file there written 13 seconds earlier by someone else. A flake that misattributes is
|
||
worse than one that just fails. And an autouse fixture alone isolates too late: it runs at test
|
||
setup, after every module in the session has been imported. Measured on the pre-change tree by
|
||
instrumenting `subprocess.Popen` for a full run and asking which launches resolve to
|
||
`$HOME/.cache/ersatztv/hook-fire`: 83, of which 37 during collection and 44 inside module- or
|
||
session-scoped fixtures — both outside any test, and so beyond an autouse function fixture's reach.
|
||
The other 2 came from an environment built from scratch with the real `HOME`, the route the second
|
||
layer exists for. Instrument `Popen` ALONE: `subprocess.run`, `call` and `check_output` all reach
|
||
it, so wrapping `run` as well counts every launch twice and doubles every figure. That predicate
|
||
also UNDERCOUNTS the from-scratch route, because such an environment
|
||
usually carries no `HOME` either and so resolves to the sink's other default, which it cannot see.
|
||
Both agree on the finding that matters: almost none of it was the module-level-snapshot route
|
||
ersatztv#809 names.
|
||
|
||
The reproduction is the part worth keeping: restore a module-level `{**os.environ}` snapshot — or
|
||
leave the helper correct and point a single `env=` argument at one — then run that file and count
|
||
records for its synthetic session ids. **58 per run**, measured identically on macOS/git 2.55 and
|
||
Linux/git 2.47.3. (The accumulated total observed before the fix ran to four figures across many
|
||
runs; that is an observation rather than a reproducible measurement, so check the per-run figure.)
|
||
Since the pre-collection isolation those 58 land in the session's own temporary directory, so the
|
||
figure now measures this file's hook traffic rather than a leak — the reproduction still shows the
|
||
mistake, not the consequence it used to have.
|
||
|
||
The pin is
|
||
`test_worktree_ownership_guard.py::test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir`, which
|
||
asserts the EFFECT — records land in the fixture's dir — rather than the shape of the fix. Its
|
||
predecessor asserted the helper's return value, which leaves the call site unguarded: `_env()`
|
||
correct, one `env=` reverted, all tests green, records still leaking. Pinning to a hand-written
|
||
revert rather than to the property is `verify-against-the-REAL-predecessor`.
|
||
|
||
It is narrower than the property: it guards the launch path it drives. A second launcher in the
|
||
same file that passed a stale environment would leave it green — measured, 18 records to the wrong
|
||
directory — because the
|
||
hooks it drives would still log correctly. That is why it is a pin on one file and not the general
|
||
property; the general property is the two conftest layers above, which do not depend on any suite
|
||
remembering to route through a helper.
|
||
|
||
**That unproven set carries no number here on purpose.** It restated the count, drifted the moment
|
||
the BOM guard was regraded, and `test_the_summary_counts_match_the_table` cannot see it — the parser
|
||
checks the formatted summary sentence and nothing else. A second hand-maintained copy of a number is
|
||
the duplication family this file argues against, so the copy is removed rather than corrected: the
|
||
set is the eight `.claude/hooks/` guards enumerated in the standing list below, and the count is in
|
||
the summary above. Naming them by event is what made the earlier wording wrong twice — they do not
|
||
share one event, and `design-sync-reminder.sh` is registered on **both** `PreToolUse` and `Stop` in
|
||
`.claude/settings.json`, so any "every X hook" phrasing double-counts it.
|
||
|
||
**They are now observable but still unproven, and the two words carry different weight.** Observable:
|
||
every hook records its own execution through `scripts/hook-fire-log.sh`, so "did this hook fire, and
|
||
what did it decide" is a measurement — run `scripts/hook-fire-log.sh report` (#776). Unproven:
|
||
nobody has demonstrated any of them is load-bearing — the first group in the standing list below.
|
||
Observability tells you a guard ran; only a mutation tells you it would have caught anything. The
|
||
BOM guard is the case that shows why the distinction matters — it was firing on every commit the
|
||
whole time it was fail-open.
|
||
|
||
The gaps are not uniform in cost, and the ranking that matters is *what a silent failure would let
|
||
through*, not test count. **All four ranked entries now carry clause-level mutation proofs (#785);
|
||
they are kept here with what each mutation established, because the ranking is the reusable part and
|
||
because two of them turned out to be worse than the ranking predicted.**
|
||
|
||
1. ~~`pretooluse-bom-guard.sh`~~ — **proven, and it was fail-open the whole time.** Ranked first
|
||
because the defect it guards has recurred three times (#311, #402, #405); that ranking turned out
|
||
to be right for a worse reason than intended. It detected a BOM with `xxd -p`, and `xxd` ships
|
||
with vim and is **absent on the Linux CI runner**, so the comparison never matched and every BOM
|
||
was allowed in silence. `od` now. The lesson for the rows below: an unproven guard is not merely
|
||
untested, it is a guard whose *current* behaviour nobody has established.
|
||
2. ~~`pretooluse-worktree-guard.sh` + `posttooluse-worktree-marker.sh`~~ — **proven as a pair.**
|
||
Four clauses were disarmed and witnessed red: the guard's marker read; the guard's ownership
|
||
comparison (inverted, because disarming it the other way only makes the guard deny more and every
|
||
deny assertion stays green); the `commit|merge` alternation, whose `merge` half every other case
|
||
in the file left untested while guarding the plumbing-merge path; and — the one that could not
|
||
exist while the halves were tested apart — the *marker hook's write*, asserted against the
|
||
*guard's* decision. Both hooks are
|
||
deliberately fail-open, so an absent mechanism and a working one produce the identical "commit
|
||
allowed".
|
||
3. ~~`.husky/pre-push:11`'s `unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE`~~ — **proven, and the case
|
||
is the normal one rather than an edge.** Git exports `GIT_DIR` to `pre-push` **when the push
|
||
comes from a worktree** and not from the main tree; `process.shared-tree-readonly` makes the
|
||
worktree the mandated way to work here, so every push takes the exposed path. With `GIT_DIR` set
|
||
and `GIT_WORK_TREE` unset git stops discovering the repo and treats the *current directory* as
|
||
the work tree, so `cd web && npm run check:api`'s `git diff --exit-code` compares against index
|
||
paths that do not exist and reports no diff. Both the deletion and the relocation are proved.
|
||
4. ~~`scripts/build_decisions_catalog.py`~~ — **proven, including the wiring.** The `--check`
|
||
comparison is mutated directly, and a separate subprocess case runs the command *derived from
|
||
`pr-checks.yml`* against a copied corpus. That second case is not redundant: replacing
|
||
`raise SystemExit(main())` with a bare `main()` leaves the script printing
|
||
`docs/decisions/README.md is stale` on stderr while exiting **0**, and the workflow step reads
|
||
nothing but the exit code. Only the subprocess case reddens — the #751/#719 shape.
|
||
|
||
**What a file-level `MUTATION` grade does and does not claim, because three of these four rows are
|
||
multi-purpose files.** The grade covers *the clause the cited case actually mutates*, per
|
||
`testing.guard-ships-with-mutation-proof`; it is not a statement about every line in the file.
|
||
`.husky/pre-push` runs four other things, and its row asserts only that line 11 is load-bearing —
|
||
the three hooks it invokes carry their own rows and their own grades. Reading the row as "pre-push
|
||
is tested" is the same relabelling error the withdrawn-guard table above exists to prevent.
|
||
Clause-level grading is tracked in #790.
|
||
|
||
**The remaining unproven guards, and why each is still `NONE` rather than merely unattended** — the
|
||
third `## Done-when` box of #785, answered here rather than left implicit. They fall into three
|
||
groups, and the split is the point: the first two are backlogs, the third is not.
|
||
|
||
- **The eight remaining hook guards** — genuinely unproven, and a real backlog. Four are
|
||
`PreToolUse` (`pretooluse-agent-model.sh`, `pretooluse-agent-ram.sh`, `pretooluse-bash-guard.sh`,
|
||
`pretooluse-nav-guard.sh`), two are pre-push (`prepush-clean-worktree-check.sh`,
|
||
`prepush-donewhen.sh`), one is pre-commit (`decisions-guard.sh`) and one is registered on both
|
||
`PreToolUse` and `Stop` (`design-sync-reminder.sh`); they are grouped by their status, not by
|
||
their event, because the events do not partition them. They are now
|
||
*observable* (`scripts/hook-fire-log.sh report`, #776), which is a weaker claim than proven and is
|
||
stated as such above.
|
||
- **`.husky/commit-msg` and `.husky/pre-commit`** — unproven, and each carries its own clause, so
|
||
they are a backlog too rather than "covered by what they call". `pre-commit` dispatches to
|
||
`lint-staged` and `decisions-guard.sh`, but the root-level-`*.png` refusal and the
|
||
`dotnet format whitespace --verify-no-changes` block are its own. `commit-msg` is not a dispatcher
|
||
at all: it is one `grep -q '^Co-Authored-By:'` plus a `MERGE_HEAD` exemption, and nothing tests
|
||
that the exemption fires only for merges. Neither is covered by what it calls — the trap is to read
|
||
"it dispatches to guards" as "its own clauses are guarded".
|
||
- **`ci-detect-already-validated.sh`, `ci-prove-ban-detects.sh`, `e2e-functional.sh`,
|
||
`check-kickoff-guard.sh`, `test_ci_release_path_scan_job.py`** — not
|
||
unattended either, but each needs its own harness rather than a proof written to clear a row.
|
||
`ci-detect-docs-only.sh` left this list in ersatztv#836 because a shipped defect gave it a real
|
||
red to reproduce, rather than because someone wrote a proof to fill the column. Its effect is
|
||
observable in-process after all — the graft it performs is a flag on a real clone — so a
|
||
skip-gate script is not automatically unprovable here.
|
||
`ci-prove-ban-detects.sh` is the exception noted in `testing.guard-ships-with-mutation-proof`: it
|
||
runs its own mutation at CI time, because what it proves is disarmable from inside pytest.
|
||
|
||
That list is prose and nothing checks it, which is the honest limit — a guard moving out of a group
|
||
will not redden anything. It is here so the next session inherits *why* a row is `NONE`, which is
|
||
the distinction #785 asked for; the machine-checked half remains the table.
|
||
|
||
Filling the rest is tracked rather than done in one pass, deliberately: a mutation proof written to
|
||
close a row is the kind of test that passes for the wrong reason.
|