Compare commits

...
Author SHA1 Message Date
timothyandClaude Opus 5 ded6cb197c docs(806): record the SPA page-size guard as assessed-and-deferred (#819)
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 7s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 18s
PR Gates / Docs update reminder (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 21s
PR Gates / decisions lifecycle (pull_request) Successful in 16s
Review verdict / Set review-verdict status (pull_request_target) Successful in 33s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m37s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m38s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 5m56s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m2s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
review-verdict/h10 Review-verdict: MERGEABLE @ ded6cb1 (base: main)
The whole-repo sweep in cold review — looking past the file list #806 names, the
same method that found `test_pr_changed_files.py` inside this PR — turned up one
more instance of the shape: `web/src/api/pageSizeCallSites.guard.test.ts` derives
its population from `import.meta.glob('/src/**/*.{ts,tsx,mts,cts}')` and then
asserts exact completeness against `REGISTRY`. An untracked `.ts` under `web/src/`
containing a page-size call enters the population and fails as unregistered on that
checkout while CI never sees it.

NOT FIXED HERE, and the reason is a documented obstacle rather than scope fatigue.
The glob is a deliberate workaround: `@types/node` is kept out of
`tsconfig.app.json` because that project covers production browser code, and a
file-local `/// <reference types="node" />` was tried and REVERTED — under `tsc -b`
it leaked Node's ambient `setTimeout` into the whole app project and broke three
unrelated `window.setTimeout` mocks. Filtering the glob against `git ls-files`
therefore needs `node:child_process` in a project that excludes Node types, which is
the wall the previous attempt hit. #819 carries the evidence and two fix shapes that
sidestep it (a Vite `define` or a vitest `globalSetup`, both of which run where Node
IS available).

The direction is the safe one — it over-enumerates, so it fails loudly rather than
going blind — which is why it is a follow-up rather than a blocker.

Recorded in the audit table as DEFERRED rather than left silent, which is what #806
asks for every population it assesses: "completeness claim over tracked files, or
not — recorded either way". A guard nobody wrote down is indistinguishable from one
nobody looked at.

refs #806
refs #819

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:44:07 +02:00
timothyandClaude Opus 5 908855644a docs(806): a duplicated article, in the sentence the previous commit rewrote
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 9s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 19s
PR Gates / Docs update reminder (pull_request) Successful in 25s
PR Gates / decisions lifecycle (pull_request) Successful in 28s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 20s
review-verdict/h10 Awaiting review verdict for 9088556
Review verdict / Set review-verdict status (pull_request_target) Successful in 19s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m42s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m14s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 5m58s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m0s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
`docs/guard-inventory.md` shipped "one per module. The The registered derivations
are listed…". The rewrite replaced the line that began "The registered
derivations…" without removing the trailing `The` the previous line carried as that
sentence's wrapped start. Markdown soft-breaks the two lines, so the published page
showed both.

Worth a line rather than a silent fix: the evidence was on screen. The `sed` output
I used to verify that rewrite printed line 140's trailing `The` directly above line
141's `The registered` and I read past it — a verification that ran, produced the
right output, and was not actually read. Cold review caught it.

Swept for the same shape across the three prose artifacts this change touches, over
JOINED lines so a duplication split across a wrap is visible: no others.

One clarification to the previous message, which enumerated what the count sweep
returns and reads as exhaustive: that enumeration covers the prose this branch
authors. `guard-derives-population-from-source.md` also carries "the 95 closed
issues carrying a `## Closing record`" and "27 of the 69 process-failure records"
— counts over growing sets, but pre-existing (#774/#775) and attributed citations
of #773's analysis rather than live tallies, so they are left alone.

refs #806

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:22:40 +02:00
timothyandClaude Opus 5 a92df7515a fix(806): finish the count sweep the previous message claimed, and name the pointer's file
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 9s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 18s
PR Gates / decisions lifecycle (pull_request) Successful in 15s
PR Gates / Docs update reminder (pull_request) Successful in 14s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 13s
review-verdict/h10 Awaiting review verdict for a92df75
Review verdict / Set review-verdict status (pull_request_target) Successful in 23s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m45s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m29s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m3s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 2m2s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Cold review returned MERGEABLE with two notes; both are corrections to claims this
change ships, so both are taken.

THE PREVIOUS COMMIT CLAIMED A REGEX OVER EVERY COUNT IN ALL FIVE ARTIFACTS and left
four. They were dated and true, which is why they read as acceptable, but a single
added `MUTATION` row stales all four at once — the same fuse as the counts already
removed, and the claim of a complete sweep is what made keeping them dishonest
rather than merely conservative:

  * "three of the fourteen entries" → "three entries" (it names them; the total
    contributes nothing)
  * "the remaining eleven redden behaviourally" → "every other entry reddens
    behaviourally"
  * "thirteen of the fourteen admit clause-level proof" → "all but one"
  * `guard-inventory.md`'s "three of the fourteen rows" → "three rows"

An independent sweep with a wider pattern now returns only counts over FIXED
structures — #790's four ranked rows, `derived_guard_files`'s four union terms,
#685's two guards — plus the machine-asserted `36 guards … 14/6/16`, and the
member figures, which are dated, carry their reproduction command, and say in place
that they illustrate the gap above the floor rather than being values to assert
against. That is the distinction worth keeping: a number describing a fixed shape,
or one that ships with the command that regenerates it, does not rot the way a
tally of a growing set does.

`docs/guard-inventory.md`'s new "listed in `DERIVATIONS` there" had the wrong
nearest antecedent — `DERIVATIONS` lives in
`scripts/tests/test_guard_populations_derive_from_git.py`, not in
`tracked_files.py`. Named explicitly.

refs #806

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:05:18 +02:00
timothyandClaude Opus 5 10dba0892e fix(806): make the drifting counts count-free instead of correcting them again
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 19s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 20s
PR Gates / decisions lifecycle (pull_request) Successful in 22s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 15s
PR Gates / Docs update reminder (pull_request) Successful in 31s
review-verdict/h10 Awaiting review verdict for 10dba08
Review verdict / Set review-verdict status (pull_request_target) Successful in 39s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 4m38s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m32s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 5m58s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 5m52s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 1s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Cold review of the previous commit found that IT had made
`docs/guard-inventory.md`'s "load-bearing for the six modules that import it"
false, by adding a seventh importer — in the file it was editing, inside the sweep
its own message claims to have done by subject. That is the third hand-maintained
count in this change to go stale, after `mutation-claims-are-executed.md`'s three
and `guard-inventory.md`'s "three of the thirteen".

Correcting a number that tracks a growing population buys one commit of accuracy.
So the counts that track populations are gone rather than updated:

  * "load-bearing for the six modules that import it" → "for every module that
    imports it"
  * "this file is not four near-copies" → "not one near-copy per derivation"
  * "adding a fifth derivation / a fifth index-derived guard" (twice) → "another"
  * "five guard modules would otherwise die" → "every module that imports this one"
  * "Exhaustive removal is cheap (~1s for 60 members)" → "about a second at the
    population sizes here"

The sweep was then done MECHANICALLY — a regex over every spelled-out and numeric
count in all five artifacts this change touches — rather than by eye for a fourth
time. What survives is verified true rather than assumed: "three of the fourteen"
and "36 guards … 14/6/16" (the latter machine-asserted by
`test_the_summary_counts_match_the_table`), "unions four contributors",
"five modules each derived a file population" (`DERIVATIONS` holds exactly five),
and the re-measured 61/39/48/58. `guard-inventory.md`'s "three of these four rows"
is #790's text about a different row set and is untouched.

The figures that remain are the ones that describe a FIXED structure or carry a
reproduction command; the ones that counted a set expected to grow are the ones
that kept breaking, which is the distinction the sweep now encodes.

Correction to the previous commit message: it called the second converted call site
"the delimiter ban over every workflow". That guard is
`test_every_workflow_expression_names_a_REAL_context_or_function`, an
expression-context check; the delimiter ban does not use `_workflow_files()`. Both
call sites are completeness claims over every workflow and both were demonstrated
red against an untracked `.yaml` at the predecessor sha — only the label was wrong.

refs #806

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 16:44:26 +02:00
timothyandClaude Opus 5 03955675a8 fix(806): assess test_pr_changed_files.py too, and correct three counts this change invalidated
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 12s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 27s
review-verdict/h10 Awaiting review verdict for 0395567
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m51s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m21s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 5m58s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Canceled after 5m47s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Canceled after 0s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Canceled after 0s
Two findings from the cold review of the pushed head, one of them the more
interesting kind: a defect in a file the issue did not list.

`test_pr_changed_files.py` DERIVED A WORKFLOW POPULATION FROM `Path.glob`, and it
carries two completeness claims over it — "no OTHER workflow writes the
review-verdict status", and the delimiter ban over every workflow. Cold review
demonstrated it in a disposable clone: one untracked `.yaml` dropped in
`.gitea/workflows/` reddens both guards while absent from `git ls-files`. That is
the developer-red/CI-green shape #806 exists to remove, in a guard that was never
assessed.

It was never assessed because #806's body names four files to check and I took that
list as the population — which is this issue's own defect, one altitude up, in the
work implementing it. My first sweep printed this file among the filesystem-derived
populations and I narrowed to the issue's list without saying so. Both call sites
now come from the index, `_workflow_files` is registered in `DERIVATIONS` so both
proofs cover it, and the audit table in `docs/guard-inventory.md` says plainly that
the issue's list was a starting point rather than the population.

THREE COUNTS IN `mutation-claims-are-executed.md` WERE INVALIDATED BY THIS CHANGE
and not updated: it said thirteen manifest entries, ten behavioural reds, and twelve
of thirteen clause-provable. Adding a fourteenth entry makes those fourteen, eleven,
and thirteen of fourteen. The record is where `test_mutation_harness.py`'s own
docstring sends a reader for that measurement, and the same commit corrected the
identical sentence in `docs/guard-inventory.md` — so the tree shipped two
same-dated documents contradicting each other on one number. Swept by subject this
time rather than by the phrase I had been shown; no other record is stale.

Also:
  * `docs/guard-inventory.md` still said "what falls outside that window" one
    sentence after arguing the boundary is not a window.
  * the same file's "one implementation rather than five" read as contradicting the
    commit message's "rather than two"; both are true at different scopes and the
    sentence now says which it means.

refs #806

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 16:19:54 +02:00
timothyandClaude Opus 5 b0f42f14a0 fix(806): guard populations over FILES derive from the git index, not a filesystem walk
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 10s
PR Gates / Docs update reminder (pull_request) Successful in 13s
PR Gates / decisions lifecycle (pull_request) Successful in 18s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 11s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 25s
review-verdict/h10 Awaiting review verdict for b0f42f1
Review verdict / Set review-verdict status (pull_request_target) Successful in 10s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 5m38s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m35s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m11s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 5m50s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
`testing.guard-derives-population-from-source` (#774) says a completeness guard
derives its population from an authoritative source, and its worked examples are an
enum and the generated OpenAPI document. It was silent on the commonest population
in our own guards — files in a directory — and every one of them answered with a
filesystem walk. A walk is not an authoritative source: it reports build output,
generated shims and editor droppings, and it differs per machine.

#778 measured the cost by getting the same population wrong three times in one PR
while implementing the milestone that exists to prevent it. The worst shape was
`Path.rglob` enumerating `.husky/_/` — 17 shims `npm ci` writes, gitignored and
untracked — which made that guard RED on every developer checkout and GREEN in CI,
whose `script-tests` job never runs `npm ci`. Only that one guard was fixed then.
This audits the rest.

CONVERTED (a completeness claim over tracked files):
  * `test_guard_inventory.py` — its `.husky/` walk excluded `.husky/_/` only because
    `_` is a directory, so the obvious "make it recursive" edit would have
    reintroduced #778's defect in the repo's own model guard. Both halves are gated
    on the index: the callers, and the `scripts/…` paths they name — the second was
    left on `Path.exists()` in the first cut and found by cold review.
  * `test_hook_fire_log.py` — an untracked scratch `.sh` in `.claude/hooks/` was
    demanded to carry instrumentation.
  * `test_ci_image_pin_population.py` — and `*.yaml` added: Gitea accepts both
    spellings, so a `.yaml` workflow adopting the toolchain image was structurally
    invisible while the test read as covering every workflow.
  * `test_remote_state_inventory.py` — folded onto the shared derivation, so the
    rule has one implementation rather than two.

ASSESSED AND RECORDED, not silently skipped:
  * `test_ci_release_path_scan_job.py::_repo_copy` — not a completeness claim, but
    it takes its file LIST from the index anyway for hermeticity, since `copytree`
    copied untracked files and `__pycache__` into a tree whose behaviour the probes
    measure. Content still comes from the working tree, and the copy is NOT a git
    repo, so neither file that step runs may use the helper — written down because
    `MARKED_JOBS` is left open as a residual gap, which invites editing exactly
    that file.
  * `test_ci_dropped_step_guard.py` — no filesystem population at all; its members
    come from the parsed workflow.
  * The decisions corpus keeps its walks and is recorded as unexamined rather than
    cleared. This is not "replace every glob".

`scripts/tests/tracked_files.py` is the single derivation.
`test_guard_populations_derive_from_git.py` proves it in two directions, which are
complements rather than a second opinion — measured both ways:
  * REMOVAL, exhaustively: every member of every registered derivation is dropped
    from the index in turn and must disappear while still on disk. One victim was
    not enough — `derived_guard_files` unions four contributors, so a mutant putting
    only one back on a walk passed. This catches a hardcoded `.exists()` admit and
    memoisation, which the other direction cannot.
  * The CALL LOG: a derivation may READ a file but must not LIST a directory. This
    catches an append-only source that yields nothing on this machine — #778's
    shape — which removal cannot see, because it has nothing to remove. Its limits
    are stated in one place and are true in both directions.

Both the population and the proofs are registered against a hand-written scope
mirror that carries its own equality check, and the import matcher's 16 forms are
pinned as a table so the next edit cannot silently re-open one.

The mutation this guard declares in `mutation_manifest.py` (#790) is the real
defect cold review found in its own first round.

Docs: `testing.guard-derives-population-from-source` gains the file-population case
and why the disk is not authoritative; `docs/guard-inventory.md` carries the
per-guard audit table, including the two no-change verdicts and the decisions
corpus recorded as unexamined.

Seven rounds of independent cold review (Codex GPT-5.6 and Opus, alternating) —
the per-round findings and their measurements are in the PR.

fixes #806

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 15:48:46 +02:00
15 changed files with 1131 additions and 83 deletions
+2 -2
View File
@@ -200,8 +200,8 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
| `testing.enumerating-guard-identity-not-position` | A guard that cross-checks a hand-reviewed registry against call sites discovered across the whole repo must key each entry on properties INTRINSIC to the site — file, kind, and the value source text — and never on its absolute line or column. A registry keyed on position is a function of every other file in the repo, so a branch that never touches the guard can invalidate it; and because each PR is green against its own base, that failure is structurally invisible pre-merge and lands on `main` after review and after the merge gate. Dropping the position keeps every mutation the guard exists for — a NEW site, a REMOVED site and a CHANGED value each still fail, since each changes the identity multiset — and costs exactly ONE case, which must be stated rather than implied: a SAME-IDENTITY SUBSTITUTION within one file (delete a registered site, add a different unreviewed one with the same kind and value token, net-zero count) now passes. A REPORTED failure still prints the discovered line:column, because identity and diagnostics need not share a format. Comparison stays a MULTISET count rather than set membership, so two sites in one file sharing an identity must be discovered exactly that many times and a third occurrence still fails. A SCANNER test that asserts real AST positions against FIXED inline fixtures is the opposite case and keeps its line/column identity — it has no churn, because its input does not move. | 2026-07-27 | [link](records/testing/enumerating-guard-identity-not-position.md) |
| `testing.fix-ships-a-witnessed-red-test` | A commit claiming to fix something may carry a `Proves: <pytest selector>` trailer; when it does, `scripts/prove-fix.sh` must show that selector GREEN with the fix and RED with the code side reverted, and CI enforces it per-PR. The trailer is opt-in — an unproven commit is allowed — but a claimed proof that does not hold fails the build. | 2026-08-16 | [link](records/testing/fix-ships-a-witnessed-red-test.md) |
| `testing.full-replace-asserts-field-list` | Any path that writes a WHOLE entity or a WHOLE child collection — a PUT-replace handler, a hand-built request object, a test comparer standing in for one — derives its field list from the authoritative type and asserts SET EQUALITY against it, rather than enumerating the fields by hand. A hand-written list is correct on the day it is written and structurally unable to report the day it stops being: the field that drifts is the one nobody wrote a line for, so no amount of care in the existing lines can reach it. The failure is silent by construction — a full replace with a field omitted returns HTTP 200 and destroys that field's value (#754 drifted from a 28-property DTO by one and cleared it; the symptom arrived hours later as missing pixels). SECOND CLAUSE, separable from the first: where a replaced child row carries state keyed to its identity — progression, ordering, an enumerator position — the handler RECONCILES BY ID rather than delete-and-reinsert, because reinsertion silently resets state a client never asked to touch (#252: a schedule PUT reset fill-group progression; #500: a dedup fix became permanent data loss because the add filter and the remove filter used different keys, so the two halves must agree on the key). Delete-and-reinsert is acceptable ONLY where no such state exists, and that emptiness is a fact about today's schema that a later feature can silently invalidate — so record it where the handler is, dated, rather than leaving it to be re-derived. The canonical worked example is `ToolCatalogTests.Every_Write_Tool_Should_Declare_Exactly_Its_OpenApi_Request_Body_Fields`, which reads the accepted fields from the generated OpenAPI document and compares both directions. | 2026-08-21 | [link](records/testing/full-replace-asserts-field-list.md) |
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
| `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) |
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
| `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) |
| `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin <its-own-name> <label> <capture\|stream>` as its FIRST act, before anything reads stdin. Two records are appended per invocation — a `fire` record on entry and an `exit` record carrying the exit status and the decision — to a session-scoped JSONL log. THE DECISION IS READ FROM WHAT THE HOOK ACTUALLY EMITTED, never declared by the hook author: Claude Code hooks (`capture` mode) always exit 0 and communicate by PRINTING JSON, so their stdout is diverted and replayed, and the recorded decision is parsed from those bytes; git hooks (`stream` mode) decide by EXIT CODE and their stdout is live progress text a human is watching, so it is not diverted and the decision is the status. That split is not a tuning knob — capturing a slow pre-push hook's output would hold it back until the end and read as a hang, and inferring a git hook's decision from absent JSON would put the report back into the guessing business this record exists to end. The population is DERIVED from `.claude/hooks/*.sh` by `scripts/tests/test_hook_fire_log.py`, so a new hook is uninstrumented-and-red rather than silently unobserved, and the report lists every hook that EXISTS rather than every hook that appears in the log — a report built from the log alone can only show hooks that fired, which makes the never-fired hook, the one finding worth having, invisible. THE INSTRUMENTATION MUST BE INVISIBLE TO THE HARNESS, and this is the load-bearing half: it sits in the stdin and stdout path of the most authoritative guards in the repo, so a differential test drives EVERY hook with and without it over a payload matrix and demands byte-equal stdout and equal exit status. It fails OPEN in exactly one direction — if the log cannot be written the hook behaves exactly as before — because observability that breaks a guard is worse than the blindness it replaces. Two mechanical traps are pinned by tests rather than left to care: stdout must be replayed from the FILE, since `out=$(cat f)` strips trailing newlines and delivers a guard's JSON one byte short with no parser anywhere to complain; and stdin must never be slurped when it is a TTY, because an interactive `git commit` hands its hooks a terminal and `cat` would block forever, hanging the commit the instrumentation was added to observe. | 2026-08-14 | [link](records/testing/hook-reports-its-own-execution.md) |
| `testing.live-e2e-prepush-timing` | Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab. | 2026-07-21 | [link](records/testing/live-e2e-prepush-timing.md) |
| `testing.mutation-claims-are-executed` | A `MUTATION` row in `docs/guard-inventory.md` is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`, and `scripts/tests/test_mutation_harness.py` applies that mutation to an isolated copy of the repository on every run and requires the row's OWN named test to go red. The manifest and the MUTATION rows are compared for SET EQUALITY in both directions, so a row cannot claim the grade without a mutation and a mutation cannot outlive the grade it justifies. EXIT STATUS IS NOT THE VERDICT: each entry also declares the DIAGNOSTIC its red must carry, matched against pytest's exception output alone, because pytest reports a crashing test exactly as it reports a detecting one and a red for an unrelated reason is evidence about nothing. WHERE THE GUARD IS ITSELF A TEST, `target` may differ from `guard` and the exact-once check applies to the declared TARGET. Two shapes are admissible and the choice is not free. Where the guard's assertion IS the check — a completeness comparison against a Markdown inventory — the mutation goes into the guarded ARTIFACT, per `testing.guard-ships-with-mutation-proof`'s checker-guard exception, because mutating such a checker's own POPULATION demonstrates a false POSITIVE while proving nothing about the detection the row claims. Where the guard is a test module wrapping a separately mutable DETECTOR or helper, the clause may be in that detector, since disarming it is a real clause disarm and the module's own assertion is what notices. THE MUTATION IS DECLARED, NEVER INFERRED: a harness that guessed which clause of a 90-line hook is the guard would manufacture the confident-but-empty coverage this exists to prevent, which is why `testing.guard-ships-with-mutation-proof` rejected a generic runner. Where a proof test already names its clause in source, the manifest reuses THAT string, so a retarget in either place is caught by the other. COARSENESS IS RECORDED, NOT HIDDEN: each entry is graded `CLAUSE` or `DETECTOR`, and a `DETECTOR` entry — one whose detector accumulates faults from independent arms, so disarming any single arm leaves its proof test green — must CARRY the finer mutation that survived, which is re-run every time and required to keep surviving. Guards that are not graded `MUTATION` each carry a STATED reason in that same manifest, keyed on the guard and compared for SET EQUALITY against the inventory's `GUARD` rows in both directions — so a new guard cannot arrive without someone writing what a proof would need, and a reason cannot outlive the row it is about. Keying the reason on the row's GRADE instead is tautological (a new guard inherits one and nobody looks at it) and a pinned COUNT moves only on net change; both were tried and are rejected. The sandbox is a real git repository built from `git ls-files` with working-tree content, never a filesystem walk. | 2026-08-22 | [link](records/testing/mutation-claims-are-executed.md) |
@@ -5,8 +5,8 @@ status: active
since: '2026-08-13'
supersedes: none
superseded-by: none
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`.'
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · issues: #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down.'
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
mechanics: 'Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.'
---
@@ -78,6 +78,103 @@ the answer is to create one, never to approximate it with a predicate over text.
accompanying prose a reviewer can falsify each round is worse than no guard, because — by this
record's own argument — a guard described as sound stops being re-examined.
**When the population is FILES, the authoritative source is the git index (#806).** The worked
examples above are an enum and a generated document, both unambiguously authoritative, and the
record was silent on the commonest population in this repo's own guards: files in a directory. Every
one of them answered with a filesystem walk, and #778 measured what that costs by getting the same
population wrong three times in one PR — a content filter that omitted `git fetch`, a non-recursive
`glob` that missed four nested files, and finally `rglob`, which enumerated `.husky/_/`: 17 husky
shims generated by `npm ci`, gitignored and untracked. That last one made the guard **red on every
developer checkout and green in CI**, whose `script-tests` job pip-installs but never runs `npm ci`.
The reason a walk keeps losing is not that each traversal was written carelessly; two of the three
were the obvious correction to the one before. It is that the disk answers a question about the
MACHINE and the guard is asking one about the REPO. Those coincide often enough for a walk to look
right and diverge exactly where generated output lands, which is to say wherever the tooling is
installed and nowhere else. The index is the repo's own statement of what it contains — the same
set of files every checkout receives from a clone, and excluding untracked files by construction
rather than by an exclusion list somebody maintains. It is not immutable and it is per-worktree; the
claim is not that it never changes, but that it changes only through a deliberate git operation — staging, a checkout, a
reset, a merge — whereas the disk changes whenever a build runs. Note what that buys over `.gitignore`-awareness: `.husky/_/`
happens to carry its own `.gitignore`, but a stray `foo.sh` in `.claude/hooks/` carries nothing, and
only the index knows it is not part of the repo.
The direction of the failure is worth naming, because it inverts the usual worry about a guard.
Under-enumeration hides a defect; this over-enumerated, and reddened correct trees. A guard that
fails everywhere except where it runs is not a cautious guard, it is a guard nobody reads — and it
had done that to the artifact whose entire thesis is population correctness.
**This did not become "replace every glob", and the boundary is the same one drawn above.** The
question per guard is whether it makes a completeness claim over TRACKED files. `_repo_copy` in
`scripts/tests/test_ci_release_path_scan_job.py` assembles a fixture and asserts nothing about
which files it found; it takes its file list from the index for HERMETICITY, which is a different
reason, and its docstring distinguishes the two.
`scripts/tests/test_ci_dropped_step_guard.py` has no filesystem population at all — it reads the
parsed workflow. Converting either would have been a change with no defect behind it, which spends
the credibility this rule needs when it does bite. The per-guard verdicts, including the two
no-change ones and the decisions corpus recorded as unexamined rather than cleared, are tabled in
`docs/guard-inventory.md`.
**The residue, named.** `git ls-files` reports INDEX entries, so a guard joins the population when
it is STAGED rather than when the file appears. Nothing local runs these checks — `.husky/pre-commit`
runs lint-staged, the decisions guard, the root-PNG check and `dotnet format`, and no husky hook runs
pytest — so the red arrives from `pr-checks.yml::script-tests` on the PR. A file deleted from the
working tree but not yet staged is still listed; `tracked_paths` asserts existence rather than
filtering it out, because a filter is what makes a missing member unrepresentable, which is this
record's first paragraph applied to its own implementation.
**Two traps specific to converting an existing guard, both of which this rule caught inside its own
implementation.** FIRST, follow the data to where members are actually ADMITTED, not just to where
the walk starts: a scrape that reads its caller files from the index and then admits the paths they
name on `Path.exists()` is half-derived and reads as fully derived. SECOND, a proof that a
derivation excludes untracked files must remove EVERY member in turn, not one. `derived_guard_files`
unions four contributors; a single victim is always drawn from whichever sorts first, so a mutant
putting only one contributor back on a filesystem walk passes. A sample cannot see the source it did
not draw from — this record's opening argument, one level down, inside the artifact written to
enforce it. Exhaustive removal is cheap — about a second at the population sizes here.
**Removal is only half the property, and the second half must not itself be machine-dependent.** A
source contributing ONLY untracked members adds and never removes, so a removal-based check has
nothing of its to take away: an `rglob` appending `.husky/_/` leaves the removal proof GREEN. Scope
that claim on both axes, because it is narrower than it first reads. A broader `rglob` that also
displaces tracked members DOES redden removal, so the blind spot is the append-only shape rather
than every filesystem walk; and the append-only shape is blind only where the walked directory
yields nothing — with the shims present its members are there to remove and removal reddens too. The
gap is therefore an append-only source that is empty ON THIS MACHINE, which is exactly the CI
checkout, and exactly where a guard going quiet is invisible.
The obvious complement — arrange an untracked file and require it not to enter — is a trap this
change fell into and backed out of twice, and the reason is worth more than the rule. Writing probe
files into the checkout under test means a probe in the `test_*.py` scope is a file pytest COLLECTS
mid-session, probe names collide across concurrent runs, `finally` does not survive a SIGKILL, and a
concurrent `git add -A` can stage one — defects in the test rather than in the thing tested.
Neutralising the shared derivation and requiring the population to go empty is clean but misses the
`.husky/_/` source on any machine where `.husky/_/` does not exist, which is the `script-tests`
checkout, since that job never runs `npm ci`. Both formulations reproduce the green-in-CI /
red-on-a-laptop asymmetry this record exists to abolish, inside the proof written to abolish it.
**So watch for the property that needs no arranged state: a directory LISTING issued while the
derivation runs.** Listing is the commonest way a derivation discovers a member the index does not
know about, and a walk issued during the derivation is caught on any machine — an `rglob` fails even
where the directory it walks is empty, because the evidence is the call rather than what it
returned. State its reach honestly, and state it once: what is observed is any
call that goes THROUGH ONE OF THE SPIES, whenever it happens — the check's docstring works through
the instances, and this record does not copy them, because the copy drifted from the original inside
a single commit. It is a regression guard against the shapes that arrive by accident, not a
boundary, and a guard sold as a boundary stops being re-examined.
**Three drafts of that one sentence were wrong, all in the same direction, and the third was wrong
in the copies after the original had been fixed** — which is the completeness rule biting the prose
that describes it. "Synchronously inside the call" was falsified by a thread finishing during the
drain; "while the patch is active" was falsified by a spy reference captured inside the window and
invoked after it, which still records. Each draft named the mechanism the author had in mind rather
than the one that decides, and each understated the coverage. Understating is the safe direction and
still worth correcting: a limit stated too narrowly invites someone to build the case it appears to
exclude. The durable lesson is the one this record already gives for populations — do not keep a
second copy of a statement that is still being corrected. Its complement is removal, which catches the shape that admits a HARDCODED
path without listing anything (`if (REPO_ROOT / "x").exists(): add` — the defect this change shipped
in its own first round). Neither alone is the property.
**What this record does not cover.** A population of *sites in code* — the places that dispatch on a
value — has no external enumerator. #403 is that case: `PlaybackOrder`'s values are enumerable, but
the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs
@@ -5,7 +5,7 @@ status: active
since: '2026-08-13'
supersedes: none
superseded-by: none
rule: 'A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD''S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631''s suite was invoked by no CI job, #751''s step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719''s new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row''s own named test required to go red.'
rule: 'A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD''S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631''s suite was invoked by no CI job, #751''s step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719''s new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row''s own named test required to go red.'
signals: 'mutation proof · delete the guard alone see red · disarm the clause not the file · behaviour-only is not a proof · anti-vacuity assertion · guard wired is not guard running · a green job with no step output · `docs/guard-inventory.md` · set equality against the row set · proof ref resolves to a real def · the unproven majority is a moving figure — read it off the inventory · PROOF kind stops the regress · hook wiring is not hook existence · paths: `scripts/tests/test_guard_inventory.py`, `scripts/tests/test_ci_dropped_step_guard.py`, `scripts/ci-prove-ban-detects.sh` · issues: #775, #773, #751, #756, #719, #685, #631, #621, #510, #445'
mechanics: 'Proof classes are a closed vocabulary enforced by the inventory test; a `TOOLING` row may not claim a proof. `scripts/ci-prove-ban-detects.sh` is the one guard that runs its own mutation at CI time rather than in pytest, because the thing it proves — that the ban test is not deselected — is disarmable from inside pytest configuration.'
---
@@ -31,17 +31,17 @@ declared `target`, which is not always the guard's own file; and — via the pos
unmutated sandbox, so "the mutation was noticed" cannot be confused with "the test was already red".
The proofs that carry a disarm of their own run it during that control; the two that are plain
production set-equality checks have none, and the harness supplies theirs. It does **not** prove the declared clause is the only thing the guard hangs on.
For three of the thirteen entries (measured 2026-08-22: the BOM guard, `build_decisions_catalog.py`
For three entries (measured 2026-08-22: the BOM guard, `build_decisions_catalog.py`
and `prove-fix.sh`) the redness arrives through the proof test's own "this clause has moved, RETARGET
it" assertion rather than through changed behaviour. That is the intended reading rather than a hole:
those tests perform their behavioural disarm themselves on every green run; what they could not do
was notice their own clause reference going stale. The remaining ten redden behaviourally, and each
was notice their own clause reference going stale. Every other entry reddens behaviourally, and each
one's declared diagnostic says which.
**The `DETECTOR` grade is the honest half, and it exists because measuring found a case.** #775's
record concedes that neutering `pin_population_faults` wholesale is "coarser than disarming one
clause at a time — coarse enough that a single surviving clause would not be noticed". Running the
finer mutations settled which guards actually admit clause-level proof: twelve of the thirteen do,
finer mutations settled which guards actually admit clause-level proof: all but one do,
including
`pin_population_faults`, where disarming the against-the-registry comparison alone reddens the proof
because a job that loses its `container:` block leaves the two derived sets equal. One does not.
+90 -15
View File
@@ -1,9 +1,16 @@
# Guard inventory (ersatztv#774 / #775)
Every executable guard **file** in this repo, what it blocks, and whether it ships a proof it can go
red. `scripts/tests/test_guard_inventory.py` derives the population from the filesystem and the
red. `scripts/tests/test_guard_inventory.py` derives the population from the **git index** and the
workflow/hook call sites and asserts **set equality** against the `Guard` column, so a new guard
cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.
The index rather than a filesystem walk since ersatztv#806 — the practical consequence is that a new
guard joins the population when it is **staged**, not when the file appears. Nothing local runs
these checks at all: `.husky/pre-commit` runs lint-staged, the decisions guard, the root-PNG check
and `dotnet format`, and `grep -rn pytest .husky/` returns nothing. The runner is
`pr-checks.yml::script-tests`, `on: pull_request`, so the red arrives in CI — plus
`docker-build.yml:723` on the release path, which re-runs two of these files
(`test_ci_dropped_step_guard.py` and `test_ci_release_path_scan_job.py`) as a `needs:` of `build`.
**Read `docs/decisions/records/testing/guard-derives-population-from-source.md` and
`…/guard-ships-with-mutation-proof.md` before editing a guard or adding a row.**
@@ -40,11 +47,11 @@ exists because arguments of that shape have been wrong here six times.
## Scope limit, stated rather than implied
This inventory covers guard **files**, discovered by globbing `.claude/hooks/*.sh`, `.husky/*` and
`scripts/tests/test_*.py`, plus every `scripts/…` path referenced by a workflow or a hook. The
classes below are outside that population. They are listed because the first version of this section named
only the first one, and cold review found that the very guards this inventory shipped with were
sitting in the gap:
This inventory covers guard **files**, discovered by reading `.claude/hooks/*.sh`, `.husky/*` and
`scripts/tests/test_*.py` **out of the git index**, plus every `scripts/…` path referenced by a
workflow or a hook. The classes below are outside that population. They are listed because the first
version of this section named only the first one, and cold review found that the very guards this
inventory shipped with were sitting in the gap:
1. **Guards inline in workflow YAML** — most importantly `pr-checks.yml:ci-image-pin`. "Which jobs
are guards" needs a judgement call per job the filesystem cannot supply. Two were audited under
@@ -56,15 +63,24 @@ sitting in the gap:
`pr-checks.yml` error message, so removing the step that runs it would leave its row intact.
4. **Nested and non-lowercase paths** beyond `scripts/tests/` — a guard under
`scripts/scripted-schedules/`, or with an uppercase name, is invisible to the scrape.
5. **`.yaml` workflows and non-`.sh` hooks** — the globs are `*.yml` and `*.sh` only.
5. **Non-`.sh` hooks** — the hook pattern is `*.sh` only. (`.yaml` workflows are no longer in this
gap: the caller scan matches `*.yml` and `*.yaml` since #806, because Gitea accepts both.)
6. **Transitive calls** — a script invoked only by another script, rather than by a workflow or
hook, is not discovered.
7. **Non-`test_` modules under `scripts/tests/`** — the glob is `test_*.py`, so `conftest.py`,
`mutation_harness_lib.py` and `mutation_manifest.py` are outside the population and hold no rows.
They are not guards (they assert nothing on their own), but the second and third ARE what
`test_mutation_harness.py` is made of, so gutting either would take that guard with it. What
catches that is the guard's own row: its declared mutation targets `mutation_harness_lib.py`, and
its proof test refuses to run if the clause it names has moved.
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
population and hold no rows. They are not guards (they assert nothing on their own), but the
middle two ARE what `test_mutation_harness.py` is made of, so gutting either would take that
guard with it. What catches that is the guard's own row: its declared mutation targets
`mutation_harness_lib.py`, and its proof test refuses to run if the clause it names has moved.
`tracked_files.py` has no such backstop — it is load-bearing for every module that imports it
(#806), and an edit to it is covered only by those modules' own proofs. Recorded rather than
force-fitted: a row for a library would need a `Kind` the vocabulary does not have.
8. **Nested workflow directories** — the workflow scope is direct children of `.gitea/workflows`, so
a tracked `.gitea/workflows/nested/x.yml` is invisible to the caller scan and to
`test_ci_image_pin_population.py`. Left as scope rather than widened: whether Gitea executes
nested workflow files was not verified here, and widening on an unverified premise risks a
permanent red on a correct tree, which is how a correct guard gets deleted.
Hook **wiring** is checked (`test_every_hook_file_is_actually_WIRED` reads `.claude/settings.json`
and the husky hooks with full-line comments stripped), so a hook file whose registration is deleted
@@ -75,6 +91,64 @@ through a wrapper or a constructed path reads as unwired. It catches deletion, w
case; it does not catch deliberate disablement. The check does not extend to the `scripts/` half at
all.
## File populations and where they come from (ersatztv#806)
Every guard here whose members are FILES derives them from the **git index**, never a filesystem
walk. The disk is not an authoritative source: it reports build output and editor droppings and
differs per machine, so a guard derived from it asserts a different population in CI than on the
laptop of the person it is meant to stop. `scripts/tests/tracked_files.py` is the single derivation
and carries the full rationale; `scripts/tests/test_guard_populations_derive_from_git.py` proves it,
in both directions: removing EVERY member of each registered derivation from the index one at a time
and requiring it to disappear while still on disk, and watching for a directory LISTING issued while
the derivation runs (reading files stays allowed). Removal alone is blind to a source that
contributes only untracked members — an `rglob` reaching `.husky/_/` adds and never removes — and
any check phrased as "an untracked file must not enter" is itself machine-dependent, because the
untracked file has to exist. Watching for the call needs no arranged state.
That second check is a regression guard against the accidental shapes, not a boundary: what is
observed is any call that goes **through one of the spies**, whenever it happens — the spy records
into a list that outlives the patch, so a reference captured during the window and invoked after it
still counts. Whether the call goes through a spy is what decides, not when. The instance list — what
never reaches a spy at all — lives in the check's own docstring and is deliberately not restated
here, because a second copy of it drifted from the first within one commit.
One deliberate exception to "from the index" sits in the same file: the registration check lists
`scripts/tests/test_*.py` from disk on purpose, because it is a superset check over what pytest
collects — an untracked stray there makes it MORE demanding, never blind, whereas using the index
would let an unstaged new guard escape registration.
Its own limit, stated because a check described as complete stops being re-examined: it finds
derivations by PARSING each `test_*.py` for an import of the shared helper, so a module that derives
a file population some other way is invisible to it, and no mechanical check can close that (#774
reached the same conclusion about detecting filter-shaped guards by token).
The audit #806 asked for, recorded whichever way it came out, because "we looked and left it" and
"we never looked" are indistinguishable a year later:
| Guard | Population | Completeness claim over tracked files? | Outcome |
| --- | --- | --- | --- |
| `test_guard_inventory.py` | `.claude/hooks/*.sh`, `.husky/*`, `scripts/tests/test_*.py`, workflow/hook callers | **yes** — set equality against this table | converted to the index; `.husky/_/` had been excluded only because `_` is a directory, so the obvious "make it recursive" edit would have reintroduced #778's defect here |
| `test_hook_fire_log.py` | `.claude/hooks/*.sh` | **yes** — every hook must be instrumented | converted; an untracked scratch `.sh` used to demand instrumentation and redden the suite on that checkout alone |
| `test_ci_image_pin_population.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — "docker-build is the ONLY workflow pinning the toolchain image" | converted, and `*.yaml` added: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible while the test read as covering all of them |
| `test_pr_changed_files.py` | `.gitea/workflows/*.y*ml` | **yes** — "no OTHER workflow writes the review-verdict status" | converted. Not on #806's list and found by cold review on the pushed head: an untracked `.yaml` dropped in `.gitea/workflows/` reddened two guards while absent from the index — the issue's list of files to assess was a starting point, not the population |
| `web/src/api/pageSizeCallSites.guard.test.ts` | `web/src/**/*.{ts,tsx,mts,cts}` via `import.meta.glob` | **yes** — an unregistered discovered site fails | DEFERRED to #819, assessed not skipped. The glob is a documented workaround: `@types/node` is deliberately out of `tsconfig.app.json`, and wiring it in was tried and reverted (it leaked Node's `setTimeout` into the app project and broke three unrelated tests), so there is no `node:child_process` to reach the index from. Over-enumerates, so it fails loudly rather than going blind |
| `test_ci_release_path_scan_job.py` | `.gitea/workflows/*.y*ml` + `scripts/**` | **no** — a fixture assembling a tmp harness, asserted about behaviour not membership | takes its file LIST from the index anyway, for hermeticity not completeness: `shutil.copytree` copied whatever was on disk, including untracked files and `scripts/__pycache__`, into a tree whose behaviour the probes then measure. Content still comes from the working tree. The copy is not a git repo, so the two files this step RUNS may not use the helper — see the fixture docstring |
| `test_ci_dropped_step_guard.py` | the parsed workflow document | **no filesystem population at all** | unchanged; its residual is `MARKED_JOBS`, a SCOPE mirror of the required contexts on `main`, which #806 does not close |
`test_remote_state_inventory.py` fixed its own population under #778 and kept a private copy of the
derivation; #806 folded it onto the shared one, so that module is covered by the proof above like
the rest. Across the whole change, every module that derived a file population its own way now goes
through `tracked_files.py` — one implementation of the rule instead of one per module. The
registered derivations are listed in `DERIVATIONS` in
`scripts/tests/test_guard_populations_derive_from_git.py`; this page deliberately keeps no count
of them.
**Still on filesystem walks, deliberately out of scope:** the decisions corpus
(`scripts/decisions_lib.py`'s `active_files()`, and the suites over it). Its members are `docs/`
Markdown with no generated-file pressure and a different lifecycle, and folding it in here would
have been the reflex this milestone argues against — a change with no defect behind it. It is
recorded as unexamined rather than as cleared.
## Inventory
| Guard | Blocks | Kind | Proof | Proof ref |
@@ -125,6 +199,7 @@ all.
| `scripts/tests/test_decisions_lib.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_decisions_validate.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_guard_inventory.py` | the `script-tests` job, on an unclassified guard or a stale proof ref | GUARD | MUTATION | `test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist` |
| `scripts/tests/test_guard_populations_derive_from_git.py` | the `script-tests` job, on a guard whose file population admits a file git does not track | GUARD | MUTATION | `test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file` |
| `scripts/tests/test_hook_fire_log.py` | the `script-tests` job, on a hook that stops reporting that it fired, or whose reporting changes what the harness sees | GUARD | MUTATION | `test_hook_fire_log.py::test_a_hook_that_LOSES_its_instrumentation_is_DETECTED` |
| `scripts/tests/test_jq_preflight.py` | the `script-tests` job | PROOF | NONE | — |
| `scripts/tests/test_merge_consent_base_change.py` | the `script-tests` job | PROOF | NONE | — |
@@ -150,7 +225,7 @@ graded `MUTATION` without declaring its clause fails the suite. Full contract an
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
Two things that column still does not say. It does not say the declared clause is the ONLY thing the
guard hangs on — for three of the thirteen rows (measured 2026-08-22) the redness arrives through the
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
behaviour, which catches the recorded proof going stale but not much else. And one row is graded `DETECTOR` rather than `CLAUSE` in the
manifest: `instrumentation_faults` accumulates from four independent arms and a stripped hook trips
@@ -160,7 +235,7 @@ clause-provable and the entry is regraded.
## What the numbers say
35 guards, 6 tooling scripts, 19 proof files. **13 guards carry a mutation proof; 6 are
36 guards, 6 tooling scripts, 19 proof files. **14 guards carry a mutation proof; 6 are
behaviour-only; 16 have none.** These figures are asserted against the table by
`test_the_summary_counts_match_the_table` — they were wrong in the first draft (28/4/6/3/19 against
a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it,
+19
View File
@@ -127,6 +127,25 @@ MUTATIONS: tuple[Mutation, ...] = (
"comparison notices it has moved to the bare runner. This is the clause #790 asked for "
"instead of neutering `pin_population_faults` wholesale.",
),
Mutation(
guard="scripts/tests/test_guard_populations_derive_from_git.py",
target="scripts/tests/test_guard_inventory.py",
clause="if ref in tracked:",
replacement="if (REPO_ROOT / ref).exists():",
proof="test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file",
granularity=CLAUSE,
expect="after git stopped tracking them",
why="THE GUARD IS A TEST, so the mutation goes into the guarded ARTIFACT — one of the "
"derivations it watches — rather than into the checker, per the checker-guard exception. "
"The clause is the exact defect this guard was written after: `derived_guard_files` read "
"its CALLERS from the index and then admitted the paths they name on `Path.exists()`, so a "
"tracked workflow naming a script that exists on one machine only entered the population "
"there, red on that checkout and green in CI (#778's third shape, found by cold review "
"inside #806 itself). Note what this mutation does NOT do: on a clean tree the mutated set "
"is identical, so `test_guard_inventory.py`'s own assertions stay green — only narrowing "
"the index, which is what the proof does, separates them. That is why the proof has to "
"remove EVERY member rather than sample one.",
),
Mutation(
guard="scripts/tests/test_guard_inventory.py",
target="docs/guard-inventory.md",
@@ -30,6 +30,12 @@ import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
# ASSESSED FOR ersatztv#806: this file has NO filesystem-derived population. Its members come from
# the PARSED workflow (`_marked(job)` reads the marked steps out of `_DOC`), which is already an
# authoritative machine-readable source, so the index changes nothing here. Its known residual is at
# the other altitude — `MARKED_JOBS` is a hand-written mirror of the required contexts on `main`,
# a SCOPE rather than a population, and `testing.guard-derives-population-from-source` already
# carries it as this repo's canonical residual gap. #806 does not close it.
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
+22 -2
View File
@@ -38,9 +38,22 @@ from pathlib import Path
import pytest
import yaml
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
WORKFLOW = WORKFLOWS_DIR / "docker-build.yml"
# Resolved against the GIT INDEX rather than `Path.glob` (ersatztv#806), and `*.yaml` alongside
# `*.yml`: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible to the
# scope check below while reading as covered.
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
def workflow_files() -> list[Path]:
"""THE WORKFLOW POPULATION, from the git index. Named rather than inline so the shared proof in
`test_guard_populations_derive_from_git.py` can assert it never admits an untracked file."""
return tracked_paths(*WORKFLOWS)
# The image repository, without the tag. Matched as a whole path rather than by the bare
# `ersatztv-ci` token so a job pointing at a LOOK-ALIKE registry (a personal fork, a typo'd host)
@@ -265,7 +278,7 @@ def test_docker_build_is_the_ONLY_workflow_pinning_the_toolchain_image():
"""This file reads ONE workflow, which is itself a scope mirror needing its own check.
`WORKFLOW` hardcodes `docker-build.yml`, and the implicit claim that no other workflow uses
the toolchain image mirrors a machine-readable source (the glob of `.gitea/workflows/*.yml`)
the toolchain image mirrors a machine-readable source (the tracked `.gitea/workflows/*.y*ml`)
that nothing consulted. `renovate.yml` already declares a `container:` with a different image,
so the shape is live. A future workflow adopting `ersatztv-ci:` would acquire no pin-population
guard, no single-tag check and no partition, silently, while `pin_population_faults`'s own error
@@ -274,13 +287,20 @@ def test_docker_build_is_the_ONLY_workflow_pinning_the_toolchain_image():
Found by cold review, which correctly noted this file criticises `MARKED_JOBS` for exactly this
and then shipped the same shape without even the dated comment `MARKED_JOBS` carries.
The population comes from the GIT INDEX (ersatztv#806). A `Path.glob` here answered a question
about the machine rather than about the repo: an untracked scratch workflow left in
`.gitea/workflows/` would be parsed and could redden this test on one checkout while CI, which
never sees it, stayed green. The pattern set gained `*.yaml` in the same change Gitea accepts
both spellings, so a `.yaml` workflow adopting the toolchain image was invisible here while this
test read as covering every workflow.
Checked by PARSING each workflow's `container.image`, not by grepping the file. A text search
reports `ci-image.yml`, which names the image because it BUILDS and PUSHES it a producer, not
a consumer. Grepping would have made this test permanently red on a correct tree, which is the
fastest route to a correct guard being deleted.
"""
others = []
for p in sorted(WORKFLOWS_DIR.glob("*.yml")):
for p in workflow_files():
if p.name == WORKFLOW.name:
continue
doc = yaml.safe_load(p.read_text()) or {}
+49 -3
View File
@@ -77,6 +77,15 @@ from pathlib import Path
import pytest
import yaml
# Imports the shared index derivation to build a HERMETIC fixture copy, not to derive a guard
# population — see `_repo_copy`. Recorded as such in POPULATION_EXEMPT in
# `test_guard_populations_derive_from_git.py`; the exemption lives there, not here, because a marker
# a file grants itself is a kill switch any prose mention can trip.
from scripts.tests import tracked_files
# Both spellings, matching what the converted guards consider the workflow set.
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
REPO_ROOT = Path(__file__).resolve().parents[2]
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
@@ -269,12 +278,49 @@ def _scan_body_and_env():
def _repo_copy(tmp_path: Path) -> Path:
"""A minimal executable copy of the repo: the workflow plus all of scripts/."""
"""A minimal executable copy of the repo: the tracked workflows plus the tracked `scripts/`.
ASSESSED FOR ersatztv#806. This is NOT a completeness guard — it is a fixture assembling a
harness, and no assertion in this file is about which files it found; the probes assert what the
scan command DOES to the copy. It takes its file LIST from the index anyway, for hermeticity
rather than completeness: `shutil.copytree` copied whatever was on disk, so untracked files and
`scripts/__pycache__` entered a tree whose behaviour the probes then measure.
WHAT THAT DOES AND DOES NOT BUY, stated exactly, because a fixture described as hermetic stops
being questioned. The list comes from the index; the CONTENT comes from the working tree, so an
unstaged edit to a tracked `scripts/**` file is still copied in. Making the content hermetic too
would need `git show`/`git archive` and would mean the probes stop testing the tree under edit,
which is the wrong trade for a test whose job is to catch a disarm in that tree.
An extra WORKFLOW in the copy is inert, but check what the step runs before relying on that:
`docker-build.yml:723` runs TWO files, `test_ci_dropped_step_guard.py` AND this one, and both
parse only `docker-build.yml`.
THE COPY IS NOT A GIT REPOSITORY, and that is the constraint to know before touching either file
that step runs. Neither may derive a population through `scripts/tests/tracked_files.py`:
`git ls-files` inside the copy fails, and the release-path scan step fails with it. The copying
happens HERE, in the real repo, which is why this fixture may use the index while its subjects
may not. `test_ci_dropped_step_guard.py`'s `MARKED_JOBS` is explicitly left open as a residual
gap, so a future session is invited to edit exactly that file this paragraph is what stands
between that edit and a broken release gate.
"""
dst = tmp_path / "repo"
(dst / ".gitea" / "workflows").mkdir(parents=True)
for wf in (REPO_ROOT / ".gitea" / "workflows").glob("*.yml"):
for wf in tracked_files.tracked_paths(*WORKFLOWS):
shutil.copy2(wf, dst / ".gitea" / "workflows" / wf.name)
shutil.copytree(REPO_ROOT / "scripts", dst / "scripts")
for rel in tracked_files._git_ls_files():
if not rel.startswith("scripts/"):
continue
source = REPO_ROOT / rel
# Diagnosed, not raised as a bare FileNotFoundError one line after using the module whose
# whole point is reporting this case comprehensibly.
assert source.is_file(), (
f"git tracks {rel} but there is no file there, so the harness copy would be incomplete "
"and the probes below would measure a tree that is missing part of the thing under test."
)
target = dst / rel
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, target)
return dst
+63 -30
View File
@@ -18,9 +18,11 @@ about a sample, but you CAN mechanically guarantee that every guard has been *cl
and that its claimed proof exists. That converts both rules from "remember to do this" into "the
suite goes red until you have". Specifically:
* the guard population is DERIVED globbed from `.claude/hooks/` and `.husky/`, plus every
`scripts/*.sh|py` referenced by a workflow or a hook and compared for SET EQUALITY against the
inventory's rows, in both directions;
* the guard population is DERIVED read out of the GIT INDEX for `.claude/hooks/` and
`.husky/`, plus every `scripts/*.sh|py` referenced by a workflow or a hook and compared for
SET EQUALITY against the inventory's rows, in both directions. The index rather than the disk
per ersatztv#806: a filesystem walk reports build output and editor droppings and differs per
machine, so it cannot be the authoritative source a completeness claim needs;
* every row's `Proof ref` is resolved to a real file and a real `def` in it;
* every row's `Kind` and `Proof` come from a closed vocabulary, so a typo cannot invent a state.
@@ -35,11 +37,22 @@ import re
from collections import Counter
from pathlib import Path
from scripts.tests import tracked_files
from scripts.tests.tracked_files import tracked_children, tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
HOOKS_DIR = REPO_ROOT / ".claude" / "hooks"
HUSKY_DIR = REPO_ROOT / ".husky"
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
# THE POPULATION SCOPES, resolved against the GIT INDEX rather than the filesystem (ersatztv#806).
# Directory + patterns instead of `Path.glob`; `scripts/tests/tracked_files.py` carries why the disk
# is not an authoritative source. `.husky` is the sharp case: it holds an untracked `_/` of 17
# npm-generated shims, and the previous `iterdir() ... if p.is_file()` excluded them only because
# `_` happens to be a directory — by accident, not by design, so the obvious "make it recursive"
# edit would have reintroduced #778's third defect inside the repo's own model guard.
HOOKS = (".claude/hooks", ("*.sh",))
HUSKY = (".husky", ("*",))
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
GUARD_TESTS = ("scripts/tests", ("test_*.py",))
KINDS = {"GUARD", "TOOLING", "PROOF"}
PROOFS = {"MUTATION", "BEHAVIOUR-ONLY", "NONE"}
@@ -64,31 +77,52 @@ _SUMMARY = re.compile(
def derived_guard_files() -> set[str]:
"""THE AUTHORITATIVE POPULATION, from the filesystem and the call sites — never a list.
"""THE AUTHORITATIVE POPULATION, from the git index and the call sites — never a list.
Three sources, unioned. The hook and husky directories are globbed whole, so a new hook is in
the population the moment it exists. The scripts half is discovered by scanning what the
workflows and hooks actually INVOKE, rather than globbing `scripts/` a script nothing calls is
not a guard, and globbing would drag in every helper and make the inventory a chore that gets
rubber-stamped.
Four contributors, unioned: three scope directories plus the paths those files REFERENCE. The
hook and husky directories are taken whole, so a new hook is in the population the moment it is
STAGED. The scripts half is discovered by scanning what the workflows and hooks actually
INVOKE, rather than taking `scripts/` whole a script nothing calls is not a guard, and taking
it whole would drag in every helper and make the inventory a chore that gets rubber-stamped.
Both halves are gated on the index: the callers by `tracked_paths`, the targets they name by the
`tracked` set below.
"The moment it is STAGED" rather than "the moment it exists" is the ersatztv#806 change, and
it is a strengthening: an untracked `foo.sh` dropped in `.claude/hooks/` used to enter this
population and demand an inventory row for a file that is not part of the repo red on that
checkout, green in CI, which is #778's third shape. Nothing weakens, because a guard that is not
staged is not on its way to anyone else either.
"""
found = (
{str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
| {str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()}
| {
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
# invoked and none is individually named anywhere. Globbing is the only derivation that
# matches how they actually run.
str(p.relative_to(REPO_ROOT))
for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
}
tracked_children(*HOOKS)
| tracked_children(*HUSKY)
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is invoked
# and none is individually named anywhere. Taking the directory whole is the only derivation
# that matches how they actually run.
| tracked_children(*GUARD_TESTS)
)
callers = list(WORKFLOWS_DIR.glob("*.yml")) + list(HOOKS_DIR.glob("*.sh"))
callers += [p for p in HUSKY_DIR.iterdir() if p.is_file()]
# THE REFERENCED TARGETS ARE GATED ON THE INDEX, NOT ON `Path.exists()`. Converting the CALLERS
# and leaving the members they contribute on a disk check would have left a quarter of this
# population answering a question about the machine: a tracked workflow naming
# `scripts/generated/helper.sh` that exists on one laptop only would enter there, demand an
# inventory row for a file that is not in the repo, and go red on that checkout while CI stayed
# green — #778's third shape, in the guard this file calls its model.
#
# A referenced path that git does not track is therefore dropped silently, and that is the right
# residual rather than an assertion: `_SCRIPT_REF` matches any occurrence, including inside a
# comment or an `::error::` string (limit 3 in `docs/guard-inventory.md`), so demanding that
# every matched path be tracked would redden a correct tree on a prose mention.
# Called through the MODULE, never `from … import _git_ls_files`. A direct name binding is
# captured at import time, and the exhaustive proof in
# `test_guard_populations_derive_from_git.py` then cannot narrow the index for this branch at
# all — every referenced target reports as surviving removal, which is a red for the wrong
# reason and, worse, means the branch is untested however the proof reads.
tracked = set(tracked_files._git_ls_files())
callers = tracked_paths(*WORKFLOWS) + tracked_paths(*HOOKS) + tracked_paths(*HUSKY)
for caller in callers:
for ref in _SCRIPT_REF.findall(caller.read_text()):
if (REPO_ROOT / ref).exists():
if ref in tracked:
found.add(ref)
return found
@@ -112,10 +146,9 @@ def wired_hook_files() -> set[str]:
comments, so every occurrence there is in a real command string.
"""
text = (REPO_ROOT / ".claude" / "settings.json").read_text()
for husky in HUSKY_DIR.iterdir():
if husky.is_file():
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
return {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text}
for husky in tracked_paths(*HUSKY):
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
return {rel for rel in tracked_children(*HOOKS) if rel.rpartition("/")[2] in text}
def inventory_rows() -> list[tuple[str, str, str, str]]:
@@ -210,8 +243,8 @@ def test_every_claimed_proof_names_a_test_that_exists():
def test_every_hook_file_is_actually_WIRED():
"""A hook file nothing registers is dead code holding an inventory row that reads as coverage."""
on_disk = {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
unwired = sorted(on_disk - wired_hook_files())
staged = tracked_children(*HOOKS)
unwired = sorted(staged - wired_hook_files())
assert not unwired, (
f"these hook files exist and have inventory rows but are referenced by neither "
f".claude/settings.json nor any .husky/ hook: {unwired}. They do not run. Either wire them "
@@ -0,0 +1,604 @@
"""No file population in this repo admits a file git does not track (ersatztv#806).
The regression for #778's third defect, hoisted to cover every guard that shares the mechanism
rather than being copied into each of them. `Path.rglob` enumerated `.husky/_/` 17 husky shims
generated by `npm ci`, gitignored and untracked so `test_remote_state_inventory.py` was RED on
every developer checkout and GREEN in CI, whose `script-tests` job checks out and pip-installs but
never runs `npm ci`. A guard that fails everywhere except where it runs trains its readers to ignore
it, and it did so on the artifact whose entire thesis is population correctness.
TWO PROOFS, because they fail differently and either alone leaves a hole.
* `test_the_primitive_REALLY_excludes_an_untracked_file` builds a throwaway git repo, commits one
file, leaves an identical sibling untracked, and runs the real derivation against it. It proves
the mechanism by EXECUTING it rather than by recognising its shape no monkeypatching, no
stand-in for git. Nothing here is a claim about `git ls-files`; it is `git ls-files`.
* `test_no_derivation_admits_an_untracked_file` narrows the tracked set under each real derivation
and requires the dropped member to vanish from the population even though the file is still on
disk and still matches the scope. That is the property stated over the ACTUAL guards, so a
future refactor that quietly reintroduces a filesystem walk in any one of them fails here rather
than on somebody's laptop.
`DERIVATIONS` is the reason this file is not one near-copy per derivation: a guard that starts deriving a file
population registers here, and both proofs cover it for free. The register is hand-written and that
is a SCOPE decision, not a population one per `testing.guard-derives-population-from-source`, a
scope mirroring an authoritative source needs its own equality check, and
`test_every_index_derived_module_is_registered` is it: it reads which modules import the shared
helper and demands each one appear below, so adding another derivation and forgetting this file is
red rather than silently uncovered.
"""
from __future__ import annotations
import ast
import glob
import os
import subprocess
import sys
from pathlib import Path
import pytest
from scripts.tests import test_ci_image_pin_population as image_pin
from scripts.tests import test_guard_inventory as guard_inventory
from scripts.tests import test_hook_fire_log as hook_fire
from scripts.tests import test_pr_changed_files as pr_changed
from scripts.tests import test_remote_state_inventory as remote_state
from scripts.tests import tracked_files
REPO_ROOT = Path(__file__).resolve().parents[2]
TESTS_DIR = REPO_ROOT / "scripts" / "tests"
class _EmptyScan:
"""An exhausted ITERATOR that is also a context manager, standing in for `os.scandir`.
Both halves are load-bearing and each was missing in turn. `os.scandir` is used as
`with os.scandir(...) as it`, so a bare iterator broke the context-manager protocol; and
`os.walk` does `entry = next(scandir_it)` on the result, so an ITERABLE defining only
`__iter__` broke that. Either way the enumeration assertion still fired with the right message,
but the report also carried a TypeError about the harness and a finding that arrives beside a
harness error invites doubting the finding rather than the code.
"""
def __iter__(self):
return self
def __next__(self):
raise StopIteration
def __enter__(self):
return self
def __exit__(self, *_exc):
return False
def close(self):
return None
def _as_relative_strings(members) -> set[str]:
"""Derivations return either repo-relative strings or absolute `Path`s; compare on one form.
Iterating into a set is also what DRAINS a derivation that returns a generator, which the
enumeration proof depends on see its call site.
An unexpected member type is REPORTED rather than stringified. `str(m)` on anything at all meant
a derivation yielding, say, nested generators produced plausible-looking members and compared
equal to nothing, which is a population check passing over data it did not understand.
"""
out = set()
for m in members:
assert isinstance(m, str | Path), (
f"a derivation yielded {type(m).__name__} ({m!r}); populations here are repo-relative "
"strings or absolute Paths, and stringifying anything else would compare a plausible "
"value against a set that can never contain it."
)
out.add(str(Path(m).relative_to(REPO_ROOT)) if isinstance(m, Path) else str(m))
return out
# (label, callable, a floor below which the derivation has plainly broken). `test_hook_fire_log`
# floors the same population at the same number for its own coverage assertions; that is not a
# duplicate guard masking another, because the two protect different consumers from going
# vacuous — delete this one and THIS file's proofs iterate over nothing while reporting success.
DERIVATIONS = (
("test_guard_inventory.derived_guard_files", guard_inventory.derived_guard_files, 25),
("test_hook_fire_log.hook_scripts", hook_fire.hook_scripts, 10),
("test_ci_image_pin_population.workflow_files", image_pin.workflow_files, 5),
("test_remote_state_inventory.derived_population", remote_state.derived_population, 40),
("test_pr_changed_files._workflow_files", pr_changed._workflow_files, 5),
)
# The floor matters only to the anti-vacuity test; the two property tests take the pair, so an
# unused parameter cannot drift into looking like an assertion they make.
_IDS = [d[0] for d in DERIVATIONS]
_PAIRS = [(label, derive) for label, derive, _ in DERIVATIONS]
# Modules that import the shared derivation WITHOUT deriving a guard population. Kept here, beside
# DERIVATIONS, so adding one is an edit to this file that a reviewer sees.
POPULATION_EXEMPT = {
# Uses the index to assemble a HERMETIC tmp fixture copy; nothing in it asserts membership.
"test_ci_release_path_scan_job.py": "index-derived fixture copy, not a population",
}
_HELPER = "scripts.tests.tracked_files"
_PACKAGE = ["scripts", "tests"]
# ------------------------------------------------------------------------------------------------
# ANTI-VACUITY FIRST — every assertion below compares sets, and a derivation that collapsed to
# nothing would satisfy all of them while proving nothing.
# ------------------------------------------------------------------------------------------------
@pytest.mark.parametrize(("label", "derive", "floor"), DERIVATIONS, ids=_IDS)
def test_each_derivation_found_something(label, derive, floor):
members = _as_relative_strings(derive())
assert len(members) >= floor, (
f"{label} derived only {len(members)} members, below its floor of {floor} — the derivation "
"is broken, not the repo, and every set comparison built on it is vacuous."
)
# ------------------------------------------------------------------------------------------------
# PROOF 1 — the primitive, executed against a real git repo rather than described
# ------------------------------------------------------------------------------------------------
def test_the_primitive_REALLY_excludes_an_untracked_file(tmp_path, monkeypatch):
"""A tracked and an untracked file, identical in name shape and both on disk. Only one is in.
Run rather than reasoned about. This class of defect is produced by arguments about what a
traversal WOULD enumerate, and such arguments are locally convincing whether or not they are
right; only executing the traversal distinguishes the two.
"""
repo = tmp_path / "repo"
(repo / ".claude" / "hooks").mkdir(parents=True)
(repo / ".claude" / "hooks" / "committed.sh").write_text("#!/bin/sh\n")
(repo / ".claude" / "hooks" / "untracked.sh").write_text("#!/bin/sh\n")
def git(*args):
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True)
git("init", "-q")
git("config", "user.email", "guard@example.invalid")
git("config", "user.name", "guard")
git("add", ".claude/hooks/committed.sh")
git("commit", "-qm", "one tracked hook")
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
found = tracked_files.tracked_children(".claude/hooks", ("*.sh",))
assert (repo / ".claude" / "hooks" / "untracked.sh").is_file(), (
"the untracked file must still be on disk, or this proves nothing about the index winning over the filesystem"
)
assert found == {".claude/hooks/committed.sh"}, (
f"the derivation returned {sorted(found)}. A filesystem walk returns both files here; only "
"the index distinguishes them, and that difference is the entire point of ersatztv#806."
)
def test_the_primitive_does_not_recurse_into_an_untracked_subdirectory(tmp_path, monkeypatch):
"""`.husky/_/` in miniature — the shape that made #778 red on every checkout.
Even a TRACKED nested file must stay out: `tracked_children` is direct-children-only by design,
and recursion is what dragged the shims in. Proving it with a tracked file makes the assertion
about the traversal rather than about the index, so the two properties cannot mask each other.
"""
repo = tmp_path / "repo"
(repo / ".husky" / "_").mkdir(parents=True)
(repo / ".husky" / "pre-commit").write_text("#!/bin/sh\n")
(repo / ".husky" / "_" / "husky.sh").write_text("#!/bin/sh\n")
def git(*args):
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True)
git("init", "-q")
git("config", "user.email", "guard@example.invalid")
git("config", "user.name", "guard")
git("add", "-A")
git("commit", "-qm", "husky plus a nested shim, both tracked")
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
assert tracked_files.tracked_children(".husky", ("*",)) == {".husky/pre-commit"}, (
"a nested file entered a flat population — this is the `.husky/_/` shape, and it was red on "
"every developer checkout the last time it shipped"
)
def test_an_empty_index_FAILS_LOUDLY_rather_than_reporting_an_empty_population(tmp_path, monkeypatch):
"""The floor under every floor. A silent empty population is how a completeness guard reports
total coverage having examined nothing, which is the failure mode this repo has shipped twice
(#631, #751)."""
repo = tmp_path / "repo"
repo.mkdir()
subprocess.run(["git", "-C", str(repo), "init", "-q"], check=True, capture_output=True)
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
with pytest.raises(AssertionError, match="reported nothing"):
tracked_files.tracked_children(".claude/hooks", ("*.sh",))
# ------------------------------------------------------------------------------------------------
# PROOF 2 — the property, over the real guards
# ------------------------------------------------------------------------------------------------
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
def test_no_derivation_admits_an_untracked_file(label, derive):
"""Narrow the index, leave the disk alone, and require the member to disappear — for EVERY
member, one at a time.
EXHAUSTIVE RATHER THAN ONE VICTIM, and the difference is not thoroughness for its own sake.
`derived_guard_files` unions four sources; one victim is always drawn from whichever sorts
first, so a mutant putting only the third source back on a filesystem walk passes while the
proof reports on all four. A sample cannot see the source it did not draw from this file
applying `testing.guard-derives-population-from-source` to itself.
The floors cannot substitute, and the numbers say why. Reproduce with:
PYTHONPATH=. python3 -c "from scripts.tests import test_guard_inventory as g; \
print(len(g.derived_guard_files()))"
61 members on 2026-08-22; suppressing a single contributor leaves 39 (`scripts/tests`), 48
(hooks) or 58 (husky), all far above the anti-vacuity floor of 25. The figures move whenever a
guard is added they were 60/39/47/57 one commit earlier so read them as an illustration of
the GAP, not as values to assert against. A floor tight enough to catch a lost source
would go red every time a guard is legitimately deleted, which is the wrong instrument.
Its own `monkeypatch` context, never the shared fixture instance: the function-scoped fixture is
the same object `conftest.py`'s autouse `isolate_hook_fire_log` patched, so calling `undo()` on
it here also unsets `ETV_HOOK_FIRE_LOG_DIR` and silently re-points a later hook-driving test at
the PRODUCTION log the #776 isolation disarmed from inside the file that argues for proofs.
WHAT REMOVAL CANNOT SEE, so it is not read as more than it is: a source contributing ONLY
untracked members has nothing here to remove, and #778's defect was exactly that shape (an
`rglob` over `.husky/_/` adds 17 untracked members and removes none). That direction is
`test_no_derivation_ENUMERATES_the_filesystem` below; the two are complements, not duplicates.
"""
before = _as_relative_strings(derive())
assert before, f"{label} derived nothing; there is no victim to remove"
real = tracked_files._git_ls_files()
survivors = []
absent = []
with pytest.MonkeyPatch.context() as m:
for victim in sorted(before):
if not (REPO_ROOT / victim).is_file():
absent.append(victim)
continue
m.setattr(tracked_files, "_git_ls_files", lambda v=victim: [p for p in real if p != v])
if victim in _as_relative_strings(derive()):
survivors.append(victim)
assert not absent, (
f"{label} contains {absent}, which git tracks but are not on disk. The proof below asserts "
"that the INDEX decides while the file is still present; it cannot mean that for a member "
"that is missing, so this is reported rather than skipped."
)
assert not survivors, (
f"{label} still contains {survivors} after git stopped tracking them. Every one of those is "
"still on disk, so the derivation is reading the filesystem for that member and untracked "
"build output can redden it on a developer checkout while CI stays green (ersatztv#778)."
)
# The directory-listing APIs a Python file population is realistically written with. NOT every way a
# process can list a directory — `subprocess.run(["ls"])`, a module-level alias captured before the
# patch, and any C-level call all walk straight past this, all three verified by cold review. That
# bounds what the check below can claim, and the docstring says so rather than implying a sandbox.
# Reading a file stays allowed: `derived_guard_files` must read workflow bodies.
_ENUMERATORS = (
(Path, "glob"),
(Path, "rglob"),
(Path, "iterdir"),
(Path, "walk"),
(os, "listdir"),
(os, "walk"),
(os, "scandir"),
(glob, "glob"),
(glob, "iglob"),
)
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
def test_no_derivation_ENUMERATES_the_filesystem(label, derive):
"""The ADD direction, without arranging any state: a derivation may READ files, but it may not
LIST a directory while it runs.
WHY THE ADD DIRECTION NEEDS ITS OWN TEST. Removing members from the index cannot see a source
that contributes ONLY untracked members: it adds and never takes away, so nothing of its is
available to remove. `.husky/_/` is that shape exactly 17 shims `npm ci` writes, none ever in
the index and it is the shape #778 shipped. Scope the claim precisely: a source appending
`(REPO_ROOT / ".husky" / "_").rglob("*")` leaves the removal proof green ON A MACHINE WHERE THAT
DIRECTORY IS ABSENT, which is the `script-tests` checkout. Where the shims exist, the appended
members are present and removal reddens too. The blind spot is an append-only source that yields
nothing HERE which is exactly the CI shape, and exactly where a guard going quiet matters.
WHY THE PROPERTY IS "DOES NOT ENUMERATE" RATHER THAN "DOES NOT RETURN AN UNTRACKED FILE". Both
obvious formulations of the latter are machine-dependent, which is the very fault #806 exists to
remove:
* creating real probe files in the checkout and requiring they not enter. That needs a
`test_*.py` probe to reach the `scripts/tests` scope a file pytest COLLECTS mid-session
and its parametrised names collide across `-n auto` workers and concurrent sessions,
`finally` does not survive SIGKILL, and a concurrent `git add -A` can stage one. Defects in
the test, not in the thing tested.
* neutralising the shared primitive and requiring the derivation to go empty. That misses the
`.husky/_/` source on any machine where `.husky/_/` does not exist which is every CI
checkout of `script-tests`, which never runs `npm ci`. Green where it runs, red only on a
laptop: the inverted asymmetry again, inside the proof written to abolish it.
Watching for the CALL needs no arranged state: an `rglob` issued while deriving is caught even
where the directory it walks is empty, because the evidence is the call rather than what it
returned. Reading is untouched, so a derivation may still parse the workflow bodies it scrapes
for referenced scripts.
WHAT IT DOES NOT COVER. The boundary is not "synchronous", which is what two earlier drafts
said and what measurement disproved a thread that outlives the `derive()` call but finishes
while its result is being drained IS caught, as is a `__del__` firing during that drain. The
boundary is mechanical rather than temporal: a call to one of the SPIES is observed, wherever and
whenever it happens in this process before the assertion below. "While the patch is active"
under-claims it the spy appends to a list that outlives the patch, so a reference captured
during the window and invoked after it still records. What decides observation is whether the
call goes through a spy, not when. The result is drained while the patch is installed, so a lazy
generator is reached.
NOT REACHED, because no spy was ever installed on that path. Enumeration HOISTED TO MODULE SCOPE
runs at import, before this test exists the likeliest instance rather than a contrivance, since
`test_ci_image_pin_population.py` already precomputes `_DOC` that way as does an `atexit` hook,
and a cached property warmed by the baseline call below. `from os import listdir` binds the real
function before any patch; `from os import walk` IS caught (it routes through the patched
`os.scandir`) and so is `from glob import glob` (through the patched `glob.iglob`). And anything
listing in ANOTHER PROCESS a deliberate `subprocess.run(["ls"])`, or a forked child. This is a
regression guard against the shapes that arrive by accident, not a sandbox.
It cuts the other way too: any spy call at all reddens this test, so unrelated background thread
activity touching a patched name during the window would too. Nothing in this suite does that
today, and the report names the call, so a false red would be diagnosable rather than
mysterious.
It also cannot see a derivation that admits a HARDCODED path without listing anything
(`if (REPO_ROOT / "x.sh").exists(): add`) listing is the commonest way to discover an untracked
member, not the only one. The removal proof above catches that shape, and catches memoisation,
which this one cannot: the baseline call below warms any cache outside the patch. The two are
complements.
"""
assert derive(), f"{label} derived nothing; this proof needs a baseline"
calls: list[str] = []
def _spy(what):
# RECORDS and returns empty rather than raising. Raising made the assertion "did an
# exception reach us", which a derivation defeats by catching it: a `try: ... except
# Exception: return set()` around an `rglob` enumerated the filesystem and this test passed,
# measured. The evidence is the CALL, so the call is what is asserted on.
def spy(*_args, **_kwargs):
calls.append(what)
# Iterator AND context manager: `os.scandir` is used as `with os.scandir(...) as it`,
# and a bare iterator made the failure report carry a test-induced TypeError about the
# context manager protocol alongside the real finding. The assertion fired correctly
# either way, but a report that blames the harness invites doubting the finding.
return _EmptyScan()
return spy
failure = None
with pytest.MonkeyPatch.context() as m:
for owner, name in _ENUMERATORS:
m.setattr(owner, name, _spy(f"{getattr(owner, '__name__', owner)}.{name}"), raising=False)
try:
# DRAINED inside the context, never `derive()` discarded. A derivation returning a lazy
# generator does its work when the caller drains it, so discarding the result moved the
# whole walk outside the patch: a generator yielding the index population and then
# appending `.husky/_` passed here and admitted 17 untracked shims on a checkout where
# that directory exists. `_as_relative_strings` is what drains it — it iterates into a
# set — so the call must stay here rather than being hoisted out or wrapped in something
# lazier.
_as_relative_strings(derive())
except BaseException as exc: # re-raised below, after the evidence has been judged
failure = exc
assert not calls, (
f"{label} enumerated the filesystem via {sorted(set(calls))} while deriving its population. "
"Directory listings report build output, generated shims and editor droppings, and differ "
"between the CI checkout and a developer's, so the member set stops being a property of the "
f"repo (ersatztv#778, #806)." + (f" It also raised: {failure!r}" if failure is not None else "")
)
if failure is not None:
raise failure
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
def test_every_derived_member_is_tracked(label, derive):
"""The same property as an invariant over the real tree, which is the form that catches a
refactor going back to a filesystem walk without also touching this file."""
tracked = set(tracked_files._git_ls_files())
stray = sorted(m for m in _as_relative_strings(derive()) if m not in tracked)
assert not stray, f"{label} contains untracked path(s): {stray}"
# ------------------------------------------------------------------------------------------------
# THE SCOPE MIRROR ABOVE IS ITSELF CHECKED
# ------------------------------------------------------------------------------------------------
def _modules_importing_the_helper() -> set[str]:
"""Which `scripts/tests/test_*.py` import the shared derivation, by PARSING them.
`ast` rather than a substring scan, and the distinction is the point rather than tidiness. A
substring scan over source both EVADES and FALSELY FIRES here: `import scripts.tests.
tracked_files as tf` and `from scripts.tests import tracked_files as tf` escape a scan for
`"tracked_files import"`, while a comment merely citing `scripts/tests/tracked_files.py` matches
a scan for `"tracked_files."` and would redden a correct tree over prose.
`docs/decisions/records/testing/guard-derives-population-from-source.md` records why patching
such a predicate does not converge: it is not a parser. Python ships the parser, and a comment
is not a node at all.
WHAT THE PARSE DOES NOT REACH, stated rather than implied by the word "parsing": STATIC import
statements naming the helper. `importlib.import_module("scripts.tests.tracked_files")`, a
re-export through `scripts/tests/__init__.py`, and `from scripts.tests import *` are invisible,
verified by executing each. (`from scripts.tests.tracked_files import *` IS seen it names the
module.) Those sit inside the same residual as a module that derives a population without the
helper at all the residual named below and no mechanical check closes it.
The FILE LIST is a filesystem walk on purpose, and it is not the defect this file forbids: it is
a superset check over what pytest itself collects, so an untracked stray `test_x.py` here makes
the guard MORE demanding, never blind. Using the index would let an unstaged new guard escape
registration, which is the wrong direction for a check about coverage.
"""
found = set()
for path in sorted(TESTS_DIR.glob("test_*.py")):
for node in ast.walk(ast.parse(path.read_text())):
# RESOLVE the name to an absolute module and compare exactly. Testing `base ==
# "tracked_files"` handled `from .tracked_files import x` but silently missed
# `from ..tests.tracked_files import x`, which resolves to the same helper — a false
# NEGATIVE, the direction that lets a module adopt the helper and escape registration.
# A suffix match instead over-accepts `from unrelated.package import tracked_files`,
# reddening a correct tree over a module this repo does not own. Resolution is the only
# form with neither failure: these files live in `scripts.tests`, so level 1 resolves to
# `scripts.tests` and level 2 to `scripts` — see the guard below for why there is no
# level 3.
if isinstance(node, ast.Import):
hit = any(a.name == _HELPER for a in node.names)
elif isinstance(node, ast.ImportFrom):
if node.level > len(_PACKAGE):
# Beyond the top-level package: Python raises ImportError for this, so it cannot
# be an import of the helper. Guarded explicitly because `_PACKAGE[:negative]`
# silently WRAPS — level 4 produced the same prefix as level 2 — which pinned an
# unimportable form as a valid detection. Valid levels here are exactly 1 and 2:
# 1 resolves to `scripts.tests`, 2 to `scripts`, and 3 or more is beyond the
# top-level package, which Python refuses.
continue
prefix = _PACKAGE[: len(_PACKAGE) - (node.level - 1)] if node.level else []
base_parts = prefix + ([node.module] if node.module else [])
base = ".".join(base_parts)
hit = base == _HELPER or any(f"{base}.{a.name}" == _HELPER for a in node.names)
else:
continue
if hit:
found.add(path.name)
break
return found
# (source, should the matcher see it). Every row is a form that has actually been mis-classified; the
# table is here so the next edit to `_modules_importing_the_helper` cannot re-open one silently.
# A false NEGATIVE lets a module adopt the helper and escape registration; a false POSITIVE reddens
# a correct tree over a module this repo does not own. Both directions are pinned.
_IMPORT_FORMS = (
("import scripts.tests.tracked_files as tf", True),
("from scripts.tests import tracked_files as tf", True),
("from scripts.tests.tracked_files import tracked_paths", True),
("from scripts.tests.tracked_files import *", True),
("from . import tracked_files", True),
("from .tracked_files import tracked_paths", True),
("from ..tests.tracked_files import tracked_paths", True),
# Beyond the top-level package from `scripts.tests`: Python raises ImportError, so there is
# nothing to detect. Pinned False so the negative-slicing wrap that once made it look
# detectable cannot come back.
("from ...scripts.tests.tracked_files import tracked_paths", False),
# Witnesses the negative-slice wrap specifically: without the level guard this one
# resolves through `scripts` and matches.
("from ....tests.tracked_files import tracked_paths", False),
("def f():\n from scripts.tests import tracked_files\n return tracked_files", True),
("from unrelated.package import tracked_files", False),
("from ..something import tracked_files", False),
("from .something import tracked_files", False),
("from ..other.tracked_files import x", False),
("# see scripts/tests/tracked_files.py for the rationale\nimport re", False),
# Pins PARSER VISIBILITY, not importability: executing this line really does import the helper.
# The row records that a static parse cannot see it — a known gap, pinned so it is not a
# surprise — and closing it would mean updating this row, which is the intended friction.
("import importlib\nimportlib.import_module('scripts.tests.tracked_files')", False),
)
@pytest.mark.parametrize(("source", "expected"), _IMPORT_FORMS, ids=[s.splitlines()[0][:48] for s, _ in _IMPORT_FORMS])
def test_the_import_matcher_classifies_every_reviewed_form(source, expected, monkeypatch):
class _Fake:
name = "test_probe.py"
def read_text(self):
return source
def __lt__(self, other):
return True
class _Dir:
def glob(self, _pattern):
return [_Fake()]
monkeypatch.setattr(sys.modules[__name__], "TESTS_DIR", _Dir())
seen = "test_probe.py" in _modules_importing_the_helper()
assert seen is expected, f"the import matcher {'missed' if expected else 'falsely matched'} this form:\n{source}"
def test_every_index_derived_module_is_registered():
"""`DERIVATIONS` is a hand-written mirror, so it gets an equality check rather than a promise.
Without this, adding another index-derived guard and forgetting to register it leaves that guard
unproven while this file reads as covering them all a completeness claim standing behind a
hand-maintained list, which is the defect one altitude up (#773 Family C, and the shape that put
`MARKED_JOBS` in the record as a residual gap).
THE SCOPE THIS CANNOT SEE, stated because a check described as complete stops being re-examined:
it detects modules that IMPORT the shared helper. A module deriving a file population some other
way shelling out to `git ls-files` itself, or going back to `Path.rglob` is invisible to it,
and no mechanical check can close that. `test_guard_inventory.py`'s own header argues the same
point about flagging filter-shaped guards by token, and #774 concluded there that the honest
answer is no. What is mechanised here is the case that actually recurs: someone adopts the
helper and forgets this file.
Registration is MODULE-level, not derivation-level, so a second population added inside an
already-registered module is covered only if it is registered too.
`POPULATION_EXEMPT` is the opt-out, and it lives HERE rather than as a marker comment in the
exempt file because the two directions are not symmetric: a false import-match only reddens,
while a false EXEMPTION is silent. A marker a file grants itself by containing a token is
trippable from that file's prose — this file's own error message names the token so it would
be a one-line silent kill switch, the shape `test_ci_release_path_scan_job.py` argues against
for its recursion fence. Listing exemptions beside the registrations makes adding one a visible
edit here.
WHAT NO ASSERTION CAN DECIDE, dated so it is re-examined rather than assumed: whether an
exemption is still WARRANTED. A stale key and an unexplained one are both caught below, but an
exempt module that later grows a real derived population stays uncovered and silent. Reviewed
2026-08-22 the single entry uses the helper only to assemble a tmp fixture copy and asserts
nothing about membership.
"""
exempt = {Path(__file__).name} | set(POPULATION_EXEMPT)
importers = _modules_importing_the_helper()
registered = {label.split(".", 1)[0] + ".py" for label, _, _ in DERIVATIONS}
# ANTI-VACUITY, and only that. A broken parse is caught loudly by `phantom` below — every
# registered module would go missing at once — so this is the cheaper, more specific signal, not
# the thing standing between a broken parse and a green run.
assert len(importers) >= len(DERIVATIONS), (
f"the import parse found only {sorted(importers)}, fewer modules than DERIVATIONS registers "
f"({sorted(registered)}) — the parse has broken."
)
stale = sorted(name for name in POPULATION_EXEMPT if name not in importers)
assert not stale, (
f"POPULATION_EXEMPT lists {stale}, which the parser no longer sees importing the shared "
"helper (renamed, "
"deleted, or the import removed). A stale exemption is worse than none: if the filename is "
"ever reused, the new module is exempt from birth without anyone deciding that."
)
thin = sorted(name for name, why in POPULATION_EXEMPT.items() if not str(why).strip())
assert not thin, f"POPULATION_EXEMPT entries with no stated reason: {thin}"
unregistered = sorted(importers - registered - exempt)
assert not unregistered, (
f"{unregistered} import the shared index derivation but are not in DERIVATIONS, so neither "
"proof in this file covers them. Add a named derivation function and register it, or add "
"the module to POPULATION_EXEMPT in this file if it imports the helper without deriving a "
"population."
)
phantom = sorted(registered - importers)
assert not phantom, (
f"DERIVATIONS registers {phantom}, which no longer import the shared helper. A row for "
"a derivation that is not there reads as coverage and is not."
)
+25 -8
View File
@@ -2,9 +2,12 @@
Two separable claims, and conflating them is how instrumentation ships as a regression:
1. COVERAGE every hook script records its own execution. The population is DERIVED from
`.claude/hooks/*.sh`, never listed, per `testing.guard-derives-population-from-source`, so a
hook added tomorrow is uninstrumented-and-red rather than silently unobserved.
1. COVERAGE every hook script records its own execution. The population is DERIVED from the
tracked `.claude/hooks/*.sh`, never listed, per `testing.guard-derives-population-from-source`,
so a hook staged tomorrow is uninstrumented-and-red rather than silently unobserved. From
the git index rather than a filesystem walk since ersatztv#806 — an untracked file on one
machine is not part of the repo, and a guard whose population differs per checkout is one
nobody trusts.
2. TRANSPARENCY the wrapper is invisible to the harness. It slurps stdin and replays it, and it
diverts stdout and replays it, which puts it directly in the path of the most load-bearing
@@ -34,7 +37,13 @@ from pathlib import Path
import pytest
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
# Directory + patterns resolved against the GIT INDEX, not `Path.glob` (ersatztv#806) — see
# `scripts/tests/tracked_files.py`. `HOOKS_DIR` survives only for error messages.
HOOKS = (".claude/hooks", ("*.sh",))
HUSKY = (".husky", ("*",))
HOOKS_DIR = REPO_ROOT / ".claude" / "hooks"
SINK = REPO_ROOT / "scripts" / "hook-fire-log.sh"
@@ -46,8 +55,15 @@ _BEGINS = re.compile(r"^etv_hook_fire_begin (\S+) .*\|\| true$", re.M)
def hook_scripts() -> list[Path]:
"""THE POPULATION, from the filesystem. Never a list."""
return sorted(HOOKS_DIR.glob("*.sh"))
"""THE POPULATION, from the GIT INDEX. Never a list, and never the filesystem (ersatztv#806).
The filesystem is not an authoritative source: an untracked `.sh` dropped in `.claude/hooks/`
a scratch copy, a half-written hook used to enter this population and be demanded to carry
instrumentation, reddening the suite on that checkout while CI, which never sees the file, stayed
green. That is #778's third shape, and a guard that fails everywhere except where it runs trains
its readers to ignore it.
"""
return tracked_paths(*HOOKS)
def expected_mode(name: str) -> str:
@@ -64,8 +80,8 @@ def expected_mode(name: str) -> str:
"""
if name in (REPO_ROOT / ".claude" / "settings.json").read_text():
return "capture"
for husky in (REPO_ROOT / ".husky").iterdir():
if husky.is_file() and name in husky.read_text():
for husky in tracked_paths(*HUSKY):
if name in husky.read_text():
return "stream"
return "capture"
@@ -150,7 +166,8 @@ def strip_instrumentation(text: str) -> str:
def test_the_population_is_not_empty():
hooks = hook_scripts()
assert len(hooks) >= 10, (
f"only found {len(hooks)} hook scripts under {HOOKS_DIR} — the glob has stopped matching, "
f"only found {len(hooks)} hook scripts tracked under {HOOKS_DIR} — the derivation has "
"stopped matching, "
"so every coverage assertion in this file is vacuous."
)
assert SINK.exists(), "the shared sink is missing; the instrumentation cannot work"
+8 -6
View File
@@ -30,6 +30,8 @@ from pathlib import Path
import pytest
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
SCRIPT = REPO_ROOT / "scripts" / "pr-changed-files.sh"
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
@@ -842,9 +844,7 @@ def test_no_OTHER_workflow_writes_the_review_verdict_status():
plainly so nobody reads a green here as "only one writer is possible".
"""
offenders = []
# `*.y*ml`, not `*.yml` — a workflow added as `.yaml` is just as executable and would otherwise
# be silently unscanned. Latent today (no `.yaml` exists), which is exactly when it is cheap.
for wf in sorted((REPO_ROOT / ".gitea" / "workflows").glob("*.y*ml")):
for wf in _workflow_files():
if wf.name == WORKFLOW.name:
continue
# CODE lines only — `docs`-style commentary in a workflow header legitimately discusses the
@@ -2505,9 +2505,11 @@ def _yaml_string_scalars(path: Path):
def _workflow_files():
# `*.y*ml` for the reason test_no_OTHER_workflow_writes_the_review_verdict_status gives: a
# workflow added as `.yaml` is just as executable and would otherwise go unscanned.
return sorted((REPO_ROOT / ".gitea" / "workflows").glob("*.y*ml"))
# `*.yml` AND `*.yaml`: a workflow added as `.yaml` is just as executable and would otherwise go
# unscanned. From the GIT INDEX rather than a directory listing (ersatztv#806) — a stray
# untracked `.yaml` in `.gitea/workflows/` is not part of the repo, and enumerating it made
# these two completeness claims red on that checkout and green in CI, which is #778's shape.
return tracked_paths(".gitea/workflows", ("*.yml", "*.yaml"))
def test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body():
+20 -11
View File
@@ -34,9 +34,10 @@ from __future__ import annotations
import fnmatch
import re
import subprocess
from pathlib import Path
from scripts.tests import tracked_files
REPO_ROOT = Path(__file__).resolve().parents[2]
INVENTORY = REPO_ROOT / "docs" / "remote-state-inventory.md"
@@ -86,8 +87,9 @@ SCOPE = (
# The lesson each time was the same one this repo already wrote down — derive the population from an
# AUTHORITATIVE source — and the filesystem is not one. It reports build output, editor droppings and
# anything else that happens to be on disk, and it varies per machine. The repo's index is
# authoritative, versioned, identical for every checkout and for CI, and it excludes untracked
# generated files by construction rather than by an exclusion list that must be maintained.
# authoritative: it holds the same set of files every checkout receives from a clone, and it
# excludes untracked generated files by construction rather than by an exclusion list that must be
# maintained.
#
# `scripts/tests/` is still excluded explicitly, because those files ARE tracked. That exclusion is a
# scope decision, reviewable in one line: they run only under pytest and authorize nothing. Their
@@ -104,15 +106,22 @@ _ROW = re.compile(r"^\|\s*`([^`]+?)`[^|]*\|\s*`([^`]+)`\s*\|", re.M)
def _tracked_files() -> list[str]:
"""Every file git tracks, as repo-relative posix paths.
Fails LOUDLY rather than returning nothing: an empty population would make every completeness
assertion below pass vacuously, which is the exact failure this guard exists to prevent.
A thin wrapper over the SHARED derivation rather than a second copy of it (ersatztv#806): one
definition of the rule, not two, which is detector C dedup by construction applied to the
file that first stated the rule. The wrapper survives because this module patches
`_tracked_files` by name in its own proofs, and because its scope is RECURSIVE over `scripts/`
where `tracked_children` is deliberately flat.
It therefore carries BOTH its own proofs and a row in that file's `DERIVATIONS`, which is not
the duplication that masks: they cut at different seams the in-file pair patches
`_tracked_files` (this wrapper), the shared pair patches `_git_ls_files` (the subprocess) and
each was witnessed red independently, so neither can hide the other's total failure.
Fails LOUDLY rather than returning nothing, now including git's own stderr: an empty population
would make every completeness assertion below pass vacuously, which is the exact failure this
guard exists to prevent.
"""
proc = subprocess.run(
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
capture_output=True,
check=True,
)
return [p for p in proc.stdout.decode().split("\0") if p]
return tracked_files._git_ls_files()
def derived_population() -> set[str]:
+120
View File
@@ -0,0 +1,120 @@
"""The authoritative population for a guard whose members are FILES: the git index (ersatztv#806).
`testing.guard-derives-population-from-source` (#774) says a completeness guard derives its
population from a machine-readable authoritative source, and its worked examples are an enum and the
generated OpenAPI document. It does not say what to do when the population is *files*, and every
guard in this repo answered that with a filesystem walk. **A filesystem walk is not an authoritative
source.** It reports build output, editor droppings and whatever else happens to be on disk, and it
differs per machine, so a guard derived from it asserts a different population in CI than on the
laptop of the person it is supposed to stop.
#778 got that wrong three times in one PR, each time with an argument for why the traversal
sufficed, and the third shape is the one that motivates this module:
* a **content filter** on outbound-network tokens that omitted `git fetch`, making a hook that
fetches `origin/main` and derives a push decision structurally invisible;
* a **non-recursive `Path.glob`**, missing four nested files, one of which calls a live ErsatzTV
API and acts on the reply;
* **`Path.rglob`**, which then enumerated `.husky/_/` 17 husky shims generated by `npm ci` via
`web/package.json`'s `prepare` script, gitignored (`.husky/_/.gitignore` is `*`) and untracked.
That made the guard **RED on every developer checkout and GREEN in CI**, whose `script-tests`
job checks out and pip-installs but never runs `npm ci`. A guard that fails everywhere except
where it runs teaches its readers to ignore it.
The index holds the same set of files every checkout receives from a clone, and excludes untracked
generated files **by construction** rather than by an exclusion list somebody has to maintain and
keep correct. It is not immutable and it is per-worktree; the claim is not that it never changes,
but that it changes only through a deliberate git operation staging, a checkout, a reset, a
merge whereas the disk changes whenever a build runs.
Note what that buys over `.gitignore`-awareness: `.husky/_/` happens to carry its own `.gitignore`,
but a stray untracked `foo.sh` in `.claude/hooks/` carries nothing, and only the index knows it is
not part of the repo.
**This is not "replace every glob".** The question per guard is whether it makes a COMPLETENESS
claim over tracked files. If it does, the index is the authoritative source. If it does not a
fixture copying files into a tmp tree, a walk selecting the SUBJECT of a per-member property say
so in the guard and leave it, per the boundary #774 already draws between scope and population.
`git ls-files` lists INDEX entries, so a file deleted in the working tree but not yet staged is
still reported. That is deliberate: callers that read member contents assert existence with their
own message rather than filtering, because filtering is what makes a missing member unrepresentable.
"""
from __future__ import annotations
import fnmatch
import subprocess
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
def _git_ls_files() -> list[str]:
"""Every path git tracks, as repo-relative posix strings.
The single place this package shells out to git, so a regression proof can narrow the tracked
set once and have every derivation built on it react.
Fails LOUDLY on an empty result rather than returning it: an empty population makes every
completeness assertion downstream pass vacuously, which is the exact failure these guards exist
to prevent.
"""
proc = subprocess.run(
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
capture_output=True,
check=False,
)
# `check=True` would raise "returned non-zero exit status 128" and leave git's own diagnostic
# trapped in `e.stderr`. The two real triggers — a tree that is not a repository, and CI's
# `detected dubious ownership` — are both diagnosable ONLY from that text, and every module
# that imports this one would otherwise die with the same inscrutable line.
assert proc.returncode == 0, (
f"`git ls-files` failed in {REPO_ROOT} (exit {proc.returncode}). Every file population here "
f"comes from the index, so this is fatal rather than empty. git said:\n"
f"{proc.stderr.decode(errors='replace').strip() or '(no stderr)'}"
)
paths = [p for p in proc.stdout.decode().split("\0") if p]
assert paths, (
f"`git ls-files` reported nothing under {REPO_ROOT} — the derivation is broken, not the "
"repo. Every file population built on it would be empty and every completeness assertion "
"would pass vacuously."
)
return paths
def tracked_children(directory: str, patterns: tuple[str, ...]) -> set[str]:
"""Tracked files that are DIRECT children of `directory` and match one of `patterns`.
Direct children only, and that is the point rather than a limitation: every population here is
a flat directory (`.claude/hooks/*.sh`, `.husky/*`, `.gitea/workflows/*.yml`,
`scripts/tests/test_*.py`), and recursing is what dragged `.husky/_/` in. A guard that genuinely
needs a nested population should say so and ask for it explicitly.
Returns repo-relative posix paths, matching what the guards' inventories and error messages use.
"""
found: set[str] = set()
for path in _git_ls_files():
parent, _, name = path.rpartition("/")
if parent != directory:
continue
if any(fnmatch.fnmatch(name, pattern) for pattern in patterns):
found.add(path)
return found
def tracked_paths(directory: str, patterns: tuple[str, ...]) -> list[Path]:
"""`tracked_children` as absolute `Path`s, sorted — for guards that read member contents.
Existence is ASSERTED, never filtered: a path in the index with no file on disk means the tree
is mid-edit, and reporting that is strictly better than silently shrinking the population, which
is the defect this module exists to remove.
"""
paths = []
for rel in sorted(tracked_children(directory, patterns)):
absolute = REPO_ROOT / rel
assert absolute.is_file(), (
f"git tracks {rel} but there is no file there. The population comes from the index, so "
"a working tree mid-delete is reported rather than silently shrinking the population."
)
paths.append(absolute)
return paths