Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ded6cb197c | ||
|
|
908855644a | ||
|
|
a92df7515a | ||
|
|
10dba0892e | ||
|
|
03955675a8 | ||
|
|
b0f42f14a0 |
@@ -200,8 +200,8 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `testing.enumerating-guard-identity-not-position` | A guard that cross-checks a hand-reviewed registry against call sites discovered across the whole repo must key each entry on properties INTRINSIC to the site — file, kind, and the value source text — and never on its absolute line or column. A registry keyed on position is a function of every other file in the repo, so a branch that never touches the guard can invalidate it; and because each PR is green against its own base, that failure is structurally invisible pre-merge and lands on `main` after review and after the merge gate. Dropping the position keeps every mutation the guard exists for — a NEW site, a REMOVED site and a CHANGED value each still fail, since each changes the identity multiset — and costs exactly ONE case, which must be stated rather than implied: a SAME-IDENTITY SUBSTITUTION within one file (delete a registered site, add a different unreviewed one with the same kind and value token, net-zero count) now passes. A REPORTED failure still prints the discovered line:column, because identity and diagnostics need not share a format. Comparison stays a MULTISET count rather than set membership, so two sites in one file sharing an identity must be discovered exactly that many times and a third occurrence still fails. A SCANNER test that asserts real AST positions against FIXED inline fixtures is the opposite case and keeps its line/column identity — it has no churn, because its input does not move. | 2026-07-27 | [link](records/testing/enumerating-guard-identity-not-position.md) |
|
||||
| `testing.fix-ships-a-witnessed-red-test` | A commit claiming to fix something may carry a `Proves: <pytest selector>` trailer; when it does, `scripts/prove-fix.sh` must show that selector GREEN with the fix and RED with the code side reverted, and CI enforces it per-PR. The trailer is opt-in — an unproven commit is allowed — but a claimed proof that does not hold fails the build. | 2026-08-16 | [link](records/testing/fix-ships-a-witnessed-red-test.md) |
|
||||
| `testing.full-replace-asserts-field-list` | Any path that writes a WHOLE entity or a WHOLE child collection — a PUT-replace handler, a hand-built request object, a test comparer standing in for one — derives its field list from the authoritative type and asserts SET EQUALITY against it, rather than enumerating the fields by hand. A hand-written list is correct on the day it is written and structurally unable to report the day it stops being: the field that drifts is the one nobody wrote a line for, so no amount of care in the existing lines can reach it. The failure is silent by construction — a full replace with a field omitted returns HTTP 200 and destroys that field's value (#754 drifted from a 28-property DTO by one and cleared it; the symptom arrived hours later as missing pixels). SECOND CLAUSE, separable from the first: where a replaced child row carries state keyed to its identity — progression, ordering, an enumerator position — the handler RECONCILES BY ID rather than delete-and-reinsert, because reinsertion silently resets state a client never asked to touch (#252: a schedule PUT reset fill-group progression; #500: a dedup fix became permanent data loss because the add filter and the remove filter used different keys, so the two halves must agree on the key). Delete-and-reinsert is acceptable ONLY where no such state exists, and that emptiness is a fact about today's schema that a later feature can silently invalidate — so record it where the handler is, dated, rather than leaving it to be re-derived. The canonical worked example is `ToolCatalogTests.Every_Write_Tool_Should_Declare_Exactly_Its_OpenApi_Request_Body_Fields`, which reads the accepted fields from the generated OpenAPI document and compares both directions. | 2026-08-21 | [link](records/testing/full-replace-asserts-field-list.md) |
|
||||
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
|
||||
| `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) |
|
||||
| `testing.guard-derives-population-from-source` | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. | 2026-08-13 | [link](records/testing/guard-derives-population-from-source.md) |
|
||||
| `testing.guard-ships-with-mutation-proof` | A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD'S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631's suite was invoked by no CI job, #751's step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719's new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row's own named test required to go red. | 2026-08-13 | [link](records/testing/guard-ships-with-mutation-proof.md) |
|
||||
| `testing.hook-reports-its-own-execution` | Every script in `.claude/hooks/` sources `scripts/hook-fire-log.sh` and calls `etv_hook_fire_begin <its-own-name> <label> <capture\|stream>` as its FIRST act, before anything reads stdin. Two records are appended per invocation — a `fire` record on entry and an `exit` record carrying the exit status and the decision — to a session-scoped JSONL log. THE DECISION IS READ FROM WHAT THE HOOK ACTUALLY EMITTED, never declared by the hook author: Claude Code hooks (`capture` mode) always exit 0 and communicate by PRINTING JSON, so their stdout is diverted and replayed, and the recorded decision is parsed from those bytes; git hooks (`stream` mode) decide by EXIT CODE and their stdout is live progress text a human is watching, so it is not diverted and the decision is the status. That split is not a tuning knob — capturing a slow pre-push hook's output would hold it back until the end and read as a hang, and inferring a git hook's decision from absent JSON would put the report back into the guessing business this record exists to end. The population is DERIVED from `.claude/hooks/*.sh` by `scripts/tests/test_hook_fire_log.py`, so a new hook is uninstrumented-and-red rather than silently unobserved, and the report lists every hook that EXISTS rather than every hook that appears in the log — a report built from the log alone can only show hooks that fired, which makes the never-fired hook, the one finding worth having, invisible. THE INSTRUMENTATION MUST BE INVISIBLE TO THE HARNESS, and this is the load-bearing half: it sits in the stdin and stdout path of the most authoritative guards in the repo, so a differential test drives EVERY hook with and without it over a payload matrix and demands byte-equal stdout and equal exit status. It fails OPEN in exactly one direction — if the log cannot be written the hook behaves exactly as before — because observability that breaks a guard is worse than the blindness it replaces. Two mechanical traps are pinned by tests rather than left to care: stdout must be replayed from the FILE, since `out=$(cat f)` strips trailing newlines and delivers a guard's JSON one byte short with no parser anywhere to complain; and stdin must never be slurped when it is a TTY, because an interactive `git commit` hands its hooks a terminal and `cat` would block forever, hanging the commit the instrumentation was added to observe. | 2026-08-14 | [link](records/testing/hook-reports-its-own-execution.md) |
|
||||
| `testing.live-e2e-prepush-timing` | Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab. | 2026-07-21 | [link](records/testing/live-e2e-prepush-timing.md) |
|
||||
| `testing.mutation-claims-are-executed` | A `MUTATION` row in `docs/guard-inventory.md` is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`, and `scripts/tests/test_mutation_harness.py` applies that mutation to an isolated copy of the repository on every run and requires the row's OWN named test to go red. The manifest and the MUTATION rows are compared for SET EQUALITY in both directions, so a row cannot claim the grade without a mutation and a mutation cannot outlive the grade it justifies. EXIT STATUS IS NOT THE VERDICT: each entry also declares the DIAGNOSTIC its red must carry, matched against pytest's exception output alone, because pytest reports a crashing test exactly as it reports a detecting one and a red for an unrelated reason is evidence about nothing. WHERE THE GUARD IS ITSELF A TEST, `target` may differ from `guard` and the exact-once check applies to the declared TARGET. Two shapes are admissible and the choice is not free. Where the guard's assertion IS the check — a completeness comparison against a Markdown inventory — the mutation goes into the guarded ARTIFACT, per `testing.guard-ships-with-mutation-proof`'s checker-guard exception, because mutating such a checker's own POPULATION demonstrates a false POSITIVE while proving nothing about the detection the row claims. Where the guard is a test module wrapping a separately mutable DETECTOR or helper, the clause may be in that detector, since disarming it is a real clause disarm and the module's own assertion is what notices. THE MUTATION IS DECLARED, NEVER INFERRED: a harness that guessed which clause of a 90-line hook is the guard would manufacture the confident-but-empty coverage this exists to prevent, which is why `testing.guard-ships-with-mutation-proof` rejected a generic runner. Where a proof test already names its clause in source, the manifest reuses THAT string, so a retarget in either place is caught by the other. COARSENESS IS RECORDED, NOT HIDDEN: each entry is graded `CLAUSE` or `DETECTOR`, and a `DETECTOR` entry — one whose detector accumulates faults from independent arms, so disarming any single arm leaves its proof test green — must CARRY the finer mutation that survived, which is re-run every time and required to keep surviving. Guards that are not graded `MUTATION` each carry a STATED reason in that same manifest, keyed on the guard and compared for SET EQUALITY against the inventory's `GUARD` rows in both directions — so a new guard cannot arrive without someone writing what a proof would need, and a reason cannot outlive the row it is about. Keying the reason on the row's GRADE instead is tautological (a new guard inherits one and nobody looks at it) and a pinned COUNT moves only on net change; both were tried and are rejected. The sandbox is a real git repository built from `git ls-files` with working-tree content, never a filesystem walk. | 2026-08-22 | [link](records/testing/mutation-claims-are-executed.md) |
|
||||
|
||||
@@ -5,8 +5,8 @@ status: active
|
||||
since: '2026-08-13'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`.'
|
||||
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · issues: #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
|
||||
rule: 'A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down.'
|
||||
signals: 'guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403'
|
||||
mechanics: 'Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.'
|
||||
---
|
||||
|
||||
@@ -78,6 +78,103 @@ the answer is to create one, never to approximate it with a predicate over text.
|
||||
accompanying prose a reviewer can falsify each round is worse than no guard, because — by this
|
||||
record's own argument — a guard described as sound stops being re-examined.
|
||||
|
||||
**When the population is FILES, the authoritative source is the git index (#806).** The worked
|
||||
examples above are an enum and a generated document, both unambiguously authoritative, and the
|
||||
record was silent on the commonest population in this repo's own guards: files in a directory. Every
|
||||
one of them answered with a filesystem walk, and #778 measured what that costs by getting the same
|
||||
population wrong three times in one PR — a content filter that omitted `git fetch`, a non-recursive
|
||||
`glob` that missed four nested files, and finally `rglob`, which enumerated `.husky/_/`: 17 husky
|
||||
shims generated by `npm ci`, gitignored and untracked. That last one made the guard **red on every
|
||||
developer checkout and green in CI**, whose `script-tests` job pip-installs but never runs `npm ci`.
|
||||
|
||||
The reason a walk keeps losing is not that each traversal was written carelessly; two of the three
|
||||
were the obvious correction to the one before. It is that the disk answers a question about the
|
||||
MACHINE and the guard is asking one about the REPO. Those coincide often enough for a walk to look
|
||||
right and diverge exactly where generated output lands, which is to say wherever the tooling is
|
||||
installed and nowhere else. The index is the repo's own statement of what it contains — the same
|
||||
set of files every checkout receives from a clone, and excluding untracked files by construction
|
||||
rather than by an exclusion list somebody maintains. It is not immutable and it is per-worktree; the
|
||||
claim is not that it never changes, but that it changes only through a deliberate git operation — staging, a checkout, a
|
||||
reset, a merge — whereas the disk changes whenever a build runs. Note what that buys over `.gitignore`-awareness: `.husky/_/`
|
||||
happens to carry its own `.gitignore`, but a stray `foo.sh` in `.claude/hooks/` carries nothing, and
|
||||
only the index knows it is not part of the repo.
|
||||
|
||||
The direction of the failure is worth naming, because it inverts the usual worry about a guard.
|
||||
Under-enumeration hides a defect; this over-enumerated, and reddened correct trees. A guard that
|
||||
fails everywhere except where it runs is not a cautious guard, it is a guard nobody reads — and it
|
||||
had done that to the artifact whose entire thesis is population correctness.
|
||||
|
||||
**This did not become "replace every glob", and the boundary is the same one drawn above.** The
|
||||
question per guard is whether it makes a completeness claim over TRACKED files. `_repo_copy` in
|
||||
`scripts/tests/test_ci_release_path_scan_job.py` assembles a fixture and asserts nothing about
|
||||
which files it found; it takes its file list from the index for HERMETICITY, which is a different
|
||||
reason, and its docstring distinguishes the two.
|
||||
`scripts/tests/test_ci_dropped_step_guard.py` has no filesystem population at all — it reads the
|
||||
parsed workflow. Converting either would have been a change with no defect behind it, which spends
|
||||
the credibility this rule needs when it does bite. The per-guard verdicts, including the two
|
||||
no-change ones and the decisions corpus recorded as unexamined rather than cleared, are tabled in
|
||||
`docs/guard-inventory.md`.
|
||||
|
||||
**The residue, named.** `git ls-files` reports INDEX entries, so a guard joins the population when
|
||||
it is STAGED rather than when the file appears. Nothing local runs these checks — `.husky/pre-commit`
|
||||
runs lint-staged, the decisions guard, the root-PNG check and `dotnet format`, and no husky hook runs
|
||||
pytest — so the red arrives from `pr-checks.yml::script-tests` on the PR. A file deleted from the
|
||||
working tree but not yet staged is still listed; `tracked_paths` asserts existence rather than
|
||||
filtering it out, because a filter is what makes a missing member unrepresentable, which is this
|
||||
record's first paragraph applied to its own implementation.
|
||||
|
||||
**Two traps specific to converting an existing guard, both of which this rule caught inside its own
|
||||
implementation.** FIRST, follow the data to where members are actually ADMITTED, not just to where
|
||||
the walk starts: a scrape that reads its caller files from the index and then admits the paths they
|
||||
name on `Path.exists()` is half-derived and reads as fully derived. SECOND, a proof that a
|
||||
derivation excludes untracked files must remove EVERY member in turn, not one. `derived_guard_files`
|
||||
unions four contributors; a single victim is always drawn from whichever sorts first, so a mutant
|
||||
putting only one contributor back on a filesystem walk passes. A sample cannot see the source it did
|
||||
not draw from — this record's opening argument, one level down, inside the artifact written to
|
||||
enforce it. Exhaustive removal is cheap — about a second at the population sizes here.
|
||||
|
||||
**Removal is only half the property, and the second half must not itself be machine-dependent.** A
|
||||
source contributing ONLY untracked members adds and never removes, so a removal-based check has
|
||||
nothing of its to take away: an `rglob` appending `.husky/_/` leaves the removal proof GREEN. Scope
|
||||
that claim on both axes, because it is narrower than it first reads. A broader `rglob` that also
|
||||
displaces tracked members DOES redden removal, so the blind spot is the append-only shape rather
|
||||
than every filesystem walk; and the append-only shape is blind only where the walked directory
|
||||
yields nothing — with the shims present its members are there to remove and removal reddens too. The
|
||||
gap is therefore an append-only source that is empty ON THIS MACHINE, which is exactly the CI
|
||||
checkout, and exactly where a guard going quiet is invisible.
|
||||
|
||||
The obvious complement — arrange an untracked file and require it not to enter — is a trap this
|
||||
change fell into and backed out of twice, and the reason is worth more than the rule. Writing probe
|
||||
files into the checkout under test means a probe in the `test_*.py` scope is a file pytest COLLECTS
|
||||
mid-session, probe names collide across concurrent runs, `finally` does not survive a SIGKILL, and a
|
||||
concurrent `git add -A` can stage one — defects in the test rather than in the thing tested.
|
||||
Neutralising the shared derivation and requiring the population to go empty is clean but misses the
|
||||
`.husky/_/` source on any machine where `.husky/_/` does not exist, which is the `script-tests`
|
||||
checkout, since that job never runs `npm ci`. Both formulations reproduce the green-in-CI /
|
||||
red-on-a-laptop asymmetry this record exists to abolish, inside the proof written to abolish it.
|
||||
|
||||
**So watch for the property that needs no arranged state: a directory LISTING issued while the
|
||||
derivation runs.** Listing is the commonest way a derivation discovers a member the index does not
|
||||
know about, and a walk issued during the derivation is caught on any machine — an `rglob` fails even
|
||||
where the directory it walks is empty, because the evidence is the call rather than what it
|
||||
returned. State its reach honestly, and state it once: what is observed is any
|
||||
call that goes THROUGH ONE OF THE SPIES, whenever it happens — the check's docstring works through
|
||||
the instances, and this record does not copy them, because the copy drifted from the original inside
|
||||
a single commit. It is a regression guard against the shapes that arrive by accident, not a
|
||||
boundary, and a guard sold as a boundary stops being re-examined.
|
||||
|
||||
**Three drafts of that one sentence were wrong, all in the same direction, and the third was wrong
|
||||
in the copies after the original had been fixed** — which is the completeness rule biting the prose
|
||||
that describes it. "Synchronously inside the call" was falsified by a thread finishing during the
|
||||
drain; "while the patch is active" was falsified by a spy reference captured inside the window and
|
||||
invoked after it, which still records. Each draft named the mechanism the author had in mind rather
|
||||
than the one that decides, and each understated the coverage. Understating is the safe direction and
|
||||
still worth correcting: a limit stated too narrowly invites someone to build the case it appears to
|
||||
exclude. The durable lesson is the one this record already gives for populations — do not keep a
|
||||
second copy of a statement that is still being corrected. Its complement is removal, which catches the shape that admits a HARDCODED
|
||||
path without listing anything (`if (REPO_ROOT / "x").exists(): add` — the defect this change shipped
|
||||
in its own first round). Neither alone is the property.
|
||||
|
||||
**What this record does not cover.** A population of *sites in code* — the places that dispatch on a
|
||||
value — has no external enumerator. #403 is that case: `PlaybackOrder`'s values are enumerable, but
|
||||
the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs
|
||||
|
||||
@@ -5,7 +5,7 @@ status: active
|
||||
since: '2026-08-13'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD''S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631''s suite was invoked by no CI job, #751''s step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719''s new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the filesystem and the call sites, asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row''s own named test required to go red.'
|
||||
rule: 'A guard is not considered tested because a test involving it passes. It ships with a MUTATION PROOF: remove or disarm THAT GUARD''S CLAUSE ALONE, and a NAMED test must go red. ONE NAMED EXCEPTION, with its limits, because the rule degenerates without it: where the guard IS a test (a checker enforcing a repo invariant, with no separate script behind it), disarming it makes it ABSENT rather than red, so the proof is the contrapositive — INTRODUCE THE DEFECT THE GUARD EXISTS TO CATCH into an isolated copy of the guarded artifact, and the named test must go red. That is a mutation of the guarded SYSTEM rather than of the assertion, and it is admissible ONLY for checker-guards and ONLY when the mutation was executed and witnessed. It is NOT a licence to grade an ordinary script-guard MUTATION for having a bad-input test: feeding a script an input its clause rejects is BEHAVIOUR-ONLY, which is what three rows were regraded for. A file-level grade under this exception covers the clause its cited case actually mutates, not every assertion that later lands in the same file. Three things this excludes, each of which has already shipped here as a green suite over a dead check. FIRST, a behavioural test — one that feeds the guard a good input and a bad input and checks it passes and fails — proves the guard REACTS, never that it is LOAD-BEARING; #685 had two guards on one condition where deleting either left the whole suite green while every behavioural test passed. SECOND, mutating the WHOLE FILE does not count (#510): a whole-file revert cannot show that a test reaches a particular clause, so the mutation must target the clause. THIRD, the guard being WIRED is not the guard RUNNING — #631''s suite was invoked by no CI job, #751''s step was dropped by the runner and the job reported success in 6s against a normal 14-17s, and #719''s new logic was never connected to stdin. Every guard that DERIVES A POPULATION also carries an ANTI-VACUITY assertion, because the characteristic failure of a completeness check is reporting that it proved everything while its population was empty; a guard with no population has nothing for such an assertion to be about, and stating it universally reads as coverage the unproven rows do not have. Mechanical enforcement is possible for the BOOKKEEPING and not for the JUDGEMENT, and the split is the decision: `docs/guard-inventory.md` lists every guard file with its Kind, its Proof class (`MUTATION`/`BEHAVIOUR-ONLY`/`NONE`) and a `file::function` ref, and `scripts/tests/test_guard_inventory.py` derives the guard population from the GIT INDEX and the call sites (#806), asserts SET EQUALITY against the rows, and resolves every claimed ref to a real `def`. So a new guard cannot ship unclassified and a renamed test cannot leave a row silently claiming coverage. Whether a row claiming `MUTATION` is telling the truth is no longer left to review: `testing.mutation-claims-are-executed` (#790) requires each such row to carry a DECLARED clause mutation that is applied to an isolated copy of the repository on every run, with the row''s own named test required to go red.'
|
||||
signals: 'mutation proof · delete the guard alone see red · disarm the clause not the file · behaviour-only is not a proof · anti-vacuity assertion · guard wired is not guard running · a green job with no step output · `docs/guard-inventory.md` · set equality against the row set · proof ref resolves to a real def · the unproven majority is a moving figure — read it off the inventory · PROOF kind stops the regress · hook wiring is not hook existence · paths: `scripts/tests/test_guard_inventory.py`, `scripts/tests/test_ci_dropped_step_guard.py`, `scripts/ci-prove-ban-detects.sh` · issues: #775, #773, #751, #756, #719, #685, #631, #621, #510, #445'
|
||||
mechanics: 'Proof classes are a closed vocabulary enforced by the inventory test; a `TOOLING` row may not claim a proof. `scripts/ci-prove-ban-detects.sh` is the one guard that runs its own mutation at CI time rather than in pytest, because the thing it proves — that the ban test is not deselected — is disarmable from inside pytest configuration.'
|
||||
---
|
||||
|
||||
@@ -31,17 +31,17 @@ declared `target`, which is not always the guard's own file; and — via the pos
|
||||
unmutated sandbox, so "the mutation was noticed" cannot be confused with "the test was already red".
|
||||
The proofs that carry a disarm of their own run it during that control; the two that are plain
|
||||
production set-equality checks have none, and the harness supplies theirs. It does **not** prove the declared clause is the only thing the guard hangs on.
|
||||
For three of the thirteen entries (measured 2026-08-22: the BOM guard, `build_decisions_catalog.py`
|
||||
For three entries (measured 2026-08-22: the BOM guard, `build_decisions_catalog.py`
|
||||
and `prove-fix.sh`) the redness arrives through the proof test's own "this clause has moved, RETARGET
|
||||
it" assertion rather than through changed behaviour. That is the intended reading rather than a hole:
|
||||
those tests perform their behavioural disarm themselves on every green run; what they could not do
|
||||
was notice their own clause reference going stale. The remaining ten redden behaviourally, and each
|
||||
was notice their own clause reference going stale. Every other entry reddens behaviourally, and each
|
||||
one's declared diagnostic says which.
|
||||
|
||||
**The `DETECTOR` grade is the honest half, and it exists because measuring found a case.** #775's
|
||||
record concedes that neutering `pin_population_faults` wholesale is "coarser than disarming one
|
||||
clause at a time — coarse enough that a single surviving clause would not be noticed". Running the
|
||||
finer mutations settled which guards actually admit clause-level proof: twelve of the thirteen do,
|
||||
finer mutations settled which guards actually admit clause-level proof: all but one do,
|
||||
including
|
||||
`pin_population_faults`, where disarming the against-the-registry comparison alone reddens the proof
|
||||
because a job that loses its `container:` block leaves the two derived sets equal. One does not.
|
||||
|
||||
+90
-15
@@ -1,9 +1,16 @@
|
||||
# Guard inventory (ersatztv#774 / #775)
|
||||
|
||||
Every executable guard **file** in this repo, what it blocks, and whether it ships a proof it can go
|
||||
red. `scripts/tests/test_guard_inventory.py` derives the population from the filesystem and the
|
||||
red. `scripts/tests/test_guard_inventory.py` derives the population from the **git index** and the
|
||||
workflow/hook call sites and asserts **set equality** against the `Guard` column, so a new guard
|
||||
cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.
|
||||
The index rather than a filesystem walk since ersatztv#806 — the practical consequence is that a new
|
||||
guard joins the population when it is **staged**, not when the file appears. Nothing local runs
|
||||
these checks at all: `.husky/pre-commit` runs lint-staged, the decisions guard, the root-PNG check
|
||||
and `dotnet format`, and `grep -rn pytest .husky/` returns nothing. The runner is
|
||||
`pr-checks.yml::script-tests`, `on: pull_request`, so the red arrives in CI — plus
|
||||
`docker-build.yml:723` on the release path, which re-runs two of these files
|
||||
(`test_ci_dropped_step_guard.py` and `test_ci_release_path_scan_job.py`) as a `needs:` of `build`.
|
||||
|
||||
**Read `docs/decisions/records/testing/guard-derives-population-from-source.md` and
|
||||
`…/guard-ships-with-mutation-proof.md` before editing a guard or adding a row.**
|
||||
@@ -40,11 +47,11 @@ exists because arguments of that shape have been wrong here six times.
|
||||
|
||||
## Scope limit, stated rather than implied
|
||||
|
||||
This inventory covers guard **files**, discovered by globbing `.claude/hooks/*.sh`, `.husky/*` and
|
||||
`scripts/tests/test_*.py`, plus every `scripts/…` path referenced by a workflow or a hook. The
|
||||
classes below are outside that population. They are listed because the first version of this section named
|
||||
only the first one, and cold review found that the very guards this inventory shipped with were
|
||||
sitting in the gap:
|
||||
This inventory covers guard **files**, discovered by reading `.claude/hooks/*.sh`, `.husky/*` and
|
||||
`scripts/tests/test_*.py` **out of the git index**, plus every `scripts/…` path referenced by a
|
||||
workflow or a hook. The classes below are outside that population. They are listed because the first
|
||||
version of this section named only the first one, and cold review found that the very guards this
|
||||
inventory shipped with were sitting in the gap:
|
||||
|
||||
1. **Guards inline in workflow YAML** — most importantly `pr-checks.yml:ci-image-pin`. "Which jobs
|
||||
are guards" needs a judgement call per job the filesystem cannot supply. Two were audited under
|
||||
@@ -56,15 +63,24 @@ sitting in the gap:
|
||||
`pr-checks.yml` error message, so removing the step that runs it would leave its row intact.
|
||||
4. **Nested and non-lowercase paths** beyond `scripts/tests/` — a guard under
|
||||
`scripts/scripted-schedules/`, or with an uppercase name, is invisible to the scrape.
|
||||
5. **`.yaml` workflows and non-`.sh` hooks** — the globs are `*.yml` and `*.sh` only.
|
||||
5. **Non-`.sh` hooks** — the hook pattern is `*.sh` only. (`.yaml` workflows are no longer in this
|
||||
gap: the caller scan matches `*.yml` and `*.yaml` since #806, because Gitea accepts both.)
|
||||
6. **Transitive calls** — a script invoked only by another script, rather than by a workflow or
|
||||
hook, is not discovered.
|
||||
7. **Non-`test_` modules under `scripts/tests/`** — the glob is `test_*.py`, so `conftest.py`,
|
||||
`mutation_harness_lib.py` and `mutation_manifest.py` are outside the population and hold no rows.
|
||||
They are not guards (they assert nothing on their own), but the second and third ARE what
|
||||
`test_mutation_harness.py` is made of, so gutting either would take that guard with it. What
|
||||
catches that is the guard's own row: its declared mutation targets `mutation_harness_lib.py`, and
|
||||
its proof test refuses to run if the clause it names has moved.
|
||||
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
|
||||
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
|
||||
population and hold no rows. They are not guards (they assert nothing on their own), but the
|
||||
middle two ARE what `test_mutation_harness.py` is made of, so gutting either would take that
|
||||
guard with it. What catches that is the guard's own row: its declared mutation targets
|
||||
`mutation_harness_lib.py`, and its proof test refuses to run if the clause it names has moved.
|
||||
`tracked_files.py` has no such backstop — it is load-bearing for every module that imports it
|
||||
(#806), and an edit to it is covered only by those modules' own proofs. Recorded rather than
|
||||
force-fitted: a row for a library would need a `Kind` the vocabulary does not have.
|
||||
8. **Nested workflow directories** — the workflow scope is direct children of `.gitea/workflows`, so
|
||||
a tracked `.gitea/workflows/nested/x.yml` is invisible to the caller scan and to
|
||||
`test_ci_image_pin_population.py`. Left as scope rather than widened: whether Gitea executes
|
||||
nested workflow files was not verified here, and widening on an unverified premise risks a
|
||||
permanent red on a correct tree, which is how a correct guard gets deleted.
|
||||
|
||||
Hook **wiring** is checked (`test_every_hook_file_is_actually_WIRED` reads `.claude/settings.json`
|
||||
and the husky hooks with full-line comments stripped), so a hook file whose registration is deleted
|
||||
@@ -75,6 +91,64 @@ through a wrapper or a constructed path reads as unwired. It catches deletion, w
|
||||
case; it does not catch deliberate disablement. The check does not extend to the `scripts/` half at
|
||||
all.
|
||||
|
||||
## File populations and where they come from (ersatztv#806)
|
||||
|
||||
Every guard here whose members are FILES derives them from the **git index**, never a filesystem
|
||||
walk. The disk is not an authoritative source: it reports build output and editor droppings and
|
||||
differs per machine, so a guard derived from it asserts a different population in CI than on the
|
||||
laptop of the person it is meant to stop. `scripts/tests/tracked_files.py` is the single derivation
|
||||
and carries the full rationale; `scripts/tests/test_guard_populations_derive_from_git.py` proves it,
|
||||
in both directions: removing EVERY member of each registered derivation from the index one at a time
|
||||
and requiring it to disappear while still on disk, and watching for a directory LISTING issued while
|
||||
the derivation runs (reading files stays allowed). Removal alone is blind to a source that
|
||||
contributes only untracked members — an `rglob` reaching `.husky/_/` adds and never removes — and
|
||||
any check phrased as "an untracked file must not enter" is itself machine-dependent, because the
|
||||
untracked file has to exist. Watching for the call needs no arranged state.
|
||||
|
||||
That second check is a regression guard against the accidental shapes, not a boundary: what is
|
||||
observed is any call that goes **through one of the spies**, whenever it happens — the spy records
|
||||
into a list that outlives the patch, so a reference captured during the window and invoked after it
|
||||
still counts. Whether the call goes through a spy is what decides, not when. The instance list — what
|
||||
never reaches a spy at all — lives in the check's own docstring and is deliberately not restated
|
||||
here, because a second copy of it drifted from the first within one commit.
|
||||
|
||||
One deliberate exception to "from the index" sits in the same file: the registration check lists
|
||||
`scripts/tests/test_*.py` from disk on purpose, because it is a superset check over what pytest
|
||||
collects — an untracked stray there makes it MORE demanding, never blind, whereas using the index
|
||||
would let an unstaged new guard escape registration.
|
||||
|
||||
Its own limit, stated because a check described as complete stops being re-examined: it finds
|
||||
derivations by PARSING each `test_*.py` for an import of the shared helper, so a module that derives
|
||||
a file population some other way is invisible to it, and no mechanical check can close that (#774
|
||||
reached the same conclusion about detecting filter-shaped guards by token).
|
||||
|
||||
The audit #806 asked for, recorded whichever way it came out, because "we looked and left it" and
|
||||
"we never looked" are indistinguishable a year later:
|
||||
|
||||
| Guard | Population | Completeness claim over tracked files? | Outcome |
|
||||
| --- | --- | --- | --- |
|
||||
| `test_guard_inventory.py` | `.claude/hooks/*.sh`, `.husky/*`, `scripts/tests/test_*.py`, workflow/hook callers | **yes** — set equality against this table | converted to the index; `.husky/_/` had been excluded only because `_` is a directory, so the obvious "make it recursive" edit would have reintroduced #778's defect here |
|
||||
| `test_hook_fire_log.py` | `.claude/hooks/*.sh` | **yes** — every hook must be instrumented | converted; an untracked scratch `.sh` used to demand instrumentation and redden the suite on that checkout alone |
|
||||
| `test_ci_image_pin_population.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — "docker-build is the ONLY workflow pinning the toolchain image" | converted, and `*.yaml` added: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible while the test read as covering all of them |
|
||||
| `test_pr_changed_files.py` | `.gitea/workflows/*.y*ml` | **yes** — "no OTHER workflow writes the review-verdict status" | converted. Not on #806's list and found by cold review on the pushed head: an untracked `.yaml` dropped in `.gitea/workflows/` reddened two guards while absent from the index — the issue's list of files to assess was a starting point, not the population |
|
||||
| `web/src/api/pageSizeCallSites.guard.test.ts` | `web/src/**/*.{ts,tsx,mts,cts}` via `import.meta.glob` | **yes** — an unregistered discovered site fails | DEFERRED to #819, assessed not skipped. The glob is a documented workaround: `@types/node` is deliberately out of `tsconfig.app.json`, and wiring it in was tried and reverted (it leaked Node's `setTimeout` into the app project and broke three unrelated tests), so there is no `node:child_process` to reach the index from. Over-enumerates, so it fails loudly rather than going blind |
|
||||
| `test_ci_release_path_scan_job.py` | `.gitea/workflows/*.y*ml` + `scripts/**` | **no** — a fixture assembling a tmp harness, asserted about behaviour not membership | takes its file LIST from the index anyway, for hermeticity not completeness: `shutil.copytree` copied whatever was on disk, including untracked files and `scripts/__pycache__`, into a tree whose behaviour the probes then measure. Content still comes from the working tree. The copy is not a git repo, so the two files this step RUNS may not use the helper — see the fixture docstring |
|
||||
| `test_ci_dropped_step_guard.py` | the parsed workflow document | **no filesystem population at all** | unchanged; its residual is `MARKED_JOBS`, a SCOPE mirror of the required contexts on `main`, which #806 does not close |
|
||||
|
||||
`test_remote_state_inventory.py` fixed its own population under #778 and kept a private copy of the
|
||||
derivation; #806 folded it onto the shared one, so that module is covered by the proof above like
|
||||
the rest. Across the whole change, every module that derived a file population its own way now goes
|
||||
through `tracked_files.py` — one implementation of the rule instead of one per module. The
|
||||
registered derivations are listed in `DERIVATIONS` in
|
||||
`scripts/tests/test_guard_populations_derive_from_git.py`; this page deliberately keeps no count
|
||||
of them.
|
||||
|
||||
**Still on filesystem walks, deliberately out of scope:** the decisions corpus
|
||||
(`scripts/decisions_lib.py`'s `active_files()`, and the suites over it). Its members are `docs/`
|
||||
Markdown with no generated-file pressure and a different lifecycle, and folding it in here would
|
||||
have been the reflex this milestone argues against — a change with no defect behind it. It is
|
||||
recorded as unexamined rather than as cleared.
|
||||
|
||||
## Inventory
|
||||
|
||||
| Guard | Blocks | Kind | Proof | Proof ref |
|
||||
@@ -125,6 +199,7 @@ all.
|
||||
| `scripts/tests/test_decisions_lib.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_decisions_validate.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_guard_inventory.py` | the `script-tests` job, on an unclassified guard or a stale proof ref | GUARD | MUTATION | `test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist` |
|
||||
| `scripts/tests/test_guard_populations_derive_from_git.py` | the `script-tests` job, on a guard whose file population admits a file git does not track | GUARD | MUTATION | `test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file` |
|
||||
| `scripts/tests/test_hook_fire_log.py` | the `script-tests` job, on a hook that stops reporting that it fired, or whose reporting changes what the harness sees | GUARD | MUTATION | `test_hook_fire_log.py::test_a_hook_that_LOSES_its_instrumentation_is_DETECTED` |
|
||||
| `scripts/tests/test_jq_preflight.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
| `scripts/tests/test_merge_consent_base_change.py` | the `script-tests` job | PROOF | NONE | — |
|
||||
@@ -150,7 +225,7 @@ graded `MUTATION` without declaring its clause fails the suite. Full contract an
|
||||
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
|
||||
|
||||
Two things that column still does not say. It does not say the declared clause is the ONLY thing the
|
||||
guard hangs on — for three of the thirteen rows (measured 2026-08-22) the redness arrives through the
|
||||
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
|
||||
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
|
||||
behaviour, which catches the recorded proof going stale but not much else. And one row is graded `DETECTOR` rather than `CLAUSE` in the
|
||||
manifest: `instrumentation_faults` accumulates from four independent arms and a stripped hook trips
|
||||
@@ -160,7 +235,7 @@ clause-provable and the entry is regraded.
|
||||
|
||||
## What the numbers say
|
||||
|
||||
35 guards, 6 tooling scripts, 19 proof files. **13 guards carry a mutation proof; 6 are
|
||||
36 guards, 6 tooling scripts, 19 proof files. **14 guards carry a mutation proof; 6 are
|
||||
behaviour-only; 16 have none.** These figures are asserted against the table by
|
||||
`test_the_summary_counts_match_the_table` — they were wrong in the first draft (28/4/6/3/19 against
|
||||
a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it,
|
||||
|
||||
@@ -127,6 +127,25 @@ MUTATIONS: tuple[Mutation, ...] = (
|
||||
"comparison notices it has moved to the bare runner. This is the clause #790 asked for "
|
||||
"instead of neutering `pin_population_faults` wholesale.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/tests/test_guard_populations_derive_from_git.py",
|
||||
target="scripts/tests/test_guard_inventory.py",
|
||||
clause="if ref in tracked:",
|
||||
replacement="if (REPO_ROOT / ref).exists():",
|
||||
proof="test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file",
|
||||
granularity=CLAUSE,
|
||||
expect="after git stopped tracking them",
|
||||
why="THE GUARD IS A TEST, so the mutation goes into the guarded ARTIFACT — one of the "
|
||||
"derivations it watches — rather than into the checker, per the checker-guard exception. "
|
||||
"The clause is the exact defect this guard was written after: `derived_guard_files` read "
|
||||
"its CALLERS from the index and then admitted the paths they name on `Path.exists()`, so a "
|
||||
"tracked workflow naming a script that exists on one machine only entered the population "
|
||||
"there, red on that checkout and green in CI (#778's third shape, found by cold review "
|
||||
"inside #806 itself). Note what this mutation does NOT do: on a clean tree the mutated set "
|
||||
"is identical, so `test_guard_inventory.py`'s own assertions stay green — only narrowing "
|
||||
"the index, which is what the proof does, separates them. That is why the proof has to "
|
||||
"remove EVERY member rather than sample one.",
|
||||
),
|
||||
Mutation(
|
||||
guard="scripts/tests/test_guard_inventory.py",
|
||||
target="docs/guard-inventory.md",
|
||||
|
||||
@@ -30,6 +30,12 @@ import pytest
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
# ASSESSED FOR ersatztv#806: this file has NO filesystem-derived population. Its members come from
|
||||
# the PARSED workflow (`_marked(job)` reads the marked steps out of `_DOC`), which is already an
|
||||
# authoritative machine-readable source, so the index changes nothing here. Its known residual is at
|
||||
# the other altitude — `MARKED_JOBS` is a hand-written mirror of the required contexts on `main`,
|
||||
# a SCOPE rather than a population, and `testing.guard-derives-population-from-source` already
|
||||
# carries it as this repo's canonical residual gap. #806 does not close it.
|
||||
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
|
||||
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
|
||||
|
||||
|
||||
@@ -38,9 +38,22 @@ from pathlib import Path
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from scripts.tests.tracked_files import tracked_paths
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
|
||||
WORKFLOW = WORKFLOWS_DIR / "docker-build.yml"
|
||||
# Resolved against the GIT INDEX rather than `Path.glob` (ersatztv#806), and `*.yaml` alongside
|
||||
# `*.yml`: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible to the
|
||||
# scope check below while reading as covered.
|
||||
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
|
||||
|
||||
|
||||
def workflow_files() -> list[Path]:
|
||||
"""THE WORKFLOW POPULATION, from the git index. Named rather than inline so the shared proof in
|
||||
`test_guard_populations_derive_from_git.py` can assert it never admits an untracked file."""
|
||||
return tracked_paths(*WORKFLOWS)
|
||||
|
||||
|
||||
# The image repository, without the tag. Matched as a whole path rather than by the bare
|
||||
# `ersatztv-ci` token so a job pointing at a LOOK-ALIKE registry (a personal fork, a typo'd host)
|
||||
@@ -265,7 +278,7 @@ def test_docker_build_is_the_ONLY_workflow_pinning_the_toolchain_image():
|
||||
"""This file reads ONE workflow, which is itself a scope mirror needing its own check.
|
||||
|
||||
`WORKFLOW` hardcodes `docker-build.yml`, and the implicit claim — that no other workflow uses
|
||||
the toolchain image — mirrors a machine-readable source (the glob of `.gitea/workflows/*.yml`)
|
||||
the toolchain image — mirrors a machine-readable source (the tracked `.gitea/workflows/*.y*ml`)
|
||||
that nothing consulted. `renovate.yml` already declares a `container:` with a different image,
|
||||
so the shape is live. A future workflow adopting `ersatztv-ci:` would acquire no pin-population
|
||||
guard, no single-tag check and no partition, silently, while `pin_population_faults`'s own error
|
||||
@@ -274,13 +287,20 @@ def test_docker_build_is_the_ONLY_workflow_pinning_the_toolchain_image():
|
||||
Found by cold review, which correctly noted this file criticises `MARKED_JOBS` for exactly this
|
||||
and then shipped the same shape without even the dated comment `MARKED_JOBS` carries.
|
||||
|
||||
The population comes from the GIT INDEX (ersatztv#806). A `Path.glob` here answered a question
|
||||
about the machine rather than about the repo: an untracked scratch workflow left in
|
||||
`.gitea/workflows/` would be parsed and could redden this test on one checkout while CI, which
|
||||
never sees it, stayed green. The pattern set gained `*.yaml` in the same change — Gitea accepts
|
||||
both spellings, so a `.yaml` workflow adopting the toolchain image was invisible here while this
|
||||
test read as covering every workflow.
|
||||
|
||||
Checked by PARSING each workflow's `container.image`, not by grepping the file. A text search
|
||||
reports `ci-image.yml`, which names the image because it BUILDS and PUSHES it — a producer, not
|
||||
a consumer. Grepping would have made this test permanently red on a correct tree, which is the
|
||||
fastest route to a correct guard being deleted.
|
||||
"""
|
||||
others = []
|
||||
for p in sorted(WORKFLOWS_DIR.glob("*.yml")):
|
||||
for p in workflow_files():
|
||||
if p.name == WORKFLOW.name:
|
||||
continue
|
||||
doc = yaml.safe_load(p.read_text()) or {}
|
||||
|
||||
@@ -77,6 +77,15 @@ from pathlib import Path
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
# Imports the shared index derivation to build a HERMETIC fixture copy, not to derive a guard
|
||||
# population — see `_repo_copy`. Recorded as such in POPULATION_EXEMPT in
|
||||
# `test_guard_populations_derive_from_git.py`; the exemption lives there, not here, because a marker
|
||||
# a file grants itself is a kill switch any prose mention can trip.
|
||||
from scripts.tests import tracked_files
|
||||
|
||||
# Both spellings, matching what the converted guards consider the workflow set.
|
||||
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
|
||||
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
|
||||
@@ -269,12 +278,49 @@ def _scan_body_and_env():
|
||||
|
||||
|
||||
def _repo_copy(tmp_path: Path) -> Path:
|
||||
"""A minimal executable copy of the repo: the workflow plus all of scripts/."""
|
||||
"""A minimal executable copy of the repo: the tracked workflows plus the tracked `scripts/`.
|
||||
|
||||
ASSESSED FOR ersatztv#806. This is NOT a completeness guard — it is a fixture assembling a
|
||||
harness, and no assertion in this file is about which files it found; the probes assert what the
|
||||
scan command DOES to the copy. It takes its file LIST from the index anyway, for hermeticity
|
||||
rather than completeness: `shutil.copytree` copied whatever was on disk, so untracked files and
|
||||
`scripts/__pycache__` entered a tree whose behaviour the probes then measure.
|
||||
|
||||
WHAT THAT DOES AND DOES NOT BUY, stated exactly, because a fixture described as hermetic stops
|
||||
being questioned. The list comes from the index; the CONTENT comes from the working tree, so an
|
||||
unstaged edit to a tracked `scripts/**` file is still copied in. Making the content hermetic too
|
||||
would need `git show`/`git archive` and would mean the probes stop testing the tree under edit,
|
||||
which is the wrong trade for a test whose job is to catch a disarm in that tree.
|
||||
|
||||
An extra WORKFLOW in the copy is inert, but check what the step runs before relying on that:
|
||||
`docker-build.yml:723` runs TWO files, `test_ci_dropped_step_guard.py` AND this one, and both
|
||||
parse only `docker-build.yml`.
|
||||
|
||||
THE COPY IS NOT A GIT REPOSITORY, and that is the constraint to know before touching either file
|
||||
that step runs. Neither may derive a population through `scripts/tests/tracked_files.py`:
|
||||
`git ls-files` inside the copy fails, and the release-path scan step fails with it. The copying
|
||||
happens HERE, in the real repo, which is why this fixture may use the index while its subjects
|
||||
may not. `test_ci_dropped_step_guard.py`'s `MARKED_JOBS` is explicitly left open as a residual
|
||||
gap, so a future session is invited to edit exactly that file — this paragraph is what stands
|
||||
between that edit and a broken release gate.
|
||||
"""
|
||||
dst = tmp_path / "repo"
|
||||
(dst / ".gitea" / "workflows").mkdir(parents=True)
|
||||
for wf in (REPO_ROOT / ".gitea" / "workflows").glob("*.yml"):
|
||||
for wf in tracked_files.tracked_paths(*WORKFLOWS):
|
||||
shutil.copy2(wf, dst / ".gitea" / "workflows" / wf.name)
|
||||
shutil.copytree(REPO_ROOT / "scripts", dst / "scripts")
|
||||
for rel in tracked_files._git_ls_files():
|
||||
if not rel.startswith("scripts/"):
|
||||
continue
|
||||
source = REPO_ROOT / rel
|
||||
# Diagnosed, not raised as a bare FileNotFoundError one line after using the module whose
|
||||
# whole point is reporting this case comprehensibly.
|
||||
assert source.is_file(), (
|
||||
f"git tracks {rel} but there is no file there, so the harness copy would be incomplete "
|
||||
"and the probes below would measure a tree that is missing part of the thing under test."
|
||||
)
|
||||
target = dst / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(source, target)
|
||||
return dst
|
||||
|
||||
|
||||
|
||||
@@ -18,9 +18,11 @@ about a sample, but you CAN mechanically guarantee that every guard has been *cl
|
||||
and that its claimed proof exists. That converts both rules from "remember to do this" into "the
|
||||
suite goes red until you have". Specifically:
|
||||
|
||||
* the guard population is DERIVED — globbed from `.claude/hooks/` and `.husky/`, plus every
|
||||
`scripts/*.sh|py` referenced by a workflow or a hook — and compared for SET EQUALITY against the
|
||||
inventory's rows, in both directions;
|
||||
* the guard population is DERIVED — read out of the GIT INDEX for `.claude/hooks/` and
|
||||
`.husky/`, plus every `scripts/*.sh|py` referenced by a workflow or a hook — and compared for
|
||||
SET EQUALITY against the inventory's rows, in both directions. The index rather than the disk
|
||||
per ersatztv#806: a filesystem walk reports build output and editor droppings and differs per
|
||||
machine, so it cannot be the authoritative source a completeness claim needs;
|
||||
* every row's `Proof ref` is resolved to a real file and a real `def` in it;
|
||||
* every row's `Kind` and `Proof` come from a closed vocabulary, so a typo cannot invent a state.
|
||||
|
||||
@@ -35,11 +37,22 @@ import re
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.tests import tracked_files
|
||||
from scripts.tests.tracked_files import tracked_children, tracked_paths
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
INVENTORY = REPO_ROOT / "docs" / "guard-inventory.md"
|
||||
HOOKS_DIR = REPO_ROOT / ".claude" / "hooks"
|
||||
HUSKY_DIR = REPO_ROOT / ".husky"
|
||||
WORKFLOWS_DIR = REPO_ROOT / ".gitea" / "workflows"
|
||||
|
||||
# THE POPULATION SCOPES, resolved against the GIT INDEX rather than the filesystem (ersatztv#806).
|
||||
# Directory + patterns instead of `Path.glob`; `scripts/tests/tracked_files.py` carries why the disk
|
||||
# is not an authoritative source. `.husky` is the sharp case: it holds an untracked `_/` of 17
|
||||
# npm-generated shims, and the previous `iterdir() ... if p.is_file()` excluded them only because
|
||||
# `_` happens to be a directory — by accident, not by design, so the obvious "make it recursive"
|
||||
# edit would have reintroduced #778's third defect inside the repo's own model guard.
|
||||
HOOKS = (".claude/hooks", ("*.sh",))
|
||||
HUSKY = (".husky", ("*",))
|
||||
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
|
||||
GUARD_TESTS = ("scripts/tests", ("test_*.py",))
|
||||
|
||||
KINDS = {"GUARD", "TOOLING", "PROOF"}
|
||||
PROOFS = {"MUTATION", "BEHAVIOUR-ONLY", "NONE"}
|
||||
@@ -64,31 +77,52 @@ _SUMMARY = re.compile(
|
||||
|
||||
|
||||
def derived_guard_files() -> set[str]:
|
||||
"""THE AUTHORITATIVE POPULATION, from the filesystem and the call sites — never a list.
|
||||
"""THE AUTHORITATIVE POPULATION, from the git index and the call sites — never a list.
|
||||
|
||||
Three sources, unioned. The hook and husky directories are globbed whole, so a new hook is in
|
||||
the population the moment it exists. The scripts half is discovered by scanning what the
|
||||
workflows and hooks actually INVOKE, rather than globbing `scripts/` — a script nothing calls is
|
||||
not a guard, and globbing would drag in every helper and make the inventory a chore that gets
|
||||
rubber-stamped.
|
||||
Four contributors, unioned: three scope directories plus the paths those files REFERENCE. The
|
||||
hook and husky directories are taken whole, so a new hook is in the population the moment it is
|
||||
STAGED. The scripts half is discovered by scanning what the workflows and hooks actually
|
||||
INVOKE, rather than taking `scripts/` whole — a script nothing calls is not a guard, and taking
|
||||
it whole would drag in every helper and make the inventory a chore that gets rubber-stamped.
|
||||
Both halves are gated on the index: the callers by `tracked_paths`, the targets they name by the
|
||||
`tracked` set below.
|
||||
|
||||
"The moment it is STAGED" rather than "the moment it exists" is the ersatztv#806 change, and
|
||||
it is a strengthening: an untracked `foo.sh` dropped in `.claude/hooks/` used to enter this
|
||||
population and demand an inventory row for a file that is not part of the repo — red on that
|
||||
checkout, green in CI, which is #778's third shape. Nothing weakens, because a guard that is not
|
||||
staged is not on its way to anyone else either.
|
||||
"""
|
||||
found = (
|
||||
{str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
|
||||
| {str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()}
|
||||
| {
|
||||
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
|
||||
# invoked and none is individually named anywhere. Globbing is the only derivation that
|
||||
# matches how they actually run.
|
||||
str(p.relative_to(REPO_ROOT))
|
||||
for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
|
||||
}
|
||||
tracked_children(*HOOKS)
|
||||
| tracked_children(*HUSKY)
|
||||
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is invoked
|
||||
# and none is individually named anywhere. Taking the directory whole is the only derivation
|
||||
# that matches how they actually run.
|
||||
| tracked_children(*GUARD_TESTS)
|
||||
)
|
||||
|
||||
callers = list(WORKFLOWS_DIR.glob("*.yml")) + list(HOOKS_DIR.glob("*.sh"))
|
||||
callers += [p for p in HUSKY_DIR.iterdir() if p.is_file()]
|
||||
# THE REFERENCED TARGETS ARE GATED ON THE INDEX, NOT ON `Path.exists()`. Converting the CALLERS
|
||||
# and leaving the members they contribute on a disk check would have left a quarter of this
|
||||
# population answering a question about the machine: a tracked workflow naming
|
||||
# `scripts/generated/helper.sh` that exists on one laptop only would enter there, demand an
|
||||
# inventory row for a file that is not in the repo, and go red on that checkout while CI stayed
|
||||
# green — #778's third shape, in the guard this file calls its model.
|
||||
#
|
||||
# A referenced path that git does not track is therefore dropped silently, and that is the right
|
||||
# residual rather than an assertion: `_SCRIPT_REF` matches any occurrence, including inside a
|
||||
# comment or an `::error::` string (limit 3 in `docs/guard-inventory.md`), so demanding that
|
||||
# every matched path be tracked would redden a correct tree on a prose mention.
|
||||
# Called through the MODULE, never `from … import _git_ls_files`. A direct name binding is
|
||||
# captured at import time, and the exhaustive proof in
|
||||
# `test_guard_populations_derive_from_git.py` then cannot narrow the index for this branch at
|
||||
# all — every referenced target reports as surviving removal, which is a red for the wrong
|
||||
# reason and, worse, means the branch is untested however the proof reads.
|
||||
tracked = set(tracked_files._git_ls_files())
|
||||
callers = tracked_paths(*WORKFLOWS) + tracked_paths(*HOOKS) + tracked_paths(*HUSKY)
|
||||
for caller in callers:
|
||||
for ref in _SCRIPT_REF.findall(caller.read_text()):
|
||||
if (REPO_ROOT / ref).exists():
|
||||
if ref in tracked:
|
||||
found.add(ref)
|
||||
return found
|
||||
|
||||
@@ -112,10 +146,9 @@ def wired_hook_files() -> set[str]:
|
||||
comments, so every occurrence there is in a real command string.
|
||||
"""
|
||||
text = (REPO_ROOT / ".claude" / "settings.json").read_text()
|
||||
for husky in HUSKY_DIR.iterdir():
|
||||
if husky.is_file():
|
||||
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
|
||||
return {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text}
|
||||
for husky in tracked_paths(*HUSKY):
|
||||
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
|
||||
return {rel for rel in tracked_children(*HOOKS) if rel.rpartition("/")[2] in text}
|
||||
|
||||
|
||||
def inventory_rows() -> list[tuple[str, str, str, str]]:
|
||||
@@ -210,8 +243,8 @@ def test_every_claimed_proof_names_a_test_that_exists():
|
||||
|
||||
def test_every_hook_file_is_actually_WIRED():
|
||||
"""A hook file nothing registers is dead code holding an inventory row that reads as coverage."""
|
||||
on_disk = {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
|
||||
unwired = sorted(on_disk - wired_hook_files())
|
||||
staged = tracked_children(*HOOKS)
|
||||
unwired = sorted(staged - wired_hook_files())
|
||||
assert not unwired, (
|
||||
f"these hook files exist and have inventory rows but are referenced by neither "
|
||||
f".claude/settings.json nor any .husky/ hook: {unwired}. They do not run. Either wire them "
|
||||
|
||||
@@ -0,0 +1,604 @@
|
||||
"""No file population in this repo admits a file git does not track (ersatztv#806).
|
||||
|
||||
The regression for #778's third defect, hoisted to cover every guard that shares the mechanism
|
||||
rather than being copied into each of them. `Path.rglob` enumerated `.husky/_/` — 17 husky shims
|
||||
generated by `npm ci`, gitignored and untracked — so `test_remote_state_inventory.py` was RED on
|
||||
every developer checkout and GREEN in CI, whose `script-tests` job checks out and pip-installs but
|
||||
never runs `npm ci`. A guard that fails everywhere except where it runs trains its readers to ignore
|
||||
it, and it did so on the artifact whose entire thesis is population correctness.
|
||||
|
||||
TWO PROOFS, because they fail differently and either alone leaves a hole.
|
||||
|
||||
* `test_the_primitive_REALLY_excludes_an_untracked_file` builds a throwaway git repo, commits one
|
||||
file, leaves an identical sibling untracked, and runs the real derivation against it. It proves
|
||||
the mechanism by EXECUTING it rather than by recognising its shape — no monkeypatching, no
|
||||
stand-in for git. Nothing here is a claim about `git ls-files`; it is `git ls-files`.
|
||||
* `test_no_derivation_admits_an_untracked_file` narrows the tracked set under each real derivation
|
||||
and requires the dropped member to vanish from the population even though the file is still on
|
||||
disk and still matches the scope. That is the property stated over the ACTUAL guards, so a
|
||||
future refactor that quietly reintroduces a filesystem walk in any one of them fails here rather
|
||||
than on somebody's laptop.
|
||||
|
||||
`DERIVATIONS` is the reason this file is not one near-copy per derivation: a guard that starts deriving a file
|
||||
population registers here, and both proofs cover it for free. The register is hand-written and that
|
||||
is a SCOPE decision, not a population one — per `testing.guard-derives-population-from-source`, a
|
||||
scope mirroring an authoritative source needs its own equality check, and
|
||||
`test_every_index_derived_module_is_registered` is it: it reads which modules import the shared
|
||||
helper and demands each one appear below, so adding another derivation and forgetting this file is
|
||||
red rather than silently uncovered.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import glob
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts.tests import test_ci_image_pin_population as image_pin
|
||||
from scripts.tests import test_guard_inventory as guard_inventory
|
||||
from scripts.tests import test_hook_fire_log as hook_fire
|
||||
from scripts.tests import test_pr_changed_files as pr_changed
|
||||
from scripts.tests import test_remote_state_inventory as remote_state
|
||||
from scripts.tests import tracked_files
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
TESTS_DIR = REPO_ROOT / "scripts" / "tests"
|
||||
|
||||
|
||||
class _EmptyScan:
|
||||
"""An exhausted ITERATOR that is also a context manager, standing in for `os.scandir`.
|
||||
|
||||
Both halves are load-bearing and each was missing in turn. `os.scandir` is used as
|
||||
`with os.scandir(...) as it`, so a bare iterator broke the context-manager protocol; and
|
||||
`os.walk` does `entry = next(scandir_it)` on the result, so an ITERABLE defining only
|
||||
`__iter__` broke that. Either way the enumeration assertion still fired with the right message,
|
||||
but the report also carried a TypeError about the harness — and a finding that arrives beside a
|
||||
harness error invites doubting the finding rather than the code.
|
||||
"""
|
||||
|
||||
def __iter__(self):
|
||||
return self
|
||||
|
||||
def __next__(self):
|
||||
raise StopIteration
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_exc):
|
||||
return False
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
|
||||
def _as_relative_strings(members) -> set[str]:
|
||||
"""Derivations return either repo-relative strings or absolute `Path`s; compare on one form.
|
||||
|
||||
Iterating into a set is also what DRAINS a derivation that returns a generator, which the
|
||||
enumeration proof depends on — see its call site.
|
||||
|
||||
An unexpected member type is REPORTED rather than stringified. `str(m)` on anything at all meant
|
||||
a derivation yielding, say, nested generators produced plausible-looking members and compared
|
||||
equal to nothing, which is a population check passing over data it did not understand.
|
||||
"""
|
||||
out = set()
|
||||
for m in members:
|
||||
assert isinstance(m, str | Path), (
|
||||
f"a derivation yielded {type(m).__name__} ({m!r}); populations here are repo-relative "
|
||||
"strings or absolute Paths, and stringifying anything else would compare a plausible "
|
||||
"value against a set that can never contain it."
|
||||
)
|
||||
out.add(str(Path(m).relative_to(REPO_ROOT)) if isinstance(m, Path) else str(m))
|
||||
return out
|
||||
|
||||
|
||||
# (label, callable, a floor below which the derivation has plainly broken). `test_hook_fire_log`
|
||||
# floors the same population at the same number for its own coverage assertions; that is not a
|
||||
# duplicate guard masking another, because the two protect different consumers from going
|
||||
# vacuous — delete this one and THIS file's proofs iterate over nothing while reporting success.
|
||||
DERIVATIONS = (
|
||||
("test_guard_inventory.derived_guard_files", guard_inventory.derived_guard_files, 25),
|
||||
("test_hook_fire_log.hook_scripts", hook_fire.hook_scripts, 10),
|
||||
("test_ci_image_pin_population.workflow_files", image_pin.workflow_files, 5),
|
||||
("test_remote_state_inventory.derived_population", remote_state.derived_population, 40),
|
||||
("test_pr_changed_files._workflow_files", pr_changed._workflow_files, 5),
|
||||
)
|
||||
|
||||
# The floor matters only to the anti-vacuity test; the two property tests take the pair, so an
|
||||
# unused parameter cannot drift into looking like an assertion they make.
|
||||
_IDS = [d[0] for d in DERIVATIONS]
|
||||
_PAIRS = [(label, derive) for label, derive, _ in DERIVATIONS]
|
||||
|
||||
# Modules that import the shared derivation WITHOUT deriving a guard population. Kept here, beside
|
||||
# DERIVATIONS, so adding one is an edit to this file that a reviewer sees.
|
||||
POPULATION_EXEMPT = {
|
||||
# Uses the index to assemble a HERMETIC tmp fixture copy; nothing in it asserts membership.
|
||||
"test_ci_release_path_scan_job.py": "index-derived fixture copy, not a population",
|
||||
}
|
||||
|
||||
_HELPER = "scripts.tests.tracked_files"
|
||||
_PACKAGE = ["scripts", "tests"]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# ANTI-VACUITY FIRST — every assertion below compares sets, and a derivation that collapsed to
|
||||
# nothing would satisfy all of them while proving nothing.
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("label", "derive", "floor"), DERIVATIONS, ids=_IDS)
|
||||
def test_each_derivation_found_something(label, derive, floor):
|
||||
members = _as_relative_strings(derive())
|
||||
assert len(members) >= floor, (
|
||||
f"{label} derived only {len(members)} members, below its floor of {floor} — the derivation "
|
||||
"is broken, not the repo, and every set comparison built on it is vacuous."
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# PROOF 1 — the primitive, executed against a real git repo rather than described
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_primitive_REALLY_excludes_an_untracked_file(tmp_path, monkeypatch):
|
||||
"""A tracked and an untracked file, identical in name shape and both on disk. Only one is in.
|
||||
|
||||
Run rather than reasoned about. This class of defect is produced by arguments about what a
|
||||
traversal WOULD enumerate, and such arguments are locally convincing whether or not they are
|
||||
right; only executing the traversal distinguishes the two.
|
||||
"""
|
||||
repo = tmp_path / "repo"
|
||||
(repo / ".claude" / "hooks").mkdir(parents=True)
|
||||
(repo / ".claude" / "hooks" / "committed.sh").write_text("#!/bin/sh\n")
|
||||
(repo / ".claude" / "hooks" / "untracked.sh").write_text("#!/bin/sh\n")
|
||||
|
||||
def git(*args):
|
||||
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True)
|
||||
|
||||
git("init", "-q")
|
||||
git("config", "user.email", "guard@example.invalid")
|
||||
git("config", "user.name", "guard")
|
||||
git("add", ".claude/hooks/committed.sh")
|
||||
git("commit", "-qm", "one tracked hook")
|
||||
|
||||
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
|
||||
found = tracked_files.tracked_children(".claude/hooks", ("*.sh",))
|
||||
|
||||
assert (repo / ".claude" / "hooks" / "untracked.sh").is_file(), (
|
||||
"the untracked file must still be on disk, or this proves nothing about the index winning over the filesystem"
|
||||
)
|
||||
assert found == {".claude/hooks/committed.sh"}, (
|
||||
f"the derivation returned {sorted(found)}. A filesystem walk returns both files here; only "
|
||||
"the index distinguishes them, and that difference is the entire point of ersatztv#806."
|
||||
)
|
||||
|
||||
|
||||
def test_the_primitive_does_not_recurse_into_an_untracked_subdirectory(tmp_path, monkeypatch):
|
||||
"""`.husky/_/` in miniature — the shape that made #778 red on every checkout.
|
||||
|
||||
Even a TRACKED nested file must stay out: `tracked_children` is direct-children-only by design,
|
||||
and recursion is what dragged the shims in. Proving it with a tracked file makes the assertion
|
||||
about the traversal rather than about the index, so the two properties cannot mask each other.
|
||||
"""
|
||||
repo = tmp_path / "repo"
|
||||
(repo / ".husky" / "_").mkdir(parents=True)
|
||||
(repo / ".husky" / "pre-commit").write_text("#!/bin/sh\n")
|
||||
(repo / ".husky" / "_" / "husky.sh").write_text("#!/bin/sh\n")
|
||||
|
||||
def git(*args):
|
||||
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True)
|
||||
|
||||
git("init", "-q")
|
||||
git("config", "user.email", "guard@example.invalid")
|
||||
git("config", "user.name", "guard")
|
||||
git("add", "-A")
|
||||
git("commit", "-qm", "husky plus a nested shim, both tracked")
|
||||
|
||||
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
|
||||
assert tracked_files.tracked_children(".husky", ("*",)) == {".husky/pre-commit"}, (
|
||||
"a nested file entered a flat population — this is the `.husky/_/` shape, and it was red on "
|
||||
"every developer checkout the last time it shipped"
|
||||
)
|
||||
|
||||
|
||||
def test_an_empty_index_FAILS_LOUDLY_rather_than_reporting_an_empty_population(tmp_path, monkeypatch):
|
||||
"""The floor under every floor. A silent empty population is how a completeness guard reports
|
||||
total coverage having examined nothing, which is the failure mode this repo has shipped twice
|
||||
(#631, #751)."""
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
subprocess.run(["git", "-C", str(repo), "init", "-q"], check=True, capture_output=True)
|
||||
|
||||
monkeypatch.setattr(tracked_files, "REPO_ROOT", repo)
|
||||
with pytest.raises(AssertionError, match="reported nothing"):
|
||||
tracked_files.tracked_children(".claude/hooks", ("*.sh",))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# PROOF 2 — the property, over the real guards
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
|
||||
def test_no_derivation_admits_an_untracked_file(label, derive):
|
||||
"""Narrow the index, leave the disk alone, and require the member to disappear — for EVERY
|
||||
member, one at a time.
|
||||
|
||||
EXHAUSTIVE RATHER THAN ONE VICTIM, and the difference is not thoroughness for its own sake.
|
||||
`derived_guard_files` unions four sources; one victim is always drawn from whichever sorts
|
||||
first, so a mutant putting only the third source back on a filesystem walk passes while the
|
||||
proof reports on all four. A sample cannot see the source it did not draw from — this file
|
||||
applying `testing.guard-derives-population-from-source` to itself.
|
||||
|
||||
The floors cannot substitute, and the numbers say why. Reproduce with:
|
||||
|
||||
PYTHONPATH=. python3 -c "from scripts.tests import test_guard_inventory as g; \
|
||||
print(len(g.derived_guard_files()))"
|
||||
|
||||
61 members on 2026-08-22; suppressing a single contributor leaves 39 (`scripts/tests`), 48
|
||||
(hooks) or 58 (husky), all far above the anti-vacuity floor of 25. The figures move whenever a
|
||||
guard is added — they were 60/39/47/57 one commit earlier — so read them as an illustration of
|
||||
the GAP, not as values to assert against. A floor tight enough to catch a lost source
|
||||
would go red every time a guard is legitimately deleted, which is the wrong instrument.
|
||||
|
||||
Its own `monkeypatch` context, never the shared fixture instance: the function-scoped fixture is
|
||||
the same object `conftest.py`'s autouse `isolate_hook_fire_log` patched, so calling `undo()` on
|
||||
it here also unsets `ETV_HOOK_FIRE_LOG_DIR` and silently re-points a later hook-driving test at
|
||||
the PRODUCTION log — the #776 isolation disarmed from inside the file that argues for proofs.
|
||||
|
||||
WHAT REMOVAL CANNOT SEE, so it is not read as more than it is: a source contributing ONLY
|
||||
untracked members has nothing here to remove, and #778's defect was exactly that shape (an
|
||||
`rglob` over `.husky/_/` adds 17 untracked members and removes none). That direction is
|
||||
`test_no_derivation_ENUMERATES_the_filesystem` below; the two are complements, not duplicates.
|
||||
"""
|
||||
before = _as_relative_strings(derive())
|
||||
assert before, f"{label} derived nothing; there is no victim to remove"
|
||||
|
||||
real = tracked_files._git_ls_files()
|
||||
survivors = []
|
||||
absent = []
|
||||
with pytest.MonkeyPatch.context() as m:
|
||||
for victim in sorted(before):
|
||||
if not (REPO_ROOT / victim).is_file():
|
||||
absent.append(victim)
|
||||
continue
|
||||
m.setattr(tracked_files, "_git_ls_files", lambda v=victim: [p for p in real if p != v])
|
||||
if victim in _as_relative_strings(derive()):
|
||||
survivors.append(victim)
|
||||
|
||||
assert not absent, (
|
||||
f"{label} contains {absent}, which git tracks but are not on disk. The proof below asserts "
|
||||
"that the INDEX decides while the file is still present; it cannot mean that for a member "
|
||||
"that is missing, so this is reported rather than skipped."
|
||||
)
|
||||
assert not survivors, (
|
||||
f"{label} still contains {survivors} after git stopped tracking them. Every one of those is "
|
||||
"still on disk, so the derivation is reading the filesystem for that member and untracked "
|
||||
"build output can redden it on a developer checkout while CI stays green (ersatztv#778)."
|
||||
)
|
||||
|
||||
|
||||
# The directory-listing APIs a Python file population is realistically written with. NOT every way a
|
||||
# process can list a directory — `subprocess.run(["ls"])`, a module-level alias captured before the
|
||||
# patch, and any C-level call all walk straight past this, all three verified by cold review. That
|
||||
# bounds what the check below can claim, and the docstring says so rather than implying a sandbox.
|
||||
# Reading a file stays allowed: `derived_guard_files` must read workflow bodies.
|
||||
_ENUMERATORS = (
|
||||
(Path, "glob"),
|
||||
(Path, "rglob"),
|
||||
(Path, "iterdir"),
|
||||
(Path, "walk"),
|
||||
(os, "listdir"),
|
||||
(os, "walk"),
|
||||
(os, "scandir"),
|
||||
(glob, "glob"),
|
||||
(glob, "iglob"),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
|
||||
def test_no_derivation_ENUMERATES_the_filesystem(label, derive):
|
||||
"""The ADD direction, without arranging any state: a derivation may READ files, but it may not
|
||||
LIST a directory while it runs.
|
||||
|
||||
WHY THE ADD DIRECTION NEEDS ITS OWN TEST. Removing members from the index cannot see a source
|
||||
that contributes ONLY untracked members: it adds and never takes away, so nothing of its is
|
||||
available to remove. `.husky/_/` is that shape exactly — 17 shims `npm ci` writes, none ever in
|
||||
the index — and it is the shape #778 shipped. Scope the claim precisely: a source appending
|
||||
`(REPO_ROOT / ".husky" / "_").rglob("*")` leaves the removal proof green ON A MACHINE WHERE THAT
|
||||
DIRECTORY IS ABSENT, which is the `script-tests` checkout. Where the shims exist, the appended
|
||||
members are present and removal reddens too. The blind spot is an append-only source that yields
|
||||
nothing HERE — which is exactly the CI shape, and exactly where a guard going quiet matters.
|
||||
|
||||
WHY THE PROPERTY IS "DOES NOT ENUMERATE" RATHER THAN "DOES NOT RETURN AN UNTRACKED FILE". Both
|
||||
obvious formulations of the latter are machine-dependent, which is the very fault #806 exists to
|
||||
remove:
|
||||
|
||||
* creating real probe files in the checkout and requiring they not enter. That needs a
|
||||
`test_*.py` probe to reach the `scripts/tests` scope — a file pytest COLLECTS mid-session —
|
||||
and its parametrised names collide across `-n auto` workers and concurrent sessions,
|
||||
`finally` does not survive SIGKILL, and a concurrent `git add -A` can stage one. Defects in
|
||||
the test, not in the thing tested.
|
||||
* neutralising the shared primitive and requiring the derivation to go empty. That misses the
|
||||
`.husky/_/` source on any machine where `.husky/_/` does not exist — which is every CI
|
||||
checkout of `script-tests`, which never runs `npm ci`. Green where it runs, red only on a
|
||||
laptop: the inverted asymmetry again, inside the proof written to abolish it.
|
||||
|
||||
Watching for the CALL needs no arranged state: an `rglob` issued while deriving is caught even
|
||||
where the directory it walks is empty, because the evidence is the call rather than what it
|
||||
returned. Reading is untouched, so a derivation may still parse the workflow bodies it scrapes
|
||||
for referenced scripts.
|
||||
|
||||
WHAT IT DOES NOT COVER. The boundary is not "synchronous", which is what two earlier drafts
|
||||
said and what measurement disproved — a thread that outlives the `derive()` call but finishes
|
||||
while its result is being drained IS caught, as is a `__del__` firing during that drain. The
|
||||
boundary is mechanical rather than temporal: a call to one of the SPIES is observed, wherever and
|
||||
whenever it happens in this process before the assertion below. "While the patch is active"
|
||||
under-claims it — the spy appends to a list that outlives the patch, so a reference captured
|
||||
during the window and invoked after it still records. What decides observation is whether the
|
||||
call goes through a spy, not when. The result is drained while the patch is installed, so a lazy
|
||||
generator is reached.
|
||||
|
||||
NOT REACHED, because no spy was ever installed on that path. Enumeration HOISTED TO MODULE SCOPE
|
||||
runs at import, before this test exists — the likeliest instance rather than a contrivance, since
|
||||
`test_ci_image_pin_population.py` already precomputes `_DOC` that way — as does an `atexit` hook,
|
||||
and a cached property warmed by the baseline call below. `from os import listdir` binds the real
|
||||
function before any patch; `from os import walk` IS caught (it routes through the patched
|
||||
`os.scandir`) and so is `from glob import glob` (through the patched `glob.iglob`). And anything
|
||||
listing in ANOTHER PROCESS — a deliberate `subprocess.run(["ls"])`, or a forked child. This is a
|
||||
regression guard against the shapes that arrive by accident, not a sandbox.
|
||||
|
||||
It cuts the other way too: any spy call at all reddens this test, so unrelated background thread
|
||||
activity touching a patched name during the window would too. Nothing in this suite does that
|
||||
today, and the report names the call, so a false red would be diagnosable rather than
|
||||
mysterious.
|
||||
|
||||
It also cannot see a derivation that admits a HARDCODED path without listing anything
|
||||
(`if (REPO_ROOT / "x.sh").exists(): add`) — listing is the commonest way to discover an untracked
|
||||
member, not the only one. The removal proof above catches that shape, and catches memoisation,
|
||||
which this one cannot: the baseline call below warms any cache outside the patch. The two are
|
||||
complements.
|
||||
"""
|
||||
assert derive(), f"{label} derived nothing; this proof needs a baseline"
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
def _spy(what):
|
||||
# RECORDS and returns empty rather than raising. Raising made the assertion "did an
|
||||
# exception reach us", which a derivation defeats by catching it: a `try: ... except
|
||||
# Exception: return set()` around an `rglob` enumerated the filesystem and this test passed,
|
||||
# measured. The evidence is the CALL, so the call is what is asserted on.
|
||||
def spy(*_args, **_kwargs):
|
||||
calls.append(what)
|
||||
# Iterator AND context manager: `os.scandir` is used as `with os.scandir(...) as it`,
|
||||
# and a bare iterator made the failure report carry a test-induced TypeError about the
|
||||
# context manager protocol alongside the real finding. The assertion fired correctly
|
||||
# either way, but a report that blames the harness invites doubting the finding.
|
||||
return _EmptyScan()
|
||||
|
||||
return spy
|
||||
|
||||
failure = None
|
||||
with pytest.MonkeyPatch.context() as m:
|
||||
for owner, name in _ENUMERATORS:
|
||||
m.setattr(owner, name, _spy(f"{getattr(owner, '__name__', owner)}.{name}"), raising=False)
|
||||
try:
|
||||
# DRAINED inside the context, never `derive()` discarded. A derivation returning a lazy
|
||||
# generator does its work when the caller drains it, so discarding the result moved the
|
||||
# whole walk outside the patch: a generator yielding the index population and then
|
||||
# appending `.husky/_` passed here and admitted 17 untracked shims on a checkout where
|
||||
# that directory exists. `_as_relative_strings` is what drains it — it iterates into a
|
||||
# set — so the call must stay here rather than being hoisted out or wrapped in something
|
||||
# lazier.
|
||||
_as_relative_strings(derive())
|
||||
except BaseException as exc: # re-raised below, after the evidence has been judged
|
||||
failure = exc
|
||||
|
||||
assert not calls, (
|
||||
f"{label} enumerated the filesystem via {sorted(set(calls))} while deriving its population. "
|
||||
"Directory listings report build output, generated shims and editor droppings, and differ "
|
||||
"between the CI checkout and a developer's, so the member set stops being a property of the "
|
||||
f"repo (ersatztv#778, #806)." + (f" It also raised: {failure!r}" if failure is not None else "")
|
||||
)
|
||||
if failure is not None:
|
||||
raise failure
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("label", "derive"), _PAIRS, ids=_IDS)
|
||||
def test_every_derived_member_is_tracked(label, derive):
|
||||
"""The same property as an invariant over the real tree, which is the form that catches a
|
||||
refactor going back to a filesystem walk without also touching this file."""
|
||||
tracked = set(tracked_files._git_ls_files())
|
||||
stray = sorted(m for m in _as_relative_strings(derive()) if m not in tracked)
|
||||
assert not stray, f"{label} contains untracked path(s): {stray}"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE SCOPE MIRROR ABOVE IS ITSELF CHECKED
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _modules_importing_the_helper() -> set[str]:
|
||||
"""Which `scripts/tests/test_*.py` import the shared derivation, by PARSING them.
|
||||
|
||||
`ast` rather than a substring scan, and the distinction is the point rather than tidiness. A
|
||||
substring scan over source both EVADES and FALSELY FIRES here: `import scripts.tests.
|
||||
tracked_files as tf` and `from scripts.tests import tracked_files as tf` escape a scan for
|
||||
`"tracked_files import"`, while a comment merely citing `scripts/tests/tracked_files.py` matches
|
||||
a scan for `"tracked_files."` and would redden a correct tree over prose.
|
||||
`docs/decisions/records/testing/guard-derives-population-from-source.md` records why patching
|
||||
such a predicate does not converge: it is not a parser. Python ships the parser, and a comment
|
||||
is not a node at all.
|
||||
|
||||
WHAT THE PARSE DOES NOT REACH, stated rather than implied by the word "parsing": STATIC import
|
||||
statements naming the helper. `importlib.import_module("scripts.tests.tracked_files")`, a
|
||||
re-export through `scripts/tests/__init__.py`, and `from scripts.tests import *` are invisible,
|
||||
verified by executing each. (`from scripts.tests.tracked_files import *` IS seen — it names the
|
||||
module.) Those sit inside the same residual as a module that derives a population without the
|
||||
helper at all — the residual named below — and no mechanical check closes it.
|
||||
|
||||
The FILE LIST is a filesystem walk on purpose, and it is not the defect this file forbids: it is
|
||||
a superset check over what pytest itself collects, so an untracked stray `test_x.py` here makes
|
||||
the guard MORE demanding, never blind. Using the index would let an unstaged new guard escape
|
||||
registration, which is the wrong direction for a check about coverage.
|
||||
"""
|
||||
found = set()
|
||||
for path in sorted(TESTS_DIR.glob("test_*.py")):
|
||||
for node in ast.walk(ast.parse(path.read_text())):
|
||||
# RESOLVE the name to an absolute module and compare exactly. Testing `base ==
|
||||
# "tracked_files"` handled `from .tracked_files import x` but silently missed
|
||||
# `from ..tests.tracked_files import x`, which resolves to the same helper — a false
|
||||
# NEGATIVE, the direction that lets a module adopt the helper and escape registration.
|
||||
# A suffix match instead over-accepts `from unrelated.package import tracked_files`,
|
||||
# reddening a correct tree over a module this repo does not own. Resolution is the only
|
||||
# form with neither failure: these files live in `scripts.tests`, so level 1 resolves to
|
||||
# `scripts.tests` and level 2 to `scripts` — see the guard below for why there is no
|
||||
# level 3.
|
||||
if isinstance(node, ast.Import):
|
||||
hit = any(a.name == _HELPER for a in node.names)
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
if node.level > len(_PACKAGE):
|
||||
# Beyond the top-level package: Python raises ImportError for this, so it cannot
|
||||
# be an import of the helper. Guarded explicitly because `_PACKAGE[:negative]`
|
||||
# silently WRAPS — level 4 produced the same prefix as level 2 — which pinned an
|
||||
# unimportable form as a valid detection. Valid levels here are exactly 1 and 2:
|
||||
# 1 resolves to `scripts.tests`, 2 to `scripts`, and 3 or more is beyond the
|
||||
# top-level package, which Python refuses.
|
||||
continue
|
||||
prefix = _PACKAGE[: len(_PACKAGE) - (node.level - 1)] if node.level else []
|
||||
base_parts = prefix + ([node.module] if node.module else [])
|
||||
base = ".".join(base_parts)
|
||||
hit = base == _HELPER or any(f"{base}.{a.name}" == _HELPER for a in node.names)
|
||||
else:
|
||||
continue
|
||||
if hit:
|
||||
found.add(path.name)
|
||||
break
|
||||
return found
|
||||
|
||||
|
||||
# (source, should the matcher see it). Every row is a form that has actually been mis-classified; the
|
||||
# table is here so the next edit to `_modules_importing_the_helper` cannot re-open one silently.
|
||||
# A false NEGATIVE lets a module adopt the helper and escape registration; a false POSITIVE reddens
|
||||
# a correct tree over a module this repo does not own. Both directions are pinned.
|
||||
_IMPORT_FORMS = (
|
||||
("import scripts.tests.tracked_files as tf", True),
|
||||
("from scripts.tests import tracked_files as tf", True),
|
||||
("from scripts.tests.tracked_files import tracked_paths", True),
|
||||
("from scripts.tests.tracked_files import *", True),
|
||||
("from . import tracked_files", True),
|
||||
("from .tracked_files import tracked_paths", True),
|
||||
("from ..tests.tracked_files import tracked_paths", True),
|
||||
# Beyond the top-level package from `scripts.tests`: Python raises ImportError, so there is
|
||||
# nothing to detect. Pinned False so the negative-slicing wrap that once made it look
|
||||
# detectable cannot come back.
|
||||
("from ...scripts.tests.tracked_files import tracked_paths", False),
|
||||
# Witnesses the negative-slice wrap specifically: without the level guard this one
|
||||
# resolves through `scripts` and matches.
|
||||
("from ....tests.tracked_files import tracked_paths", False),
|
||||
("def f():\n from scripts.tests import tracked_files\n return tracked_files", True),
|
||||
("from unrelated.package import tracked_files", False),
|
||||
("from ..something import tracked_files", False),
|
||||
("from .something import tracked_files", False),
|
||||
("from ..other.tracked_files import x", False),
|
||||
("# see scripts/tests/tracked_files.py for the rationale\nimport re", False),
|
||||
# Pins PARSER VISIBILITY, not importability: executing this line really does import the helper.
|
||||
# The row records that a static parse cannot see it — a known gap, pinned so it is not a
|
||||
# surprise — and closing it would mean updating this row, which is the intended friction.
|
||||
("import importlib\nimportlib.import_module('scripts.tests.tracked_files')", False),
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("source", "expected"), _IMPORT_FORMS, ids=[s.splitlines()[0][:48] for s, _ in _IMPORT_FORMS])
|
||||
def test_the_import_matcher_classifies_every_reviewed_form(source, expected, monkeypatch):
|
||||
class _Fake:
|
||||
name = "test_probe.py"
|
||||
|
||||
def read_text(self):
|
||||
return source
|
||||
|
||||
def __lt__(self, other):
|
||||
return True
|
||||
|
||||
class _Dir:
|
||||
def glob(self, _pattern):
|
||||
return [_Fake()]
|
||||
|
||||
monkeypatch.setattr(sys.modules[__name__], "TESTS_DIR", _Dir())
|
||||
seen = "test_probe.py" in _modules_importing_the_helper()
|
||||
assert seen is expected, f"the import matcher {'missed' if expected else 'falsely matched'} this form:\n{source}"
|
||||
|
||||
|
||||
def test_every_index_derived_module_is_registered():
|
||||
"""`DERIVATIONS` is a hand-written mirror, so it gets an equality check rather than a promise.
|
||||
|
||||
Without this, adding another index-derived guard and forgetting to register it leaves that guard
|
||||
unproven while this file reads as covering them all — a completeness claim standing behind a
|
||||
hand-maintained list, which is the defect one altitude up (#773 Family C, and the shape that put
|
||||
`MARKED_JOBS` in the record as a residual gap).
|
||||
|
||||
THE SCOPE THIS CANNOT SEE, stated because a check described as complete stops being re-examined:
|
||||
it detects modules that IMPORT the shared helper. A module deriving a file population some other
|
||||
way — shelling out to `git ls-files` itself, or going back to `Path.rglob` — is invisible to it,
|
||||
and no mechanical check can close that. `test_guard_inventory.py`'s own header argues the same
|
||||
point about flagging filter-shaped guards by token, and #774 concluded there that the honest
|
||||
answer is no. What is mechanised here is the case that actually recurs: someone adopts the
|
||||
helper and forgets this file.
|
||||
|
||||
Registration is MODULE-level, not derivation-level, so a second population added inside an
|
||||
already-registered module is covered only if it is registered too.
|
||||
|
||||
`POPULATION_EXEMPT` is the opt-out, and it lives HERE rather than as a marker comment in the
|
||||
exempt file because the two directions are not symmetric: a false import-match only reddens,
|
||||
while a false EXEMPTION is silent. A marker a file grants itself by containing a token is
|
||||
trippable from that file's prose — this file's own error message names the token — so it would
|
||||
be a one-line silent kill switch, the shape `test_ci_release_path_scan_job.py` argues against
|
||||
for its recursion fence. Listing exemptions beside the registrations makes adding one a visible
|
||||
edit here.
|
||||
|
||||
WHAT NO ASSERTION CAN DECIDE, dated so it is re-examined rather than assumed: whether an
|
||||
exemption is still WARRANTED. A stale key and an unexplained one are both caught below, but an
|
||||
exempt module that later grows a real derived population stays uncovered and silent. Reviewed
|
||||
2026-08-22 — the single entry uses the helper only to assemble a tmp fixture copy and asserts
|
||||
nothing about membership.
|
||||
"""
|
||||
exempt = {Path(__file__).name} | set(POPULATION_EXEMPT)
|
||||
importers = _modules_importing_the_helper()
|
||||
registered = {label.split(".", 1)[0] + ".py" for label, _, _ in DERIVATIONS}
|
||||
|
||||
# ANTI-VACUITY, and only that. A broken parse is caught loudly by `phantom` below — every
|
||||
# registered module would go missing at once — so this is the cheaper, more specific signal, not
|
||||
# the thing standing between a broken parse and a green run.
|
||||
assert len(importers) >= len(DERIVATIONS), (
|
||||
f"the import parse found only {sorted(importers)}, fewer modules than DERIVATIONS registers "
|
||||
f"({sorted(registered)}) — the parse has broken."
|
||||
)
|
||||
|
||||
stale = sorted(name for name in POPULATION_EXEMPT if name not in importers)
|
||||
assert not stale, (
|
||||
f"POPULATION_EXEMPT lists {stale}, which the parser no longer sees importing the shared "
|
||||
"helper (renamed, "
|
||||
"deleted, or the import removed). A stale exemption is worse than none: if the filename is "
|
||||
"ever reused, the new module is exempt from birth without anyone deciding that."
|
||||
)
|
||||
thin = sorted(name for name, why in POPULATION_EXEMPT.items() if not str(why).strip())
|
||||
assert not thin, f"POPULATION_EXEMPT entries with no stated reason: {thin}"
|
||||
|
||||
unregistered = sorted(importers - registered - exempt)
|
||||
assert not unregistered, (
|
||||
f"{unregistered} import the shared index derivation but are not in DERIVATIONS, so neither "
|
||||
"proof in this file covers them. Add a named derivation function and register it, or add "
|
||||
"the module to POPULATION_EXEMPT in this file if it imports the helper without deriving a "
|
||||
"population."
|
||||
)
|
||||
phantom = sorted(registered - importers)
|
||||
assert not phantom, (
|
||||
f"DERIVATIONS registers {phantom}, which no longer import the shared helper. A row for "
|
||||
"a derivation that is not there reads as coverage and is not."
|
||||
)
|
||||
@@ -2,9 +2,12 @@
|
||||
|
||||
Two separable claims, and conflating them is how instrumentation ships as a regression:
|
||||
|
||||
1. COVERAGE — every hook script records its own execution. The population is DERIVED from
|
||||
`.claude/hooks/*.sh`, never listed, per `testing.guard-derives-population-from-source`, so a
|
||||
hook added tomorrow is uninstrumented-and-red rather than silently unobserved.
|
||||
1. COVERAGE — every hook script records its own execution. The population is DERIVED from the
|
||||
tracked `.claude/hooks/*.sh`, never listed, per `testing.guard-derives-population-from-source`,
|
||||
so a hook staged tomorrow is uninstrumented-and-red rather than silently unobserved. From
|
||||
the git index rather than a filesystem walk since ersatztv#806 — an untracked file on one
|
||||
machine is not part of the repo, and a guard whose population differs per checkout is one
|
||||
nobody trusts.
|
||||
|
||||
2. TRANSPARENCY — the wrapper is invisible to the harness. It slurps stdin and replays it, and it
|
||||
diverts stdout and replays it, which puts it directly in the path of the most load-bearing
|
||||
@@ -34,7 +37,13 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts.tests.tracked_files import tracked_paths
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
# Directory + patterns resolved against the GIT INDEX, not `Path.glob` (ersatztv#806) — see
|
||||
# `scripts/tests/tracked_files.py`. `HOOKS_DIR` survives only for error messages.
|
||||
HOOKS = (".claude/hooks", ("*.sh",))
|
||||
HUSKY = (".husky", ("*",))
|
||||
HOOKS_DIR = REPO_ROOT / ".claude" / "hooks"
|
||||
SINK = REPO_ROOT / "scripts" / "hook-fire-log.sh"
|
||||
|
||||
@@ -46,8 +55,15 @@ _BEGINS = re.compile(r"^etv_hook_fire_begin (\S+) .*\|\| true$", re.M)
|
||||
|
||||
|
||||
def hook_scripts() -> list[Path]:
|
||||
"""THE POPULATION, from the filesystem. Never a list."""
|
||||
return sorted(HOOKS_DIR.glob("*.sh"))
|
||||
"""THE POPULATION, from the GIT INDEX. Never a list, and never the filesystem (ersatztv#806).
|
||||
|
||||
The filesystem is not an authoritative source: an untracked `.sh` dropped in `.claude/hooks/` —
|
||||
a scratch copy, a half-written hook — used to enter this population and be demanded to carry
|
||||
instrumentation, reddening the suite on that checkout while CI, which never sees the file, stayed
|
||||
green. That is #778's third shape, and a guard that fails everywhere except where it runs trains
|
||||
its readers to ignore it.
|
||||
"""
|
||||
return tracked_paths(*HOOKS)
|
||||
|
||||
|
||||
def expected_mode(name: str) -> str:
|
||||
@@ -64,8 +80,8 @@ def expected_mode(name: str) -> str:
|
||||
"""
|
||||
if name in (REPO_ROOT / ".claude" / "settings.json").read_text():
|
||||
return "capture"
|
||||
for husky in (REPO_ROOT / ".husky").iterdir():
|
||||
if husky.is_file() and name in husky.read_text():
|
||||
for husky in tracked_paths(*HUSKY):
|
||||
if name in husky.read_text():
|
||||
return "stream"
|
||||
return "capture"
|
||||
|
||||
@@ -150,7 +166,8 @@ def strip_instrumentation(text: str) -> str:
|
||||
def test_the_population_is_not_empty():
|
||||
hooks = hook_scripts()
|
||||
assert len(hooks) >= 10, (
|
||||
f"only found {len(hooks)} hook scripts under {HOOKS_DIR} — the glob has stopped matching, "
|
||||
f"only found {len(hooks)} hook scripts tracked under {HOOKS_DIR} — the derivation has "
|
||||
"stopped matching, "
|
||||
"so every coverage assertion in this file is vacuous."
|
||||
)
|
||||
assert SINK.exists(), "the shared sink is missing; the instrumentation cannot work"
|
||||
|
||||
@@ -30,6 +30,8 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from scripts.tests.tracked_files import tracked_paths
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
SCRIPT = REPO_ROOT / "scripts" / "pr-changed-files.sh"
|
||||
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
||||
@@ -842,9 +844,7 @@ def test_no_OTHER_workflow_writes_the_review_verdict_status():
|
||||
plainly so nobody reads a green here as "only one writer is possible".
|
||||
"""
|
||||
offenders = []
|
||||
# `*.y*ml`, not `*.yml` — a workflow added as `.yaml` is just as executable and would otherwise
|
||||
# be silently unscanned. Latent today (no `.yaml` exists), which is exactly when it is cheap.
|
||||
for wf in sorted((REPO_ROOT / ".gitea" / "workflows").glob("*.y*ml")):
|
||||
for wf in _workflow_files():
|
||||
if wf.name == WORKFLOW.name:
|
||||
continue
|
||||
# CODE lines only — `docs`-style commentary in a workflow header legitimately discusses the
|
||||
@@ -2505,9 +2505,11 @@ def _yaml_string_scalars(path: Path):
|
||||
|
||||
|
||||
def _workflow_files():
|
||||
# `*.y*ml` for the reason test_no_OTHER_workflow_writes_the_review_verdict_status gives: a
|
||||
# workflow added as `.yaml` is just as executable and would otherwise go unscanned.
|
||||
return sorted((REPO_ROOT / ".gitea" / "workflows").glob("*.y*ml"))
|
||||
# `*.yml` AND `*.yaml`: a workflow added as `.yaml` is just as executable and would otherwise go
|
||||
# unscanned. From the GIT INDEX rather than a directory listing (ersatztv#806) — a stray
|
||||
# untracked `.yaml` in `.gitea/workflows/` is not part of the repo, and enumerating it made
|
||||
# these two completeness claims red on that checkout and green in CI, which is #778's shape.
|
||||
return tracked_paths(".gitea/workflows", ("*.yml", "*.yaml"))
|
||||
|
||||
|
||||
def test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body():
|
||||
|
||||
@@ -34,9 +34,10 @@ from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.tests import tracked_files
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
INVENTORY = REPO_ROOT / "docs" / "remote-state-inventory.md"
|
||||
|
||||
@@ -86,8 +87,9 @@ SCOPE = (
|
||||
# The lesson each time was the same one this repo already wrote down — derive the population from an
|
||||
# AUTHORITATIVE source — and the filesystem is not one. It reports build output, editor droppings and
|
||||
# anything else that happens to be on disk, and it varies per machine. The repo's index is
|
||||
# authoritative, versioned, identical for every checkout and for CI, and it excludes untracked
|
||||
# generated files by construction rather than by an exclusion list that must be maintained.
|
||||
# authoritative: it holds the same set of files every checkout receives from a clone, and it
|
||||
# excludes untracked generated files by construction rather than by an exclusion list that must be
|
||||
# maintained.
|
||||
#
|
||||
# `scripts/tests/` is still excluded explicitly, because those files ARE tracked. That exclusion is a
|
||||
# scope decision, reviewable in one line: they run only under pytest and authorize nothing. Their
|
||||
@@ -104,15 +106,22 @@ _ROW = re.compile(r"^\|\s*`([^`]+?)`[^|]*\|\s*`([^`]+)`\s*\|", re.M)
|
||||
def _tracked_files() -> list[str]:
|
||||
"""Every file git tracks, as repo-relative posix paths.
|
||||
|
||||
Fails LOUDLY rather than returning nothing: an empty population would make every completeness
|
||||
assertion below pass vacuously, which is the exact failure this guard exists to prevent.
|
||||
A thin wrapper over the SHARED derivation rather than a second copy of it (ersatztv#806): one
|
||||
definition of the rule, not two, which is detector C — dedup by construction — applied to the
|
||||
file that first stated the rule. The wrapper survives because this module patches
|
||||
`_tracked_files` by name in its own proofs, and because its scope is RECURSIVE over `scripts/`
|
||||
where `tracked_children` is deliberately flat.
|
||||
|
||||
It therefore carries BOTH its own proofs and a row in that file's `DERIVATIONS`, which is not
|
||||
the duplication that masks: they cut at different seams — the in-file pair patches
|
||||
`_tracked_files` (this wrapper), the shared pair patches `_git_ls_files` (the subprocess) — and
|
||||
each was witnessed red independently, so neither can hide the other's total failure.
|
||||
|
||||
Fails LOUDLY rather than returning nothing, now including git's own stderr: an empty population
|
||||
would make every completeness assertion below pass vacuously, which is the exact failure this
|
||||
guard exists to prevent.
|
||||
"""
|
||||
proc = subprocess.run(
|
||||
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
return [p for p in proc.stdout.decode().split("\0") if p]
|
||||
return tracked_files._git_ls_files()
|
||||
|
||||
|
||||
def derived_population() -> set[str]:
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""The authoritative population for a guard whose members are FILES: the git index (ersatztv#806).
|
||||
|
||||
`testing.guard-derives-population-from-source` (#774) says a completeness guard derives its
|
||||
population from a machine-readable authoritative source, and its worked examples are an enum and the
|
||||
generated OpenAPI document. It does not say what to do when the population is *files*, and every
|
||||
guard in this repo answered that with a filesystem walk. **A filesystem walk is not an authoritative
|
||||
source.** It reports build output, editor droppings and whatever else happens to be on disk, and it
|
||||
differs per machine, so a guard derived from it asserts a different population in CI than on the
|
||||
laptop of the person it is supposed to stop.
|
||||
|
||||
#778 got that wrong three times in one PR, each time with an argument for why the traversal
|
||||
sufficed, and the third shape is the one that motivates this module:
|
||||
|
||||
* a **content filter** on outbound-network tokens that omitted `git fetch`, making a hook that
|
||||
fetches `origin/main` and derives a push decision structurally invisible;
|
||||
* a **non-recursive `Path.glob`**, missing four nested files, one of which calls a live ErsatzTV
|
||||
API and acts on the reply;
|
||||
* **`Path.rglob`**, which then enumerated `.husky/_/` — 17 husky shims generated by `npm ci` via
|
||||
`web/package.json`'s `prepare` script, gitignored (`.husky/_/.gitignore` is `*`) and untracked.
|
||||
That made the guard **RED on every developer checkout and GREEN in CI**, whose `script-tests`
|
||||
job checks out and pip-installs but never runs `npm ci`. A guard that fails everywhere except
|
||||
where it runs teaches its readers to ignore it.
|
||||
|
||||
The index holds the same set of files every checkout receives from a clone, and excludes untracked
|
||||
generated files **by construction** rather than by an exclusion list somebody has to maintain and
|
||||
keep correct. It is not immutable and it is per-worktree; the claim is not that it never changes,
|
||||
but that it changes only through a deliberate git operation — staging, a checkout, a reset, a
|
||||
merge — whereas the disk changes whenever a build runs.
|
||||
Note what that buys over `.gitignore`-awareness: `.husky/_/` happens to carry its own `.gitignore`,
|
||||
but a stray untracked `foo.sh` in `.claude/hooks/` carries nothing, and only the index knows it is
|
||||
not part of the repo.
|
||||
|
||||
**This is not "replace every glob".** The question per guard is whether it makes a COMPLETENESS
|
||||
claim over tracked files. If it does, the index is the authoritative source. If it does not — a
|
||||
fixture copying files into a tmp tree, a walk selecting the SUBJECT of a per-member property — say
|
||||
so in the guard and leave it, per the boundary #774 already draws between scope and population.
|
||||
|
||||
`git ls-files` lists INDEX entries, so a file deleted in the working tree but not yet staged is
|
||||
still reported. That is deliberate: callers that read member contents assert existence with their
|
||||
own message rather than filtering, because filtering is what makes a missing member unrepresentable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _git_ls_files() -> list[str]:
|
||||
"""Every path git tracks, as repo-relative posix strings.
|
||||
|
||||
The single place this package shells out to git, so a regression proof can narrow the tracked
|
||||
set once and have every derivation built on it react.
|
||||
|
||||
Fails LOUDLY on an empty result rather than returning it: an empty population makes every
|
||||
completeness assertion downstream pass vacuously, which is the exact failure these guards exist
|
||||
to prevent.
|
||||
"""
|
||||
proc = subprocess.run(
|
||||
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
# `check=True` would raise "returned non-zero exit status 128" and leave git's own diagnostic
|
||||
# trapped in `e.stderr`. The two real triggers — a tree that is not a repository, and CI's
|
||||
# `detected dubious ownership` — are both diagnosable ONLY from that text, and every module
|
||||
# that imports this one would otherwise die with the same inscrutable line.
|
||||
assert proc.returncode == 0, (
|
||||
f"`git ls-files` failed in {REPO_ROOT} (exit {proc.returncode}). Every file population here "
|
||||
f"comes from the index, so this is fatal rather than empty. git said:\n"
|
||||
f"{proc.stderr.decode(errors='replace').strip() or '(no stderr)'}"
|
||||
)
|
||||
paths = [p for p in proc.stdout.decode().split("\0") if p]
|
||||
assert paths, (
|
||||
f"`git ls-files` reported nothing under {REPO_ROOT} — the derivation is broken, not the "
|
||||
"repo. Every file population built on it would be empty and every completeness assertion "
|
||||
"would pass vacuously."
|
||||
)
|
||||
return paths
|
||||
|
||||
|
||||
def tracked_children(directory: str, patterns: tuple[str, ...]) -> set[str]:
|
||||
"""Tracked files that are DIRECT children of `directory` and match one of `patterns`.
|
||||
|
||||
Direct children only, and that is the point rather than a limitation: every population here is
|
||||
a flat directory (`.claude/hooks/*.sh`, `.husky/*`, `.gitea/workflows/*.yml`,
|
||||
`scripts/tests/test_*.py`), and recursing is what dragged `.husky/_/` in. A guard that genuinely
|
||||
needs a nested population should say so and ask for it explicitly.
|
||||
|
||||
Returns repo-relative posix paths, matching what the guards' inventories and error messages use.
|
||||
"""
|
||||
found: set[str] = set()
|
||||
for path in _git_ls_files():
|
||||
parent, _, name = path.rpartition("/")
|
||||
if parent != directory:
|
||||
continue
|
||||
if any(fnmatch.fnmatch(name, pattern) for pattern in patterns):
|
||||
found.add(path)
|
||||
return found
|
||||
|
||||
|
||||
def tracked_paths(directory: str, patterns: tuple[str, ...]) -> list[Path]:
|
||||
"""`tracked_children` as absolute `Path`s, sorted — for guards that read member contents.
|
||||
|
||||
Existence is ASSERTED, never filtered: a path in the index with no file on disk means the tree
|
||||
is mid-edit, and reporting that is strictly better than silently shrinking the population, which
|
||||
is the defect this module exists to remove.
|
||||
"""
|
||||
paths = []
|
||||
for rel in sorted(tracked_children(directory, patterns)):
|
||||
absolute = REPO_ROOT / rel
|
||||
assert absolute.is_file(), (
|
||||
f"git tracks {rel} but there is no file there. The population comes from the index, so "
|
||||
"a working tree mid-delete is reported rather than silently shrinking the population."
|
||||
)
|
||||
paths.append(absolute)
|
||||
return paths
|
||||
Reference in New Issue
Block a user