`testing.guard-derives-population-from-source` (#774) was silent on the commonest population in our own guards — files in a directory — and every one answered with a filesystem walk. A walk is not authoritative: it reports build output, generated shims and editor droppings, and differs per machine. #778 measured the cost by getting the same population wrong three times in one PR. CONVERTED (a completeness claim over tracked files): `test_guard_inventory.py`, `test_hook_fire_log.py`, `test_ci_image_pin_population.py` (which also gained `*.yaml`), `test_remote_state_inventory.py` (folded onto the shared derivation), and `test_pr_changed_files.py` (not on the issue's list — found by sweeping the whole repo). ASSESSED AND RECORDED, not silently skipped: `_repo_copy` takes its file list from the index for hermeticity though it makes no completeness claim; `test_ci_dropped_step_guard.py` has no filesystem population at all; the decisions corpus is recorded as unexamined rather than cleared; and the SPA page-size guard is deferred to #819 with its obstacle documented. This is not "replace every glob". `scripts/tests/tracked_files.py` is the single derivation. `test_guard_populations_derive_from_git.py` proves it in two measured complements: exhaustive removal catches a hardcoded `.exists()` admit and memoisation; the call log catches an append-only source that yields nothing on this machine — #778's shape — which removal cannot see because it has nothing to remove. Twelve rounds of independent cold review, alternating model families in isolated worktrees. The production derivations were confirmed sound every round; every blocking finding after the first was in the proofs or in prose claims about them. Counts over growing populations were removed rather than corrected, after three drifted. fixes #806 Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
18 KiB
key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
| key | title | status | since | supersedes | superseded-by | rule | signals | mechanics |
|---|---|---|---|---|---|---|---|---|
| testing.guard-derives-population-from-source | 2026-08-13 — a guard derives its population from the authoritative source and asserts set equality, never a filter and never a sample (#774) | active | 2026-08-13 | none | none | A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. | guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403 | Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running. |
Across the 95 closed issues carrying a ## Closing record, the largest family is one error wearing
four costumes: reasoning about a representative instead of the population (#773 §3 Family A — 27
of the 69 process-failure records, 39%). This repo has already invented the cure four times, under
four names, and never written it down once. That is Family A operating on our own process.
The three targets it hits, because a rule aimed at only one of them misses. The fix can sample
(#671 covered 4 of 10 media types; #403 was silent at 5 of 6 dispatch sites). The verification can
sample (#616 and #757 both filtered on the property whose absence WAS the defect; #633's
ShouldContain("capped at 100") passed against the text "capped at 1000"). And the guard can
sample (#644 grepped for an inflated pageSize, so #650's at-cap truncations were structurally
invisible to it). Same mistake, three altitudes.
Why a filter is not merely weaker but structurally blind. A predicate over the population partitions it into examined and excluded. A completeness claim asks whether some member is absent altogether. The absent member is in neither partition — it is not a row the filter rejected, it is a row that was never produced — so no strengthening of the predicate can reach it. This is why the fix is never "tighten the filter"; it is to obtain the population from somewhere that knows the whole of it and compare.
The canonical shape, ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters,
does five things and each earns its place. It reads accepted from the generated OpenAPI document
rather than from a list in the test. It walks ToolCatalog.All with no filter. It computes BOTH
accepted.Except(declared) and declared.Except(accepted), because an unreachable parameter and a
phantom one are opposite defects. It carries an anti-vacuity floor, so a walk that reached nothing
cannot report success. And it accumulates the whole drift set instead of throwing on the first
mismatch — failing fast hands back one member at a time and invites fixing them one at a time,
which is how the #754 twin stayed hidden.
The boundary that keeps this rule honest. Every_Query_Parameter_Should_Be_A_Declared_Property
sits eight lines away and DOES filter, correctly. Its property is per-member — each declared query
parameter must be a declared schema property — and a tool declaring none satisfies it vacuously, so
the filtered walk and the full walk assert the same thing. Read the two together before applying
this record: the trigger is not the token .Where(, it is a completeness claim standing behind one.
A rule stated as "no filters in guards" would flag the correct one and, once waved through twice,
stop being read at all.
Scope is not population, and this repo's best guard shows why the distinction matters.
scripts/tests/test_ci_dropped_step_guard.py is the model for deriving a population: _marked(job)
reads the marked steps out of the parsed workflow, and
test_the_guard_expects_EXACTLY_the_set_of_marked_keys_in_the_right_bucket asserts set equality
against the guard's own argv. Inside its scope it is exemplary. Its SCOPE, though —
MARKED_JOBS = ("test", "migrations") — is a hand-written mirror of the required contexts on
main, annotated Read live on 2026-08-10. Those contexts are machine-readable
(GET /repos/{owner}/{repo}/branch_protections → status_check_contexts), and nothing in this repo
compares the two: grep -rn 'status_check_contexts\|branch_protection' over *.py/*.sh/*.yml
returns nothing. Verified 2026-08-13 — the live contexts still match the comment, so this is a
latent gap, not a live defect. But add a fourth required context and it acquires no dropped-step
guard, silently, and the file that models the rule is the one that cannot report it. A dated comment
is a claim about the past; it is not a check.
A weak detector is the symptom-keyed mistake, demonstrated rather than argued. #774 wrote this
rule and then, in the same change, built a guard that violated it: a test asserting set equality
between the H10 verdict vocabulary's two shell copies, by extracting each with a regex. The
authoritative source it should have derived from does not exist — that is the entire problem, and
detector C's answer is to delete one copy, not to compare two. Six cold-review rounds each produced
another shell construction that either escaped the extractor (an unquoted value, a glob arm, an
indented reassignment, a comment-first arm) or made it red on a correct tree (a <word>) state= in
a heredoc, a column-zero esac in a string truncating the scope). Every round's fix was locally
correct; the sequence never converged, because a regex over shell source is not a shell parser and
cannot be made into one.
It was withdrawn, not patched a seventh time. The residue is that the duplication is now unmitigated (#788 removes the second copy, which is the fix that was available all along), and the lesson is the one this record already stated on theory: when the authoritative source is missing, the answer is to create one, never to approximate it with a predicate over text. A guard whose accompanying prose a reviewer can falsify each round is worse than no guard, because — by this record's own argument — a guard described as sound stops being re-examined.
When the population is FILES, the authoritative source is the git index (#806). The worked
examples above are an enum and a generated document, both unambiguously authoritative, and the
record was silent on the commonest population in this repo's own guards: files in a directory. Every
one of them answered with a filesystem walk, and #778 measured what that costs by getting the same
population wrong three times in one PR — a content filter that omitted git fetch, a non-recursive
glob that missed four nested files, and finally rglob, which enumerated .husky/_/: 17 husky
shims generated by npm ci, gitignored and untracked. That last one made the guard red on every
developer checkout and green in CI, whose script-tests job pip-installs but never runs npm ci.
The reason a walk keeps losing is not that each traversal was written carelessly; two of the three
were the obvious correction to the one before. It is that the disk answers a question about the
MACHINE and the guard is asking one about the REPO. Those coincide often enough for a walk to look
right and diverge exactly where generated output lands, which is to say wherever the tooling is
installed and nowhere else. The index is the repo's own statement of what it contains — the same
set of files every checkout receives from a clone, and excluding untracked files by construction
rather than by an exclusion list somebody maintains. It is not immutable and it is per-worktree; the
claim is not that it never changes, but that it changes only through a deliberate git operation — staging, a checkout, a
reset, a merge — whereas the disk changes whenever a build runs. Note what that buys over .gitignore-awareness: .husky/_/
happens to carry its own .gitignore, but a stray foo.sh in .claude/hooks/ carries nothing, and
only the index knows it is not part of the repo.
The direction of the failure is worth naming, because it inverts the usual worry about a guard. Under-enumeration hides a defect; this over-enumerated, and reddened correct trees. A guard that fails everywhere except where it runs is not a cautious guard, it is a guard nobody reads — and it had done that to the artifact whose entire thesis is population correctness.
This did not become "replace every glob", and the boundary is the same one drawn above. The
question per guard is whether it makes a completeness claim over TRACKED files. _repo_copy in
scripts/tests/test_ci_release_path_scan_job.py assembles a fixture and asserts nothing about
which files it found; it takes its file list from the index for HERMETICITY, which is a different
reason, and its docstring distinguishes the two.
scripts/tests/test_ci_dropped_step_guard.py has no filesystem population at all — it reads the
parsed workflow. Converting either would have been a change with no defect behind it, which spends
the credibility this rule needs when it does bite. The per-guard verdicts, including the two
no-change ones and the decisions corpus recorded as unexamined rather than cleared, are tabled in
docs/guard-inventory.md.
The residue, named. git ls-files reports INDEX entries, so a guard joins the population when
it is STAGED rather than when the file appears. Nothing local runs these checks — .husky/pre-commit
runs lint-staged, the decisions guard, the root-PNG check and dotnet format, and no husky hook runs
pytest — so the red arrives from pr-checks.yml::script-tests on the PR. A file deleted from the
working tree but not yet staged is still listed; tracked_paths asserts existence rather than
filtering it out, because a filter is what makes a missing member unrepresentable, which is this
record's first paragraph applied to its own implementation.
Two traps specific to converting an existing guard, both of which this rule caught inside its own
implementation. FIRST, follow the data to where members are actually ADMITTED, not just to where
the walk starts: a scrape that reads its caller files from the index and then admits the paths they
name on Path.exists() is half-derived and reads as fully derived. SECOND, a proof that a
derivation excludes untracked files must remove EVERY member in turn, not one. derived_guard_files
unions four contributors; a single victim is always drawn from whichever sorts first, so a mutant
putting only one contributor back on a filesystem walk passes. A sample cannot see the source it did
not draw from — this record's opening argument, one level down, inside the artifact written to
enforce it. Exhaustive removal is cheap — about a second at the population sizes here.
Removal is only half the property, and the second half must not itself be machine-dependent. A
source contributing ONLY untracked members adds and never removes, so a removal-based check has
nothing of its to take away: an rglob appending .husky/_/ leaves the removal proof GREEN. Scope
that claim on both axes, because it is narrower than it first reads. A broader rglob that also
displaces tracked members DOES redden removal, so the blind spot is the append-only shape rather
than every filesystem walk; and the append-only shape is blind only where the walked directory
yields nothing — with the shims present its members are there to remove and removal reddens too. The
gap is therefore an append-only source that is empty ON THIS MACHINE, which is exactly the CI
checkout, and exactly where a guard going quiet is invisible.
The obvious complement — arrange an untracked file and require it not to enter — is a trap this
change fell into and backed out of twice, and the reason is worth more than the rule. Writing probe
files into the checkout under test means a probe in the test_*.py scope is a file pytest COLLECTS
mid-session, probe names collide across concurrent runs, finally does not survive a SIGKILL, and a
concurrent git add -A can stage one — defects in the test rather than in the thing tested.
Neutralising the shared derivation and requiring the population to go empty is clean but misses the
.husky/_/ source on any machine where .husky/_/ does not exist, which is the script-tests
checkout, since that job never runs npm ci. Both formulations reproduce the green-in-CI /
red-on-a-laptop asymmetry this record exists to abolish, inside the proof written to abolish it.
So watch for the property that needs no arranged state: a directory LISTING issued while the
derivation runs. Listing is the commonest way a derivation discovers a member the index does not
know about, and a walk issued during the derivation is caught on any machine — an rglob fails even
where the directory it walks is empty, because the evidence is the call rather than what it
returned. State its reach honestly, and state it once: what is observed is any
call that goes THROUGH ONE OF THE SPIES, whenever it happens — the check's docstring works through
the instances, and this record does not copy them, because the copy drifted from the original inside
a single commit. It is a regression guard against the shapes that arrive by accident, not a
boundary, and a guard sold as a boundary stops being re-examined.
Three drafts of that one sentence were wrong, all in the same direction, and the third was wrong
in the copies after the original had been fixed — which is the completeness rule biting the prose
that describes it. "Synchronously inside the call" was falsified by a thread finishing during the
drain; "while the patch is active" was falsified by a spy reference captured inside the window and
invoked after it, which still records. Each draft named the mechanism the author had in mind rather
than the one that decides, and each understated the coverage. Understating is the safe direction and
still worth correcting: a limit stated too narrowly invites someone to build the case it appears to
exclude. The durable lesson is the one this record already gives for populations — do not keep a
second copy of a statement that is still being corrected. Its complement is removal, which catches the shape that admits a HARDCODED
path without listing anything (if (REPO_ROOT / "x").exists(): add — the defect this change shipped
in its own first round). Neither alone is the property.
What this record does not cover. A population of sites in code — the places that dispatch on a
value — has no external enumerator. #403 is that case: PlaybackOrder's values are enumerable, but
the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs
find-all-references tooling and is tracked in #777, where the relevant language server is currently
broken. Do not stretch set equality over it and call the class closed.