ersatztv/docs/decisions/records/testing/guard-derives-population-from-source.md

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics

key	title	status	since	supersedes	superseded-by	rule	signals	mechanics
testing.guard-derives-population-from-source	2026-08-13 — a guard derives its population from the authoritative source and asserts set equality, never a filter and never a sample (#774)	active	2026-08-13	none	none	A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check against that source, or the guard is complete within a scope that has silently gone stale. A DATED STALENESS MARKER does not discharge this and is no longer offered as an alternative (#787): a date records when someone last looked, so it goes stale in exactly the circumstance it exists to report, and nothing reddens when it does. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`. When a scope mirrors an authoritative source across a CREDENTIAL BOUNDARY, split the invariant at that boundary rather than dating a comment: commit the mirror verbatim, derive the scope from it offline, and reconcile the mirror against the live source wherever a credential exists — `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py` was this repo's residual gap until #787 closed it that way. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. ONE BOUNDED EXCEPTION to that existence half (#819): a guard MAY filter on existence where the OTHER direction reports the on-disk hole — where a tracked path that never reached the walk is itself compared and NAMED — since what makes filtering dangerous is the absent member becoming unrepresentable, and a paired assertion represents it. That buys tolerance of a state the assert-existence form cannot distinguish, which matters where the population is large and edited continuously and an unstaged deletion is routine rather than remarkable. Filtering on existence WITHOUT that paired direction stays the original defect. AND, where a guard PROVES a predicate by restating it (#819), that restatement must be CLOSED FORM over its raw input: it may share NO helper, at ANY depth, with the predicate it checks, and the population it reads must be cross-checked against an INDEPENDENTLY derived list. Anything shared sits on both sides of the comparison and cancels, so a narrowing there shrinks both and passes — five review rounds each fixed one shared thing and left the next: the scope shared between the two sides being compared, then an example table, then a delegated sub-predicate, then a `basename` helper, then the population array itself. Where the population comes from a derivation you also own, cross-check it against a SECOND query of that source rather than restating anything — and accept that a derivation lying consistently in both outputs is answerable only by testing the derivation directly. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down.	guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · scope mirror across a credential boundary · split the invariant at the boundary · synthesise contexts do not parse them · one classifier two callers · paths: `.gitea/required-status-contexts.json`, `scripts/check-required-contexts.sh`, `scripts/lib/branch-rule-classifier.jq` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · bounded exception: filter on existence only when the other direction NAMES the on-disk hole · unstaged deletion is routine in a large edited population · a restatement proving a scope must be CLOSED FORM, sharing no helper with the predicate it checks · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #819, #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403	Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.

Across the 95 closed issues carrying a ## Closing record, the largest family is one error wearing four costumes: reasoning about a representative instead of the population (#773 §3 Family A — 27 of the 69 process-failure records, 39%). This repo has already invented the cure four times, under four names, and never written it down once. That is Family A operating on our own process.

The three targets it hits, because a rule aimed at only one of them misses. The fix can sample (#671 covered 4 of 10 media types; #403 was silent at 5 of 6 dispatch sites). The verification can sample (#616 and #757 both filtered on the property whose absence WAS the defect; #633's ShouldContain("capped at 100") passed against the text "capped at 1000"). And the guard can sample (#644 grepped for an inflated pageSize, so #650's at-cap truncations were structurally invisible to it). Same mistake, three altitudes.

Why a filter is not merely weaker but structurally blind. A predicate over the population partitions it into examined and excluded. A completeness claim asks whether some member is absent altogether. The absent member is in neither partition — it is not a row the filter rejected, it is a row that was never produced — so no strengthening of the predicate can reach it. This is why the fix is never "tighten the filter"; it is to obtain the population from somewhere that knows the whole of it and compare.

The canonical shape, ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters, does five things and each earns its place. It reads accepted from the generated OpenAPI document rather than from a list in the test. It walks ToolCatalog.All with no filter. It computes BOTH accepted.Except(declared) and declared.Except(accepted), because an unreachable parameter and a phantom one are opposite defects. It carries an anti-vacuity floor, so a walk that reached nothing cannot report success. And it accumulates the whole drift set instead of throwing on the first mismatch — failing fast hands back one member at a time and invites fixing them one at a time, which is how the #754 twin stayed hidden.

The boundary that keeps this rule honest. Every_Query_Parameter_Should_Be_A_Declared_Property sits eight lines away and DOES filter, correctly. Its property is per-member — each declared query parameter must be a declared schema property — and a tool declaring none satisfies it vacuously, so the filtered walk and the full walk assert the same thing. Read the two together before applying this record: the trigger is not the token .Where(, it is a completeness claim standing behind one. A rule stated as "no filters in guards" would flag the correct one and, once waved through twice, stop being read at all.

Scope is not population, and a scope that MIRRORS a source needs its own check. scripts/tests/test_ci_dropped_step_guard.py models deriving a population: _marked(job) reads the marked steps out of the parsed workflow, and test_the_guard_expects_EXACTLY_the_set_of_marked_keys_in_the_right_bucket asserts set equality against the guard's own argv. Its SCOPE was long this repo's counter-example — MARKED_JOBS = ("test", "migrations"), a hand-written mirror of the required contexts on main annotated Read live on 2026-08-10, which nothing compared against the server. A dated comment is a claim about the past; it is not a check.

#787 closed it, and the shape generalises to any scope mirroring an authoritative source across a CREDENTIAL BOUNDARY. Split the invariant at that boundary and check each half where it can be checked: commit the mirror verbatim from the source (.gitea/required-status-contexts.json), derive the scope from that artifact OFFLINE — pr-checks.yml::script-tests checks out with persist-credentials: false and cannot ask Gitea — and reconcile the artifact against the live source wherever a credential does exist (scripts/check-required-contexts.sh, called from the merge-consent hook on both merge paths). Neither half is checkable alone; composed, they cover the gap — but say what the reconciliation actually is: ONE client-side hook, so a merge from the web UI, from a session without the hook, or without credentials never reconciles, and the residual is bounded staleness rather than none. This is still why "a snapshot only moves the hand-maintenance" is wrong here: the maintenance becomes a verbatim copy with an equality check on each side, and updating it to ADD a context makes CI red until the newly required job is marked or its disposition recorded. The direction is asymmetric and the asymmetry is the residual — hand-NARROWING the snapshot narrows the derived scope and stays green, exactly as the literal it replaced did.

Three details carry it. The derivation SYNTHESISES each candidate context string from the workflow (<workflow name> / <job name> (<event>)) and matches by equality rather than parsing, because the delimiters occur inside the names they delimit — Build & test (.NET) ends in a parenthesis of its own. A required context that maps to neither a job nor an explicit disposition is a HARD failure at import, which is what makes a fourth required check impossible to add silently. And the reconciliation refuses rather than guesses when a glob rule could govern the branch, delegating that question to the ONE classifier (scripts/lib/branch-rule-classifier.jq) the merge hook also loads — extracted, not copied, because two copies of a security predicate drift and the copy without authority is historically the one that goes stale.

A weak detector is the symptom-keyed mistake, demonstrated rather than argued. #774 wrote this rule and then, in the same change, built a guard that violated it: a test asserting set equality between the H10 verdict vocabulary's two shell copies, by extracting each with a regex. The authoritative source it should have derived from does not exist — that is the entire problem, and detector C's answer is to delete one copy, not to compare two. Six cold-review rounds each produced another shell construction that either escaped the extractor (an unquoted value, a glob arm, an indented reassignment, a comment-first arm) or made it red on a correct tree (a <word>) state= in a heredoc, a column-zero esac in a string truncating the scope). Every round's fix was locally correct; the sequence never converged, because a regex over shell source is not a shell parser and cannot be made into one.

It was withdrawn, not patched a seventh time. The residue is that the duplication is now unmitigated (#788 removes the second copy, which is the fix that was available all along), and the lesson is the one this record already stated on theory: when the authoritative source is missing, the answer is to create one, never to approximate it with a predicate over text. A guard whose accompanying prose a reviewer can falsify each round is worse than no guard, because — by this record's own argument — a guard described as sound stops being re-examined.

When the population is FILES, the authoritative source is the git index (#806). The worked examples above are an enum and a generated document, both unambiguously authoritative, and the record was silent on the commonest population in this repo's own guards: files in a directory. Every one of them answered with a filesystem walk, and #778 measured what that costs by getting the same population wrong three times in one PR — a content filter that omitted git fetch, a non-recursive glob that missed four nested files, and finally rglob, which enumerated .husky/_/: 17 husky shims generated by npm ci, gitignored and untracked. That last one made the guard red on every developer checkout and green in CI, whose script-tests job pip-installs but never runs npm ci.

The reason a walk keeps losing is not that each traversal was written carelessly; two of the three were the obvious correction to the one before. It is that the disk answers a question about the MACHINE and the guard is asking one about the REPO. Those coincide often enough for a walk to look right and diverge exactly where generated output lands, which is to say wherever the tooling is installed and nowhere else. The index is the repo's own statement of what it contains — the same set of files every checkout receives from a clone, and excluding untracked files by construction rather than by an exclusion list somebody maintains. It is not immutable and it is per-worktree; the claim is not that it never changes, but that it changes only through a deliberate git operation — staging, a checkout, a reset, a merge — whereas the disk changes whenever a build runs. Note what that buys over .gitignore-awareness: .husky/_/ happens to carry its own .gitignore, but a stray foo.sh in .claude/hooks/ carries nothing, and only the index knows it is not part of the repo.

The direction of the failure is worth naming, because it inverts the usual worry about a guard. Under-enumeration hides a defect; this over-enumerated, and reddened correct trees. A guard that fails everywhere except where it runs is not a cautious guard, it is a guard nobody reads — and it had done that to the artifact whose entire thesis is population correctness.

This did not become "replace every glob", and the boundary is the same one drawn above. The question per guard is whether it makes a completeness claim over TRACKED files. _repo_copy in scripts/tests/test_ci_release_path_scan_job.py assembles a fixture and asserts nothing about which files it found; it takes its file list from the index for HERMETICITY, which is a different reason, and its docstring distinguishes the two. scripts/tests/test_ci_dropped_step_guard.py has no filesystem population at all — it reads the parsed workflow. Converting either would have been a change with no defect behind it, which spends the credibility this rule needs when it does bite. The per-guard verdicts, including the two no-change ones and the decisions corpus recorded as unexamined rather than cleared, are tabled in docs/guard-inventory.md.

The residue, named. git ls-files reports INDEX entries, so a guard joins the population when it is STAGED rather than when the file appears. Nothing local runs these checks — .husky/pre-commit runs lint-staged, the decisions guard, the root-PNG check and dotnet format, and no husky hook runs pytest — so the red arrives from pr-checks.yml::script-tests on the PR. A file deleted from the working tree but not yet staged is still listed; tracked_paths asserts existence rather than filtering it out, because a filter is what makes a missing member unrepresentable, which is this record's first paragraph applied to its own implementation.

One bounded exception to that existence rule, added 2026-08-29 (#819). A guard MAY filter on existence instead of asserting it where the OTHER direction reports the on-disk hole — that is, where a tracked path missing from the walk is itself compared and named. What makes filtering dangerous is that the absent member becomes unrepresentable; when a second assertion represents it, the objection does not apply, and the filter buys tolerance of a state the assert-existence form cannot distinguish. web/src/api/pageSizeCallSites.guard.test.ts is the case: it runs over every tracked path under web/src/ (~260, of which ~141 are in scope for the scanner) in a tree a developer edits continuously, where an unstaged deletion is routine rather than remarkable, and a guard red on one is a guard its readers learn to ignore (#806). It subtracts the absent set and then asserts that every tracked, on-disk, in-scope path reached the scanner — so a dotfile, or a disk/index spelling divergence under core.ignorecase or NFD/NFC, is NAMED rather than silently dropped. The exception is that pairing, not the filter alone: filtering on existence WITHOUT that second direction is the original defect and stays forbidden.

The closed-form criterion for a restatement proof, added 2026-08-29 (#819). Where a completeness claim rests on a POLICY predicate — which files are in scope — there is no external source to derive it from, so the only available proof is to restate the predicate and require the two to agree over the derived population. That proof is worth nothing unless it is CLOSED FORM: it may share no helper, at any depth, with the predicate it checks, and the population it reads must itself be cross-checked against an independently derived list. Anything shared appears on both sides of the comparison and cancels, so narrowing it shrinks both sides and the check passes while real members leave the population. Not hypothetical: #819 took five review rounds, each fixing one shared thing and leaving the next — first the scope itself, shared between the walk side and the index side; then a table of example paths; then a delegated sub-predicate (isScannableSourceFileName); then a basename helper that read as plumbing rather than as policy; then the tracked-file array every comparison was derived from. Each intermediate version dropped real files, and two left the guard reporting success over a planted defect. State the criterion where the next author edits, because "restate the scope" gives no way to tell when you are done — and prefer deriving a second list from a genuinely different query (ls-files --others beside ls-files) over restating anything at all.

Two traps specific to converting an existing guard, both of which this rule caught inside its own implementation. FIRST, follow the data to where members are actually ADMITTED, not just to where the walk starts: a scrape that reads its caller files from the index and then admits the paths they name on Path.exists() is half-derived and reads as fully derived. SECOND, a proof that a derivation excludes untracked files must remove EVERY member in turn, not one. derived_guard_files unions four contributors; a single victim is always drawn from whichever sorts first, so a mutant putting only one contributor back on a filesystem walk passes. A sample cannot see the source it did not draw from — this record's opening argument, one level down, inside the artifact written to enforce it. Exhaustive removal is cheap — about a second at the population sizes here.

Removal is only half the property, and the second half must not itself be machine-dependent. A source contributing ONLY untracked members adds and never removes, so a removal-based check has nothing of its to take away: an rglob appending .husky/_/ leaves the removal proof GREEN. Scope that claim on both axes, because it is narrower than it first reads. A broader rglob that also displaces tracked members DOES redden removal, so the blind spot is the append-only shape rather than every filesystem walk; and the append-only shape is blind only where the walked directory yields nothing — with the shims present its members are there to remove and removal reddens too. The gap is therefore an append-only source that is empty ON THIS MACHINE, which is exactly the CI checkout, and exactly where a guard going quiet is invisible.

The obvious complement — arrange an untracked file and require it not to enter — is a trap this change fell into and backed out of twice, and the reason is worth more than the rule. Writing probe files into the checkout under test means a probe in the test_*.py scope is a file pytest COLLECTS mid-session, probe names collide across concurrent runs, finally does not survive a SIGKILL, and a concurrent git add -A can stage one — defects in the test rather than in the thing tested. Neutralising the shared derivation and requiring the population to go empty is clean but misses the .husky/_/ source on any machine where .husky/_/ does not exist, which is the script-tests checkout, since that job never runs npm ci. Both formulations reproduce the green-in-CI / red-on-a-laptop asymmetry this record exists to abolish, inside the proof written to abolish it.

So watch for the property that needs no arranged state: a directory LISTING issued while the derivation runs. Listing is the commonest way a derivation discovers a member the index does not know about, and a walk issued during the derivation is caught on any machine — an rglob fails even where the directory it walks is empty, because the evidence is the call rather than what it returned. State its reach honestly, and state it once: what is observed is any call that goes THROUGH ONE OF THE SPIES, whenever it happens — the check's docstring works through the instances, and this record does not copy them, because the copy drifted from the original inside a single commit. It is a regression guard against the shapes that arrive by accident, not a boundary, and a guard sold as a boundary stops being re-examined.

Three drafts of that one sentence were wrong, all in the same direction, and the third was wrong in the copies after the original had been fixed — which is the completeness rule biting the prose that describes it. "Synchronously inside the call" was falsified by a thread finishing during the drain; "while the patch is active" was falsified by a spy reference captured inside the window and invoked after it, which still records. Each draft named the mechanism the author had in mind rather than the one that decides, and each understated the coverage. Understating is the safe direction and still worth correcting: a limit stated too narrowly invites someone to build the case it appears to exclude. The durable lesson is the one this record already gives for populations — do not keep a second copy of a statement that is still being corrected. Its complement is removal, which catches the shape that admits a HARDCODED path without listing anything (if (REPO_ROOT / "x").exists(): add — the defect this change shipped in its own first round). Neither alone is the property.

What this record does not cover. A population of sites in code — the places that dispatch on a value — has no external enumerator. #403 is that case: PlaybackOrder's values are enumerable, but the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs find-all-references tooling and is tracked in #777, where the relevant language server is currently broken. Do not stretch set equality over it and call the class closed.