Files
ersatztv/docs/decisions/records/testing/guard-derives-population-from-source.md
T
timothy 29d7a06e65
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 27s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m53s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 5m57s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 5m56s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / Build & push image (amd64) (push) Successful in 4m20s
fix(806): guard populations over FILES derive from the git index, not a filesystem walk (#818)
`testing.guard-derives-population-from-source` (#774) was silent on the commonest
population in our own guards — files in a directory — and every one answered with a
filesystem walk. A walk is not authoritative: it reports build output, generated
shims and editor droppings, and differs per machine. #778 measured the cost by
getting the same population wrong three times in one PR.

CONVERTED (a completeness claim over tracked files): `test_guard_inventory.py`,
`test_hook_fire_log.py`, `test_ci_image_pin_population.py` (which also gained
`*.yaml`), `test_remote_state_inventory.py` (folded onto the shared derivation), and
`test_pr_changed_files.py` (not on the issue's list — found by sweeping the whole
repo).

ASSESSED AND RECORDED, not silently skipped: `_repo_copy` takes its file list from
the index for hermeticity though it makes no completeness claim;
`test_ci_dropped_step_guard.py` has no filesystem population at all; the decisions
corpus is recorded as unexamined rather than cleared; and the SPA page-size guard is
deferred to #819 with its obstacle documented. This is not "replace every glob".

`scripts/tests/tracked_files.py` is the single derivation.
`test_guard_populations_derive_from_git.py` proves it in two measured complements:
exhaustive removal catches a hardcoded `.exists()` admit and memoisation; the call
log catches an append-only source that yields nothing on this machine — #778's
shape — which removal cannot see because it has nothing to remove.

Twelve rounds of independent cold review, alternating model families in isolated
worktrees. The production derivations were confirmed sound every round; every
blocking finding after the first was in the proofs or in prose claims about them.
Counts over growing populations were removed rather than corrected, after three
drifted.

fixes #806

Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
2026-08-22 16:07:58 +00:00

18 KiB

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
testing.guard-derives-population-from-source 2026-08-13 — a guard derives its population from the authoritative source and asserts set equality, never a filter and never a sample (#774) active 2026-08-13 none none A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`. WHEN THE POPULATION IS FILES (#806), the authoritative source is the GIT INDEX and never a filesystem walk. A walk is not merely a weaker enumerator, it answers a question about the MACHINE rather than about the repo: it reports build output, generated shims and editor droppings, and it differs between CI and every checkout, so the same guard asserts a different population in each place. Derive with `git ls-files`, take direct children only unless a nested population is stated and wanted, and assert existence rather than filtering on it, because filtering is what makes a missing member unrepresentable. This is an instantiation and not a blanket rewrite: the question per guard remains whether it makes a COMPLETENESS claim over TRACKED files, and a walk that assembles a fixture or selects the SUBJECT of a per-member property stays a walk with its reason written down. guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · file population from the git index · never a filesystem walk · untracked shims redden every checkout · `.husky/_/` · direct children not rglob · paths: `scripts/tests/tracked_files.py`, `scripts/tests/test_guard_populations_derive_from_git.py`, `docs/guard-inventory.md` · issues: #806, #778, #774, #773, #757, #671, #650, #644, #633, #616, #503, #403 Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.

Across the 95 closed issues carrying a ## Closing record, the largest family is one error wearing four costumes: reasoning about a representative instead of the population (#773 §3 Family A — 27 of the 69 process-failure records, 39%). This repo has already invented the cure four times, under four names, and never written it down once. That is Family A operating on our own process.

The three targets it hits, because a rule aimed at only one of them misses. The fix can sample (#671 covered 4 of 10 media types; #403 was silent at 5 of 6 dispatch sites). The verification can sample (#616 and #757 both filtered on the property whose absence WAS the defect; #633's ShouldContain("capped at 100") passed against the text "capped at 1000"). And the guard can sample (#644 grepped for an inflated pageSize, so #650's at-cap truncations were structurally invisible to it). Same mistake, three altitudes.

Why a filter is not merely weaker but structurally blind. A predicate over the population partitions it into examined and excluded. A completeness claim asks whether some member is absent altogether. The absent member is in neither partition — it is not a row the filter rejected, it is a row that was never produced — so no strengthening of the predicate can reach it. This is why the fix is never "tighten the filter"; it is to obtain the population from somewhere that knows the whole of it and compare.

The canonical shape, ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters, does five things and each earns its place. It reads accepted from the generated OpenAPI document rather than from a list in the test. It walks ToolCatalog.All with no filter. It computes BOTH accepted.Except(declared) and declared.Except(accepted), because an unreachable parameter and a phantom one are opposite defects. It carries an anti-vacuity floor, so a walk that reached nothing cannot report success. And it accumulates the whole drift set instead of throwing on the first mismatch — failing fast hands back one member at a time and invites fixing them one at a time, which is how the #754 twin stayed hidden.

The boundary that keeps this rule honest. Every_Query_Parameter_Should_Be_A_Declared_Property sits eight lines away and DOES filter, correctly. Its property is per-member — each declared query parameter must be a declared schema property — and a tool declaring none satisfies it vacuously, so the filtered walk and the full walk assert the same thing. Read the two together before applying this record: the trigger is not the token .Where(, it is a completeness claim standing behind one. A rule stated as "no filters in guards" would flag the correct one and, once waved through twice, stop being read at all.

Scope is not population, and this repo's best guard shows why the distinction matters. scripts/tests/test_ci_dropped_step_guard.py is the model for deriving a population: _marked(job) reads the marked steps out of the parsed workflow, and test_the_guard_expects_EXACTLY_the_set_of_marked_keys_in_the_right_bucket asserts set equality against the guard's own argv. Inside its scope it is exemplary. Its SCOPE, though — MARKED_JOBS = ("test", "migrations") — is a hand-written mirror of the required contexts on main, annotated Read live on 2026-08-10. Those contexts are machine-readable (GET /repos/{owner}/{repo}/branch_protections → status_check_contexts), and nothing in this repo compares the two: grep -rn 'status_check_contexts\|branch_protection' over *.py/*.sh/*.yml returns nothing. Verified 2026-08-13 — the live contexts still match the comment, so this is a latent gap, not a live defect. But add a fourth required context and it acquires no dropped-step guard, silently, and the file that models the rule is the one that cannot report it. A dated comment is a claim about the past; it is not a check.

A weak detector is the symptom-keyed mistake, demonstrated rather than argued. #774 wrote this rule and then, in the same change, built a guard that violated it: a test asserting set equality between the H10 verdict vocabulary's two shell copies, by extracting each with a regex. The authoritative source it should have derived from does not exist — that is the entire problem, and detector C's answer is to delete one copy, not to compare two. Six cold-review rounds each produced another shell construction that either escaped the extractor (an unquoted value, a glob arm, an indented reassignment, a comment-first arm) or made it red on a correct tree (a <word>) state= in a heredoc, a column-zero esac in a string truncating the scope). Every round's fix was locally correct; the sequence never converged, because a regex over shell source is not a shell parser and cannot be made into one.

It was withdrawn, not patched a seventh time. The residue is that the duplication is now unmitigated (#788 removes the second copy, which is the fix that was available all along), and the lesson is the one this record already stated on theory: when the authoritative source is missing, the answer is to create one, never to approximate it with a predicate over text. A guard whose accompanying prose a reviewer can falsify each round is worse than no guard, because — by this record's own argument — a guard described as sound stops being re-examined.

When the population is FILES, the authoritative source is the git index (#806). The worked examples above are an enum and a generated document, both unambiguously authoritative, and the record was silent on the commonest population in this repo's own guards: files in a directory. Every one of them answered with a filesystem walk, and #778 measured what that costs by getting the same population wrong three times in one PR — a content filter that omitted git fetch, a non-recursive glob that missed four nested files, and finally rglob, which enumerated .husky/_/: 17 husky shims generated by npm ci, gitignored and untracked. That last one made the guard red on every developer checkout and green in CI, whose script-tests job pip-installs but never runs npm ci.

The reason a walk keeps losing is not that each traversal was written carelessly; two of the three were the obvious correction to the one before. It is that the disk answers a question about the MACHINE and the guard is asking one about the REPO. Those coincide often enough for a walk to look right and diverge exactly where generated output lands, which is to say wherever the tooling is installed and nowhere else. The index is the repo's own statement of what it contains — the same set of files every checkout receives from a clone, and excluding untracked files by construction rather than by an exclusion list somebody maintains. It is not immutable and it is per-worktree; the claim is not that it never changes, but that it changes only through a deliberate git operation — staging, a checkout, a reset, a merge — whereas the disk changes whenever a build runs. Note what that buys over .gitignore-awareness: .husky/_/ happens to carry its own .gitignore, but a stray foo.sh in .claude/hooks/ carries nothing, and only the index knows it is not part of the repo.

The direction of the failure is worth naming, because it inverts the usual worry about a guard. Under-enumeration hides a defect; this over-enumerated, and reddened correct trees. A guard that fails everywhere except where it runs is not a cautious guard, it is a guard nobody reads — and it had done that to the artifact whose entire thesis is population correctness.

This did not become "replace every glob", and the boundary is the same one drawn above. The question per guard is whether it makes a completeness claim over TRACKED files. _repo_copy in scripts/tests/test_ci_release_path_scan_job.py assembles a fixture and asserts nothing about which files it found; it takes its file list from the index for HERMETICITY, which is a different reason, and its docstring distinguishes the two. scripts/tests/test_ci_dropped_step_guard.py has no filesystem population at all — it reads the parsed workflow. Converting either would have been a change with no defect behind it, which spends the credibility this rule needs when it does bite. The per-guard verdicts, including the two no-change ones and the decisions corpus recorded as unexamined rather than cleared, are tabled in docs/guard-inventory.md.

The residue, named. git ls-files reports INDEX entries, so a guard joins the population when it is STAGED rather than when the file appears. Nothing local runs these checks — .husky/pre-commit runs lint-staged, the decisions guard, the root-PNG check and dotnet format, and no husky hook runs pytest — so the red arrives from pr-checks.yml::script-tests on the PR. A file deleted from the working tree but not yet staged is still listed; tracked_paths asserts existence rather than filtering it out, because a filter is what makes a missing member unrepresentable, which is this record's first paragraph applied to its own implementation.

Two traps specific to converting an existing guard, both of which this rule caught inside its own implementation. FIRST, follow the data to where members are actually ADMITTED, not just to where the walk starts: a scrape that reads its caller files from the index and then admits the paths they name on Path.exists() is half-derived and reads as fully derived. SECOND, a proof that a derivation excludes untracked files must remove EVERY member in turn, not one. derived_guard_files unions four contributors; a single victim is always drawn from whichever sorts first, so a mutant putting only one contributor back on a filesystem walk passes. A sample cannot see the source it did not draw from — this record's opening argument, one level down, inside the artifact written to enforce it. Exhaustive removal is cheap — about a second at the population sizes here.

Removal is only half the property, and the second half must not itself be machine-dependent. A source contributing ONLY untracked members adds and never removes, so a removal-based check has nothing of its to take away: an rglob appending .husky/_/ leaves the removal proof GREEN. Scope that claim on both axes, because it is narrower than it first reads. A broader rglob that also displaces tracked members DOES redden removal, so the blind spot is the append-only shape rather than every filesystem walk; and the append-only shape is blind only where the walked directory yields nothing — with the shims present its members are there to remove and removal reddens too. The gap is therefore an append-only source that is empty ON THIS MACHINE, which is exactly the CI checkout, and exactly where a guard going quiet is invisible.

The obvious complement — arrange an untracked file and require it not to enter — is a trap this change fell into and backed out of twice, and the reason is worth more than the rule. Writing probe files into the checkout under test means a probe in the test_*.py scope is a file pytest COLLECTS mid-session, probe names collide across concurrent runs, finally does not survive a SIGKILL, and a concurrent git add -A can stage one — defects in the test rather than in the thing tested. Neutralising the shared derivation and requiring the population to go empty is clean but misses the .husky/_/ source on any machine where .husky/_/ does not exist, which is the script-tests checkout, since that job never runs npm ci. Both formulations reproduce the green-in-CI / red-on-a-laptop asymmetry this record exists to abolish, inside the proof written to abolish it.

So watch for the property that needs no arranged state: a directory LISTING issued while the derivation runs. Listing is the commonest way a derivation discovers a member the index does not know about, and a walk issued during the derivation is caught on any machine — an rglob fails even where the directory it walks is empty, because the evidence is the call rather than what it returned. State its reach honestly, and state it once: what is observed is any call that goes THROUGH ONE OF THE SPIES, whenever it happens — the check's docstring works through the instances, and this record does not copy them, because the copy drifted from the original inside a single commit. It is a regression guard against the shapes that arrive by accident, not a boundary, and a guard sold as a boundary stops being re-examined.

Three drafts of that one sentence were wrong, all in the same direction, and the third was wrong in the copies after the original had been fixed — which is the completeness rule biting the prose that describes it. "Synchronously inside the call" was falsified by a thread finishing during the drain; "while the patch is active" was falsified by a spy reference captured inside the window and invoked after it, which still records. Each draft named the mechanism the author had in mind rather than the one that decides, and each understated the coverage. Understating is the safe direction and still worth correcting: a limit stated too narrowly invites someone to build the case it appears to exclude. The durable lesson is the one this record already gives for populations — do not keep a second copy of a statement that is still being corrected. Its complement is removal, which catches the shape that admits a HARDCODED path without listing anything (if (REPO_ROOT / "x").exists(): add — the defect this change shipped in its own first round). Neither alone is the property.

What this record does not cover. A population of sites in code — the places that dispatch on a value — has no external enumerator. #403 is that case: PlaybackOrder's values are enumerable, but the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs find-all-references tooling and is tracked in #777, where the relevant language server is currently broken. Do not stretch set equality over it and call the class closed.