ersatztv/docs/decisions/records/testing/guard-derives-population-from-source.md

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics

key	title	status	since	supersedes	superseded-by	rule	signals	mechanics
testing.guard-derives-population-from-source	2026-08-13 — a guard derives its population from the authoritative source and asserts set equality, never a filter and never a sample (#774)	active	2026-08-13	none	none	A guard that asserts a COMPLETENESS property enumerates its population from a machine-readable authoritative source — the enum, the generated OpenAPI document, the parsed workflow YAML, the provider list — and asserts SET EQUALITY in BOTH directions against it. It may not narrow that population with a filter, a `Where`, a `grep` or an early `continue` before the assertion, because a filter cannot see the member that is MISSING: the member whose absence is the defect is precisely the one the predicate excludes. A hand-written literal list of members is the same defect in slower motion — a filter frozen at authoring time, correct on the day it was written and unable to report the day it stopped being. Two boundaries bound the rule rather than weaken it. FIRST, filtering to select the SUBJECT of a PER-MEMBER property is legitimate and is not this defect: the excluded members satisfy the property vacuously, so the filtered walk and the whole walk assert the same thing (`ToolCatalogTests.Every_Query_Parameter_Should_Be_A_Declared_Property` filters to tools that declare query parameters, and a tool declaring none has nothing to check). The defect is filtering the population before a COMPLETENESS claim, which is what makes an absent member unrepresentable (#757 filtered on `QueryParameters is {Count: > 0}` and so could not see a tool that should have declared one and did not). SECOND, a population of VALUES always has an external authoritative source and this rule applies directly; a population of SITES IN CODE has no such list, needs find-all-references tooling, and is tracked separately in #777 — do not stretch a set-equality assertion over it. Distinguish the guard SCOPE (which subsystems it covers — a reviewed policy choice, legitimately hand-written) from the guard POPULATION (the members inside that scope — always derived). When the scope itself MIRRORS an authoritative source, the mirror needs its own equality check or a dated staleness marker, or the guard is complete within a scope that has silently gone stale. The canonical worked example in this repo is `ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters`; the canonical residual gap is `MARKED_JOBS` in `scripts/tests/test_ci_dropped_step_guard.py`.	guard derives population · set equality both directions · never filter never sample · a filter cannot see the missing member · hardcoded list is a frozen filter · missing vs phantom · unreachable vs phantom query parameter · anti-vacuity count guard · accumulate drift do not fail fast · scope versus population · dated mirror of branch protection · `status_check_contexts` · `MARKED_JOBS` · values versus sites-in-code · paths: `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`, `scripts/tests/test_ci_dropped_step_guard.py`, `web/src/api/pageSizeCallSites.guard.test.ts` · issues: #774, #773, #757, #671, #650, #644, #633, #616, #503, #403	Both directions are named separately in the failure message — `missing`/`unreachable` (in the source, absent from the guarded set) and `phantom` (in the guarded set, absent from the source) — because the two are different defects and a single "sets differ" line invites fixing one and re-running.

Across the 95 closed issues carrying a ## Closing record, the largest family is one error wearing four costumes: reasoning about a representative instead of the population (#773 §3 Family A — 27 of the 69 process-failure records, 39%). This repo has already invented the cure four times, under four names, and never written it down once. That is Family A operating on our own process.

The three targets it hits, because a rule aimed at only one of them misses. The fix can sample (#671 covered 4 of 10 media types; #403 was silent at 5 of 6 dispatch sites). The verification can sample (#616 and #757 both filtered on the property whose absence WAS the defect; #633's ShouldContain("capped at 100") passed against the text "capped at 1000"). And the guard can sample (#644 grepped for an inflated pageSize, so #650's at-cap truncations were structurally invisible to it). Same mistake, three altitudes.

Why a filter is not merely weaker but structurally blind. A predicate over the population partitions it into examined and excluded. A completeness claim asks whether some member is absent altogether. The absent member is in neither partition — it is not a row the filter rejected, it is a row that was never produced — so no strengthening of the predicate can reach it. This is why the fix is never "tighten the filter"; it is to obtain the population from somewhere that knows the whole of it and compare.

The canonical shape, ToolCatalogTests.Every_Tool_Should_Declare_Exactly_Its_OpenApi_Query_Parameters, does five things and each earns its place. It reads accepted from the generated OpenAPI document rather than from a list in the test. It walks ToolCatalog.All with no filter. It computes BOTH accepted.Except(declared) and declared.Except(accepted), because an unreachable parameter and a phantom one are opposite defects. It carries an anti-vacuity floor, so a walk that reached nothing cannot report success. And it accumulates the whole drift set instead of throwing on the first mismatch — failing fast hands back one member at a time and invites fixing them one at a time, which is how the #754 twin stayed hidden.

The boundary that keeps this rule honest. Every_Query_Parameter_Should_Be_A_Declared_Property sits eight lines away and DOES filter, correctly. Its property is per-member — each declared query parameter must be a declared schema property — and a tool declaring none satisfies it vacuously, so the filtered walk and the full walk assert the same thing. Read the two together before applying this record: the trigger is not the token .Where(, it is a completeness claim standing behind one. A rule stated as "no filters in guards" would flag the correct one and, once waved through twice, stop being read at all.

Scope is not population, and this repo's best guard shows why the distinction matters. scripts/tests/test_ci_dropped_step_guard.py is the model for deriving a population: _marked(job) reads the marked steps out of the parsed workflow, and test_the_guard_expects_EXACTLY_the_set_of_marked_keys_in_the_right_bucket asserts set equality against the guard's own argv. Inside its scope it is exemplary. Its SCOPE, though — MARKED_JOBS = ("test", "migrations") — is a hand-written mirror of the required contexts on main, annotated Read live on 2026-08-10. Those contexts are machine-readable (GET /repos/{owner}/{repo}/branch_protections → status_check_contexts), and nothing in this repo compares the two: grep -rn 'status_check_contexts\|branch_protection' over *.py/*.sh/*.yml returns nothing. Verified 2026-08-13 — the live contexts still match the comment, so this is a latent gap, not a live defect. But add a fourth required context and it acquires no dropped-step guard, silently, and the file that models the rule is the one that cannot report it. A dated comment is a claim about the past; it is not a check.

A weak detector is the symptom-keyed mistake, demonstrated rather than argued. #774 wrote this rule and then, in the same change, built a guard that violated it: a test asserting set equality between the H10 verdict vocabulary's two shell copies, by extracting each with a regex. The authoritative source it should have derived from does not exist — that is the entire problem, and detector C's answer is to delete one copy, not to compare two. Six cold-review rounds each produced another shell construction that either escaped the extractor (an unquoted value, a glob arm, an indented reassignment, a comment-first arm) or made it red on a correct tree (a <word>) state= in a heredoc, a column-zero esac in a string truncating the scope). Every round's fix was locally correct; the sequence never converged, because a regex over shell source is not a shell parser and cannot be made into one.

It was withdrawn, not patched a seventh time. The residue is that the duplication is now unmitigated (#788 removes the second copy, which is the fix that was available all along), and the lesson is the one this record already stated on theory: when the authoritative source is missing, the answer is to create one, never to approximate it with a predicate over text. A guard whose accompanying prose a reviewer can falsify each round is worse than no guard, because — by this record's own argument — a guard described as sound stops being re-examined.

What this record does not cover. A population of sites in code — the places that dispatch on a value — has no external enumerator. #403 is that case: PlaybackOrder's values are enumerable, but the defect was 5 of 6 dispatch sites, and nothing lists dispatch sites. That residue needs find-all-references tooling and is tracked in #777, where the relevant language server is currently broken. Do not stretch set equality over it and call the class closed.