#773's analysis found that the largest recorded failure family is reasoning about a
representative instead of the population (39% of process-failure records), and that the
most common is a check that never ran at all (25%). Both rules had been reinvented
repeatedly and written down nowhere.
Two decision records:
testing.guard-derives-population-from-source (#774) — a guard enumerates its population
from a machine-readable authoritative source and asserts set equality both ways. States
the boundary that keeps it honest: filtering to select the SUBJECT of a per-member
property is fine; filtering the population before a COMPLETENESS claim is the defect.
Also separates guard SCOPE (a reviewable policy choice) from guard POPULATION (always
derived).
testing.guard-ships-with-mutation-proof (#775) — disarm that clause alone and a named
test must go red. Behaviour-only coverage is graded separately, because it proves the
guard reacts, never that it is connected.
Audit findings fixed:
ci-image-pin stated an invariant it did not check. Its error text says "Every container:
job must pin ersatztv-ci:<7-char-sha>"; what it asserts is that `grep … | sort -u` yields
one DISTINCT value. Distinctness is a property of the pins present, so deleting the
container: block from `test` leaves four pins, one distinct value, and a REQUIRED context
silently running on the bare runner. test_ci_image_pin_population.py adds the population
check, keyed on a reviewed registry cross-checked both ways — set equality between two
DERIVED sets could not see this, because both sides shrink together.
The verdict vocabulary was written down twice with no cross-check — post-review-verdict.sh
(write) and check-review-verdict.sh (read). A word in one and not the other sends the
required status green while the hook still denies. Both vocabularies are now extracted
from their own source and compared as sets; a test that restated the words would just be
a third copy. The write side's comment pointing at pretooluse-merge-consent.sh was also
stale — the hook carries no copy and delegates.
Mechanical enforcement, answered explicitly for both:
No to a filter-shaped-guard lint. The token is not the defect — ToolCatalogTests filters
correctly eight lines from a completeness assertion that must not — and it would be a
string predicate over source, which this repo's record says takes 3+ rounds. Building it
would be #774 violating #774.
Yes to enforcing the bookkeeping. docs/guard-inventory.md classifies all 32 guard files;
test_guard_inventory.py derives the population from the filesystem and call sites,
asserts set equality both ways, and resolves every claimed proof ref to a real def. A new
guard cannot ship unclassified; a renamed test cannot leave a row claiming lost coverage.
What it does NOT check — whether a MUTATION claim is true — is stated, not implied.
Measured: 28 guards, 4 tooling. 6 mutation-proved, 3 behaviour-only, 19 unproven.
Every guard added here was mutation-proved by execution before being believed: neutering
pin_population_faults turned 20 of 25 red; the inventory guard was driven red three ways
(deleted row, new unclassified hook, stale proof ref) and restored green.
573 script-tests pass. Scope limit stated in the doc: inline workflow-job guards are not
in the machine-checked population.
Refs #774
Refs #775
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
timothyandClaude Opus 5
0bd59b0b6e