Files

T

timothyandClaude Opus 5 96a67c7e76 fix(774): the loose extractor shared its subject's blind spot; stop claiming the limit list is exhaustive

Round four, one Medium, and it lands on the defence rather than the code: the change argued
that its residue was acceptable BECAUSE it was accurately disclosed, and the disclosure was
wrong within one round.

`SHIP-IT) state=success ;;` is valid shell selecting `success`. Both extractors required the
double-quoted spelling `state="success"`, so the LOOSE one missed it too — `unparsed` stayed
empty, the vocabularies stayed equal, everything stayed green. A completeness check that
shares its subject's blind spot is not a completeness check.

The bug was structural, not about quoting. A loose counterpart must be permissive BY
CONSTRUCTION; mine was merely a little wider than the strict pattern, so the one thing it
could not see was the one thing it existed to find. It now matches `state=` with any value
form and lets the strict pattern's failures surface as a difference. Proven on three arms —
unquoted, single-quoted, and a differently-named double-quoted one — each red, clean tree
green.

The disclosure is corrected too, and this is the part worth keeping. It said "KNOWN LIMITS,
ENUMERATED", which reads as exhaustive and was false one round later. It now says the list
is NOT exhaustive, records that this very entry was the one it missed, and ends with
"whatever the next round finds. Assume this list is one short." Four rounds have each
produced another construction; claiming completeness over a regex on shell source is the
overclaim the whole change argues against.

Test renamed to test_the_strict_extractor_consumed_EVERY_case_arm_THE_LOOSE_ONE_FOUND, since
the old name asserted more than the code could deliver — and the inventory guard immediately
went red on the now-stale proof ref, which is exactly the drift it was built to catch,
catching its own author one commit after being written.

ruff clean, pyright clean, decisions-validate OK, 580 script-tests pass.

Refs #774

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-13 22:48:53 +02:00

13 KiB

Raw Blame History

Guard inventory (ersatztv#774 / #775)

Every executable guard file in this repo, what it blocks, and whether it ships a proof it can go red. scripts/tests/test_guard_inventory.py derives the population from the filesystem and the workflow/hook call sites and asserts set equality against the Guard column, so a new guard cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.

Read docs/decisions/records/testing/guard-derives-population-from-source.md and …/guard-ships-with-mutation-proof.md before editing a guard or adding a row.

Columns

Kind — GUARD (it can block a commit, a push, a tool call, a merge or a CI job), TOOLING (it does work but asserts nothing; listed so its absence from the guard set is a recorded decision, not an oversight), or PROOF (a scripts/tests/ file whose job is to prove another guard). PROOF exists to stop a regress: once test files entered the population, every mutation proof became a row wanting a proof of its own. A scripts/tests/ file that enforces a repo invariant with no separate guard behind it is a GUARD, graded normally, and may cite a mutation case in its own file.
Proof — MUTATION: a clause-level mutation was executed and this named test was witnessed red. The test either performs the disarm itself (a monkeypatch, a deselection, a removed marker) or a recorded incident shows it going red when the clause went away. BEHAVIOUR-ONLY: a real test drives the guard through its real entry point with good and bad input, but no one has demonstrated that removing the clause reddens it. NONE.
Proof ref — file.py::function, verified to exist by the inventory test.

The distinction between MUTATION and BEHAVIOUR-ONLY is the whole point of #775 and is not a grading curve. A behavioural test proves the guard reacts to its input; only a mutation proves the guard is load-bearing. #685 shipped two guards on one condition where deleting either left the suite green, and every behavioural test passed throughout.

The definition above is the second one. The first read "a named test disarms this guard's clause and asserts red", and three rows were graded MUTATION against it that do not disarm anything — they feed the real script an input only that clause rejects (empty stdin, a short page, a full first page). Cold review caught it, and the objection is right twice over: those tests are the same species as ones graded BEHAVIOUR-ONLY eight rows away, so the column was being applied as a curve on the very day it was introduced. They are regraded. The surviving criterion is witnessed, not plausible — "removing this clause would surely redden that test" is an argument, and this table exists because arguments of that shape have been wrong here six times.

Scope limit, stated rather than implied

This inventory covers guard files, discovered by globbing .claude/hooks/*.sh, .husky/* and scripts/tests/test_*.py, plus every scripts/… path referenced by a workflow or a hook. Six classes are outside that population. They are listed because the first version of this section named only the first one, and cold review found that the very guards this inventory shipped with were sitting in the gap:

Guards inline in workflow YAML — most importantly pr-checks.yml:ci-image-pin. "Which jobs are guards" needs a judgement call per job the filesystem cannot supply. Two were audited under #774 and one fixed; extending the population is tracked in #786.
C# and TypeScript guards — ErsatzTV.Mcp.Tests/ToolCatalogTests.cs and web/src/api/pageSizeCallSites.guard.test.ts are both structural guards and neither has a row.
Mentions counted as call sites. The scripts/… scrape matches any occurrence, including inside a comment or an ::error:: string. scripts/update-openapi.sh is named in a pr-checks.yml error message, so removing the step that runs it would leave its row intact.
Nested and non-lowercase paths beyond scripts/tests/ — a guard under scripts/scripted-schedules/, or with an uppercase name, is invisible to the scrape.
.yaml workflows and non-.sh hooks — the globs are *.yml and *.sh only.
Transitive calls — a script invoked only by another script, rather than by a workflow or hook, is not discovered.

Hook wiring is checked (test_every_hook_file_is_actually_WIRED reads .claude/settings.json and the husky hooks with full-line comments stripped), so a hook file whose registration is deleted fails rather than keeping a row that reads as coverage. Its limit, stated because the check reads stronger than it is: it is a substring test for the basename, not a parse of the invocation. : # .claude/hooks/decisions-guard.sh disabled still reads as wired, and conversely a hook invoked through a wrapper or a constructed path reads as unwired. It catches deletion, which is the common case; it does not catch deliberate disablement. The check does not extend to the scripts/ half at all.

Inventory

Guard	Blocks	Kind	Proof	Proof ref
`.claude/hooks/decisions-guard.sh`	a commit	GUARD	NONE	—
`.claude/hooks/design-sync-reminder.sh`	the first Stop after a UI change (one-shot, then allows)	GUARD	NONE	—
`.claude/hooks/posttooluse-worktree-marker.sh`	nothing (writes the marker the worktree guard reads)	GUARD	NONE	—
`.claude/hooks/prepush-clean-worktree-check.sh`	a push with uncommitted changes in the pushed set	GUARD	NONE	—
`.claude/hooks/prepush-donewhen.sh`	a direct push to `main` with unticked Done-when boxes	GUARD	NONE	—
`.claude/hooks/prepush-rebase-check.sh`	a push from a branch behind `origin/main`	GUARD	BEHAVIOUR-ONLY	`test_prepush_rebase_check_tag_exemption.py::test_zero_ref_lines_does_not_exempt`
`.claude/hooks/pretooluse-agent-model.sh`	an Agent dispatch naming no model (asks)	GUARD	NONE	—
`.claude/hooks/pretooluse-agent-ram.sh`	an Agent dispatch under 10% free RAM	GUARD	NONE	—
`.claude/hooks/pretooluse-bash-guard.sh`	a Bash call setting `ETV_UPDATE_GOLDENS`	GUARD	NONE	—
`.claude/hooks/pretooluse-bom-guard.sh`	a commit/push carrying a BOM in a touched `.cs`	GUARD	NONE	—
`.claude/hooks/pretooluse-merge-consent.sh`	a PR merge without derived consent	GUARD	BEHAVIOUR-ONLY	`test_merge_consent_exemption.py::test_protected_path_on_a_LATER_page_is_still_seen`
`.claude/hooks/pretooluse-nav-guard.sh`	a browser navigate to a streaming URL	GUARD	NONE	—
`.claude/hooks/pretooluse-worktree-guard.sh`	a commit/merge in a foreign worktree	GUARD	NONE	—
`.husky/commit-msg`	a commit with no `Co-Authored-By` trailer	GUARD	NONE	—
`.husky/pre-commit`	a commit failing lint-staged, decisions, root-PNG or format	GUARD	NONE	—
`.husky/pre-push`	a push failing any pre-push hook or the SPA gate	GUARD	NONE	—
`scripts/build_decisions_catalog.py`	the `decisions-guard` job, on a stale catalog	GUARD	NONE	—
`scripts/check-kickoff-guard.sh`	the `decisions-guard` job, on a revived #237 reference	GUARD	NONE	—
`scripts/check-review-verdict.sh`	the merge-consent hook's verdict classification	GUARD	BEHAVIOUR-ONLY	`test_check_review_verdict.py::test_falseopen_token_must_be_a_whole_word`
`scripts/ci-detect-already-validated.sh`	nothing directly (feeds the skip gate)	GUARD	NONE	—
`scripts/ci-detect-docs-only.sh`	nothing directly (feeds the skip gate)	GUARD	NONE	—
`scripts/ci-peak-anon.sh`	nothing (samples container memory)	TOOLING	NONE	—
`scripts/ci-prove-ban-detects.sh`	the release path, if the delimiter ban is disarmed	GUARD	NONE	—
`scripts/ci-step-ran.sh`	the two required contexts, on a dropped step	GUARD	MUTATION	`test_ci_dropped_step_guard.py::test_dropping_ANY_single_step_FAILS_the_guard`
`scripts/decisions_validate.py`	the `decisions-guard` job, on a lifecycle fault	GUARD	MUTATION	`test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan`
`scripts/e2e-functional.sh`	the Functional E2E job, on a failed HTTP contract assertion	GUARD	NONE	—
`scripts/e2e-local.sh`	nothing (boots a local instance)	TOOLING	NONE	—
`scripts/e2e-ui.sh`	nothing (drives the Playwright flows)	TOOLING	NONE	—
`scripts/jq-preflight.sh`	the `script-tests` job, on a jq version change	GUARD	BEHAVIOUR-ONLY	`test_jq_preflight.py::test_below_the_floor_is_LOUD`
`scripts/post-review-verdict.sh`	nothing (writes the verdict status)	GUARD	BEHAVIOUR-ONLY	`test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head`
`scripts/pr-changed-files.sh`	the verdict exemption, on an incomplete enumeration	GUARD	BEHAVIOUR-ONLY	`test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration`
`scripts/update-openapi.sh`	nothing (regenerates the spec)	TOOLING	NONE	—
`scripts/tests/test_build_catalog.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_check_review_verdict.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_ci_dropped_step_guard.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_ci_image_pin_population.py`	the `script-tests` job, when a container job loses its pin	GUARD	MUTATION	`test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED`
`scripts/tests/test_ci_release_path_scan_job.py`	the `script-tests` job, on a weakened release-path scan job	GUARD	NONE	—
`scripts/tests/test_decisions_lib.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_decisions_validate.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_guard_inventory.py`	the `script-tests` job, on an unclassified guard or a stale proof ref	GUARD	MUTATION	`test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist`
`scripts/tests/test_jq_preflight.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_merge_consent_base_change.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_merge_consent_exemption.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_migration_equivalence.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_post_review_verdict.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_pr_changed_files.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_prepush_rebase_check_tag_exemption.py`	the `script-tests` job	PROOF	NONE	—
`scripts/tests/test_review_verdict_vocabulary_parity.py`	the `script-tests` job, when the verdict vocabularies desync	GUARD	MUTATION	`test_review_verdict_vocabulary_parity.py::test_the_strict_extractor_consumed_EVERY_case_arm_THE_LOOSE_ONE_FOUND`

What the numbers say

32 guards, 4 tooling scripts, 12 proof files. 5 guards carry a mutation proof; 6 are behaviour-only; 21 have none. These figures are asserted against the table by test_the_summary_counts_match_the_table — they were wrong in the first draft (28/4/6/3/19 against a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it, which is the duplication family this change argues against. Both cold reviewers found the error independently.

The 21 include every PreToolUse hook except merge-consent — the same population #776 found we cannot even observe firing. A guard that is neither proven nor observable is an assertion about the past.

The gaps are not uniform in cost, and the ranking that matters is what a silent failure would let through, not test count:

pretooluse-bom-guard.sh — guards a defect that has recurred three times (#311, #402, #405).
pretooluse-worktree-guard.sh + posttooluse-worktree-marker.sh — a two-file mechanism guarding #289, where a regression in either half is invisible and the two halves have never been tested together.
.husky/pre-push:11's unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE — a one-line fix for a real bug (a nested git diff --exit-code silently reporting no diff) that nothing pins. Reordering it after the nested git calls reintroduces the bug silently.
scripts/build_decisions_catalog.py — its --check path is what CI runs and no test calls main() at all; the tests exercise render_catalog() directly.

Filling these is tracked rather than done in one pass, deliberately: a mutation proof written to close a row is the kind of test that passes for the wrong reason.

13 KiB Raw Blame History