timothy/ersatztv
1
0
Fork 0
Code Issues 51 Pull Requests 5 Actions Packages Projects Releases 9 Wiki Activity
Files
3473a6c889f116facc0003d7de258f5526d2cd4a
ersatztv/docs/guard-inventory.md
T
timothyandClaude Opus 5 3473a6c889 fix(774,775): close the cold-review findings — including three the change inflicted on itself 
Two independent cold reviews (Codex GPT-5.6 cross-family; Fable 5 on the patch) both
returned BLOCKED. They agreed on the counts error and the extractor hole; each found
things the other did not. Fixes, with what each was:

THE INVENTORY DID NOT COVER ITS OWN NEW GUARDS. `_SCRIPT_REF` matched `scripts/name.py`
but not `scripts/tests/*.py`, so the three guard files this change introduced had no rows
and the completeness check stayed green. A completeness guard blind to its author's new
guards is precisely the defect being legislated against. The population now globs
`scripts/tests/test_*.py` — which is how they actually run, since pr-checks.yml invokes
the directory. 32 rows -> 48.

That forced a third Kind. Once test files are in the population, every mutation proof
becomes a row wanting a proof of its own, forever. `PROOF` marks a file whose job is to
prove another guard; a scripts/tests file enforcing a repo invariant with no separate
guard behind it stays GUARD and may cite a mutation case in its own file.

HOOK EXISTENCE WAS STANDING IN FOR HOOK WIRING. Deleting a hook's registration from
.claude/settings.json left the population and the table unchanged, so the row went on
describing a guard that no longer ran — #631's shape one level down. Now derived from
settings.json plus the husky hooks.

THE SUMMARY COUNTS WERE A HAND-KEPT MIRROR AND WERE WRONG ON ARRIVAL: "28 guards, 4
tooling ... 19 have none" against a table holding 27/5/6/3/18. Both reviewers found it
independently. The prose is now parsed and asserted against the table.

TWO FALSE MUTATION GRADES, each with a concrete disarm:
  - test_full_first_page_alone_does_not_end_enumeration sends 50 docs paths then one more
    docs path; disarm pagination to treat a full page as final and it is still all-docs,
    still exempt, still green. Re-pointed at test_protected_path_on_a_LATER_page_is_still_seen,
    which does go red under that mutation.
  - test_the_scan_job_runs_the_out_of_pytest_positive_control asserts only that the script
    exists, is executable, is referenced and is marked; replace its logic with `exit 0` and
    all four pass. ci-prove-ban-detects.sh regraded NONE.

The MUTATION column was also being applied as a curve: three rows graded MUTATION fed the
real script an input only that clause rejects, which is what the rows eight lines away are
graded BEHAVIOUR-ONLY for. Definition sharpened to *witnessed* rather than plausible, and
those regraded. 5 MUTATION / 6 BEHAVIOUR-ONLY / 21 NONE across 32 guards.

THE VOCABULARY EXTRACTOR COULD RETURN A PARTIAL SET. `[A-Z|-]` cannot match `SHIP*)`, so
adding that arm leaves the extracted set non-empty AND equal to the read side — parity
green while the gate desyncs. Emptiness checks cannot see partial degradation. A loose
counterpart now asserts the strict pattern consumed every arm; proven red on exactly that
attack and green on a clean tree. Also: each verdict pattern must be assigned once, since
the extractor unions assignments while the classifier runs the last.

Also: docker-build.yml was itself an unchecked scope mirror (now asserted to be the only
workflow with toolchain container jobs, by parsing container.image rather than grepping —
ci-image.yml names the image because it builds it); the mutant floor is an equality;
e2e-functional.sh reclassified GUARD (it exits 1 on a failed contract assertion);
design-sync-reminder.sh does block the first Stop. The doc now states all six excluded
classes instead of one.

Not done here, filed instead: workflow-owned execution-class metadata to replace
TOOLCHAIN_JOBS, a single shared verdict vocabulary, and an executable clause-level
mutation harness. Each touches a CI-gating or merge-gate path and wants its own review.

580 script-tests pass.

Refs #774
Refs #775

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 22:20:51 +02:00

12 KiB
Raw Blame History

Guard inventory (ersatztv#774 / #775)

Every executable guard file in this repo, what it blocks, and whether it ships a proof it can go red. scripts/tests/test_guard_inventory.py derives the population from the filesystem and the workflow/hook call sites and asserts set equality against the Guard column, so a new guard cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.

Read docs/decisions/records/testing/guard-derives-population-from-source.md and …/guard-ships-with-mutation-proof.md before editing a guard or adding a row.

Columns

  • Kind — GUARD (it can block a commit, a push, a tool call, a merge or a CI job), TOOLING (it does work but asserts nothing; listed so its absence from the guard set is a recorded decision, not an oversight), or PROOF (a scripts/tests/ file whose job is to prove another guard). PROOF exists to stop a regress: once test files entered the population, every mutation proof became a row wanting a proof of its own. A scripts/tests/ file that enforces a repo invariant with no separate guard behind it is a GUARD, graded normally, and may cite a mutation case in its own file.
  • Proof — MUTATION: a clause-level mutation was executed and this named test was witnessed red. The test either performs the disarm itself (a monkeypatch, a deselection, a removed marker) or a recorded incident shows it going red when the clause went away. BEHAVIOUR-ONLY: a real test drives the guard through its real entry point with good and bad input, but no one has demonstrated that removing the clause reddens it. NONE.
  • Proof ref — file.py::function, verified to exist by the inventory test.

The distinction between MUTATION and BEHAVIOUR-ONLY is the whole point of #775 and is not a grading curve. A behavioural test proves the guard reacts to its input; only a mutation proves the guard is load-bearing. #685 shipped two guards on one condition where deleting either left the suite green, and every behavioural test passed throughout.

The definition above is the second one. The first read "a named test disarms this guard's clause and asserts red", and three rows were graded MUTATION against it that do not disarm anything — they feed the real script an input only that clause rejects (empty stdin, a short page, a full first page). Cold review caught it, and the objection is right twice over: those tests are the same species as ones graded BEHAVIOUR-ONLY eight rows away, so the column was being applied as a curve on the very day it was introduced. They are regraded. The surviving criterion is witnessed, not plausible — "removing this clause would surely redden that test" is an argument, and this table exists because arguments of that shape have been wrong here six times.

Scope limit, stated rather than implied

This inventory covers guard files, discovered by globbing .claude/hooks/*.sh, .husky/* and scripts/tests/test_*.py, plus every scripts/… path referenced by a workflow or a hook. Six classes are outside that population. They are listed because the first version of this section named only the first one, and cold review found that the very guards this inventory shipped with were sitting in the gap:

  1. Guards inline in workflow YAML — most importantly pr-checks.yml:ci-image-pin. "Which jobs are guards" needs a judgement call per job the filesystem cannot supply. Two were audited under #774 and one fixed; extending the population is tracked in #786.
  2. C# and TypeScript guards — ErsatzTV.Mcp.Tests/ToolCatalogTests.cs and web/src/api/pageSizeCallSites.guard.test.ts are both structural guards and neither has a row.
  3. Mentions counted as call sites. The scripts/… scrape matches any occurrence, including inside a comment or an ::error:: string. scripts/update-openapi.sh is named in a pr-checks.yml error message, so removing the step that runs it would leave its row intact.
  4. Nested and non-lowercase paths beyond scripts/tests/ — a guard under scripts/scripted-schedules/, or with an uppercase name, is invisible to the scrape.
  5. .yaml workflows and non-.sh hooks — the globs are *.yml and *.sh only.
  6. Transitive calls — a script invoked only by another script, rather than by a workflow or hook, is not discovered.

Hook wiring is checked (test_every_hook_file_is_actually_WIRED reads .claude/settings.json and the husky hooks), so a hook file whose registration is deleted fails rather than keeping a row that reads as coverage. That check does not extend to the scripts/ half.

Inventory

Guard Blocks Kind Proof Proof ref
.claude/hooks/decisions-guard.sh a commit GUARD NONE —
.claude/hooks/design-sync-reminder.sh the first Stop after a UI change (one-shot, then allows) GUARD NONE —
.claude/hooks/posttooluse-worktree-marker.sh nothing (writes the marker the worktree guard reads) GUARD NONE —
.claude/hooks/prepush-clean-worktree-check.sh a push with uncommitted changes in the pushed set GUARD NONE —
.claude/hooks/prepush-donewhen.sh a direct push to main with unticked Done-when boxes GUARD NONE —
.claude/hooks/prepush-rebase-check.sh a push from a branch behind origin/main GUARD BEHAVIOUR-ONLY test_prepush_rebase_check_tag_exemption.py::test_zero_ref_lines_does_not_exempt
.claude/hooks/pretooluse-agent-model.sh an Agent dispatch naming no model (asks) GUARD NONE —
.claude/hooks/pretooluse-agent-ram.sh an Agent dispatch under 10% free RAM GUARD NONE —
.claude/hooks/pretooluse-bash-guard.sh a Bash call setting ETV_UPDATE_GOLDENS GUARD NONE —
.claude/hooks/pretooluse-bom-guard.sh a commit/push carrying a BOM in a touched .cs GUARD NONE —
.claude/hooks/pretooluse-merge-consent.sh a PR merge without derived consent GUARD BEHAVIOUR-ONLY test_merge_consent_exemption.py::test_protected_path_on_a_LATER_page_is_still_seen
.claude/hooks/pretooluse-nav-guard.sh a browser navigate to a streaming URL GUARD NONE —
.claude/hooks/pretooluse-worktree-guard.sh a commit/merge in a foreign worktree GUARD NONE —
.husky/commit-msg a commit with no Co-Authored-By trailer GUARD NONE —
.husky/pre-commit a commit failing lint-staged, decisions, root-PNG or format GUARD NONE —
.husky/pre-push a push failing any pre-push hook or the SPA gate GUARD NONE —
scripts/build_decisions_catalog.py the decisions-guard job, on a stale catalog GUARD NONE —
scripts/check-kickoff-guard.sh the decisions-guard job, on a revived #237 reference GUARD NONE —
scripts/check-review-verdict.sh the merge-consent hook's verdict classification GUARD BEHAVIOUR-ONLY test_check_review_verdict.py::test_falseopen_token_must_be_a_whole_word
scripts/ci-detect-already-validated.sh nothing directly (feeds the skip gate) GUARD NONE —
scripts/ci-detect-docs-only.sh nothing directly (feeds the skip gate) GUARD NONE —
scripts/ci-peak-anon.sh nothing (samples container memory) TOOLING NONE —
scripts/ci-prove-ban-detects.sh the release path, if the delimiter ban is disarmed GUARD NONE —
scripts/ci-step-ran.sh the two required contexts, on a dropped step GUARD MUTATION test_ci_dropped_step_guard.py::test_dropping_ANY_single_step_FAILS_the_guard
scripts/decisions_validate.py the decisions-guard job, on a lifecycle fault GUARD MUTATION test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan
scripts/e2e-functional.sh the Functional E2E job, on a failed HTTP contract assertion GUARD NONE —
scripts/e2e-local.sh nothing (boots a local instance) TOOLING NONE —
scripts/e2e-ui.sh nothing (drives the Playwright flows) TOOLING NONE —
scripts/jq-preflight.sh the script-tests job, on a jq version change GUARD BEHAVIOUR-ONLY test_jq_preflight.py::test_below_the_floor_is_LOUD
scripts/post-review-verdict.sh nothing (writes the verdict status) GUARD BEHAVIOUR-ONLY test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head
scripts/pr-changed-files.sh the verdict exemption, on an incomplete enumeration GUARD BEHAVIOUR-ONLY test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration
scripts/update-openapi.sh nothing (regenerates the spec) TOOLING NONE —
scripts/tests/test_build_catalog.py the script-tests job PROOF NONE —
scripts/tests/test_check_review_verdict.py the script-tests job PROOF NONE —
scripts/tests/test_ci_dropped_step_guard.py the script-tests job PROOF NONE —
scripts/tests/test_ci_image_pin_population.py the script-tests job, when a container job loses its pin GUARD MUTATION test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED
scripts/tests/test_ci_release_path_scan_job.py the script-tests job, on a weakened release-path scan job GUARD NONE —
scripts/tests/test_decisions_lib.py the script-tests job PROOF NONE —
scripts/tests/test_decisions_validate.py the script-tests job PROOF NONE —
scripts/tests/test_guard_inventory.py the script-tests job, on an unclassified guard or a stale proof ref GUARD MUTATION test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist
scripts/tests/test_jq_preflight.py the script-tests job PROOF NONE —
scripts/tests/test_merge_consent_base_change.py the script-tests job PROOF NONE —
scripts/tests/test_merge_consent_exemption.py the script-tests job PROOF NONE —
scripts/tests/test_migration_equivalence.py the script-tests job PROOF NONE —
scripts/tests/test_post_review_verdict.py the script-tests job PROOF NONE —
scripts/tests/test_pr_changed_files.py the script-tests job PROOF NONE —
scripts/tests/test_prepush_rebase_check_tag_exemption.py the script-tests job PROOF NONE —
scripts/tests/test_review_verdict_vocabulary_parity.py the script-tests job, when the verdict vocabularies desync GUARD MUTATION test_review_verdict_vocabulary_parity.py::test_the_strict_extractor_consumed_EVERY_case_arm

What the numbers say

32 guards, 4 tooling scripts, 12 proof files. 5 guards carry a mutation proof; 6 are behaviour-only; 21 have none. These figures are asserted against the table by test_the_summary_counts_match_the_table — they were wrong in the first draft (28/4/6/3/19 against a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it, which is the duplication family this change argues against. Both cold reviewers found the error independently.

The 21 include every PreToolUse hook except merge-consent — the same population #776 found we cannot even observe firing. A guard that is neither proven nor observable is an assertion about the past.

The gaps are not uniform in cost, and the ranking that matters is what a silent failure would let through, not test count:

  1. pretooluse-bom-guard.sh — guards a defect that has recurred three times (#311, #402, #405).
  2. pretooluse-worktree-guard.sh + posttooluse-worktree-marker.sh — a two-file mechanism guarding #289, where a regression in either half is invisible and the two halves have never been tested together.
  3. .husky/pre-push:11's unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE — a one-line fix for a real bug (a nested git diff --exit-code silently reporting no diff) that nothing pins. Reordering it after the nested git calls reintroduces the bug silently.
  4. scripts/build_decisions_catalog.py — its --check path is what CI runs and no test calls main() at all; the tests exercise render_catalog() directly.

Filling these is tracked rather than done in one pass, deliberately: a mutation proof written to close a row is the kind of test that passes for the wrong reason.

Reference in New Issue View Git Blame Copy Permalink