Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 8s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 14s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 22s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 21s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 17s
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 7m24s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m34s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m7s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 5m43s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
review-verdict/h10 Review-verdict: MERGEABLE @ d3f470f (base: main)
Rebase-only conflict, caught by re-running the gate on the NEW base rather than trusting the green from the old one. #871 (`docs(796)`) added `scripts/tests/test_mcp_smoke.py` as a MUTATION-graded guard row while this branch added `scripts/tests/test_workflow_job_guards.py` as another. Each bumped the hand-maintained summary sentence by one and each was green against its own base; merged, the table holds 44/22 and the sentence claimed 43/21. This is `ci.whole-tree-gate-is-only-as-green-as-its-base` — the shape #780 and #784 hit. The summary is asserted by `test_the_summary_counts_match_the_table`, so it failed loudly rather than drifting, which is the whole reason that assertion exists. Refs #786
541 lines
54 KiB
Markdown
541 lines
54 KiB
Markdown
# Guard inventory (ersatztv#774 / #775)
|
|
|
|
Every executable guard **file** in this repo, what it blocks, and whether it ships a proof it can go
|
|
red. `scripts/tests/test_guard_inventory.py` derives the population from the **git index** and the
|
|
workflow/hook call sites and asserts **set equality** against the `Guard` column, so a new guard
|
|
cannot be added without acquiring a row here, and a row cannot name a proof that does not exist.
|
|
The index rather than a filesystem walk since ersatztv#806 — the practical consequence is that a new
|
|
guard joins the population when it is **staged**, not when the file appears. Nothing local runs
|
|
these checks at all: `.husky/pre-commit` runs lint-staged, the decisions guard, the root-PNG check
|
|
and `dotnet format`, and `grep -rn pytest .husky/` returns nothing. The runner is
|
|
`pr-checks.yml::script-tests`, `on: pull_request`, so the red arrives in CI — plus
|
|
`docker-build.yml:723` on the release path, which re-runs two of these files
|
|
(`test_ci_dropped_step_guard.py` and `test_ci_release_path_scan_job.py`) as a `needs:` of `build`.
|
|
|
|
**Read `docs/decisions/records/testing/guard-derives-population-from-source.md` and
|
|
`…/guard-ships-with-mutation-proof.md` before editing a guard or adding a row.**
|
|
|
|
## Columns
|
|
|
|
- **Kind** — `GUARD` (it can block a commit, a push, a tool call, a merge or a CI job), `TOOLING`
|
|
(it does work but asserts nothing; listed so its absence from the guard set is a recorded
|
|
decision, not an oversight), or `PROOF` (a `scripts/tests/` file whose job is to prove another
|
|
guard). `PROOF` exists to stop a regress: once test files entered the population, every mutation
|
|
proof became a row wanting a proof of its own. A `scripts/tests/` file that enforces a repo
|
|
invariant with no separate guard behind it is a `GUARD`, graded normally, and may cite a mutation
|
|
case in its own file.
|
|
- **Proof** — `MUTATION`: a clause-level mutation **was executed** and this named test **was
|
|
witnessed red**. The test either performs the disarm itself (a monkeypatch, a deselection, a
|
|
removed marker) or a recorded incident shows it going red when the clause went away.
|
|
`BEHAVIOUR-ONLY`: a real test drives the guard through its real entry point with good and bad
|
|
input, but no one has demonstrated that removing the clause reddens it. `NONE`.
|
|
- **Proof ref** — `file.py::function`, verified to exist by the inventory test.
|
|
|
|
The distinction between `MUTATION` and `BEHAVIOUR-ONLY` is the whole point of #775 and is not a
|
|
grading curve. A behavioural test proves the guard reacts to its input; only a mutation proves the
|
|
guard is *load-bearing*. #685 shipped two guards on one condition where deleting either left the
|
|
suite green, and every behavioural test passed throughout.
|
|
|
|
**The definition above is the second one.** The first read "a named test disarms this guard's clause
|
|
and asserts red", and three rows were graded `MUTATION` against it that do not disarm anything —
|
|
they feed the real script an input only that clause rejects (empty stdin, a short page, a full first
|
|
page). Cold review caught it, and the objection is right twice over: those tests are the same
|
|
species as ones graded `BEHAVIOUR-ONLY` eight rows away, so the column was being applied as a curve
|
|
on the very day it was introduced. They are regraded. The surviving criterion is *witnessed*, not
|
|
*plausible* — "removing this clause would surely redden that test" is an argument, and this table
|
|
exists because arguments of that shape have been wrong here six times.
|
|
|
|
## Scope limit, stated rather than implied
|
|
|
|
The `## Inventory` table covers guard **files**, discovered by reading `.claude/hooks/*.sh`,
|
|
`.husky/*` and `scripts/tests/test_*.py` **out of the git index**, plus every `scripts/…` path
|
|
referenced by a workflow or a hook. Guard **jobs** are covered separately by
|
|
`## Workflow-job guards` below. The classes below are outside BOTH populations, and are enumerated
|
|
rather than summarised: an unlisted class reads as covered, and guards this inventory itself shipped
|
|
with have sat in these gaps. Treat anything not named here as unassessed, not as included:
|
|
|
|
1. ~~**Guards inline in workflow YAML**~~ — **CLOSED by ersatztv#786**, and kept here rather than
|
|
deleted because the entry named the shape and the shape is worth keeping visible. `pr-checks.yml`'s
|
|
`ci-image-pin` — which stated an invariant it did not check (#774) — sat in this gap. Workflow jobs
|
|
now declare `env.CI_JOB_ROLE`, and `scripts/tests/test_workflow_job_guards.py` asserts set equality
|
|
between the guard-declaring jobs and `## Workflow-job guards` in both directions. The judgement
|
|
#774 said the filesystem cannot supply is still a human one; what changed is that it is now
|
|
RECORDED, next to the job, and a job that acquires no marker fails the suite. **Two residuals**:
|
|
the check proves each job was classified, never that the classification is CORRECT; and an
|
|
`inline` assertion cannot be proven to work without running the job, so those rows carry
|
|
`Proof: NONE` honestly instead of borrowing a neighbouring file's proof.
|
|
2. **C# and TypeScript guards** — `ErsatzTV.Mcp.Tests/ToolCatalogTests.cs`,
|
|
`web/src/api/pageSizeCallSites.guard.test.ts` and `web/src/api/completeRequest.guard.test.ts`
|
|
are all structural guards and none has a row. Note what that costs: this list is a HAND-WRITTEN
|
|
mirror of a population nothing derives, so it goes stale silently and CI stays green — #807
|
|
added the third entry, and only review caught that the second had become the only one named.
|
|
3. **Mentions counted as call sites.** The `scripts/…` scrape matches any occurrence, including
|
|
inside a comment or an `::error::` string. `scripts/update-openapi.sh` is named in a
|
|
`pr-checks.yml` error message, so removing the step that runs it would leave its row intact.
|
|
4. **Nested and non-lowercase paths** beyond `scripts/tests/` — a guard under
|
|
`scripts/scripted-schedules/`, or with an uppercase name, is invisible to the scrape.
|
|
5. **Non-`.sh` hooks** — the hook pattern is `*.sh` only. (`.yaml` workflows are no longer in this
|
|
gap: the caller scan matches `*.yml` and `*.yaml` since #806, because Gitea accepts both.)
|
|
6. **Transitive calls** — a script invoked only by another script, rather than by a workflow or
|
|
hook, is not discovered. Being outside this population is NOT an exemption from proof:
|
|
`testing.verification-code-needs-its-own-proof` (#796) puts such a script in the ordinary branch
|
|
whenever a proof test can drive it hermetically. A row here is not the route — the population is
|
|
derived from the call sites, so one is rejected as a phantom — but the manifest can still name
|
|
the checker, as `target`, with its proof test as `guard`. `scripts/mcp_smoke.py` is the worked
|
|
example: no row of its own, proven by `scripts/tests/test_mcp_smoke.py`, which carries the row.
|
|
That row is `GUARD` citing itself while its declared `target` is a different file — an ESTABLISHED
|
|
shape here, not a new one, and deliberately not counted: `test_review_verdict_vocabulary.py`
|
|
targets `scripts/check-review-verdict.sh` and `test_mutation_harness.py` targets
|
|
`mutation_harness_lib.py`. Worth noting only because the self-citation carve-out in
|
|
`test_every_proof_ref_points_at_a_row_marked_PROOF` is worded for a checker guarding a repo
|
|
invariant, whereas these guard another file.
|
|
7. **Non-`test_` modules under `scripts/tests/`** — the pattern is `test_*.py`, so `conftest.py`,
|
|
`mutation_harness_lib.py`, `mutation_manifest.py` and `tracked_files.py` are outside the
|
|
population and hold no rows. They are not guards (they assert nothing on their own), but the
|
|
middle two ARE what `test_mutation_harness.py` is made of, so gutting either would take that
|
|
guard with it. What catches that is the guard's own row: its declared mutation targets
|
|
`mutation_harness_lib.py`, and its proof test refuses to run if the clause it names has moved.
|
|
`tracked_files.py` has no such backstop — it is load-bearing for every module that imports it
|
|
(#806), and an edit to it is covered only by those modules' own proofs. Recorded rather than
|
|
force-fitted: a row for a library would need a `Kind` the vocabulary does not have.
|
|
8. **Nested workflow directories** — the workflow scope is direct children of `.gitea/workflows`, so
|
|
a tracked `.gitea/workflows/nested/x.yml` is invisible to the caller scan and to
|
|
`test_ci_image_pin_population.py`. Left as scope rather than widened: whether Gitea executes
|
|
nested workflow files was not verified here, and widening on an unverified premise risks a
|
|
permanent red on a correct tree, which is how a correct guard gets deleted.
|
|
|
|
Hook **wiring** is checked (`test_every_hook_file_is_actually_WIRED` reads `.claude/settings.json`
|
|
and the husky hooks with full-line comments stripped), so a hook file whose registration is deleted
|
|
fails rather than keeping a row that reads as coverage. Its limit, stated because the check reads
|
|
stronger than it is: **it is a substring test for the basename, not a parse of the invocation.**
|
|
`: # .claude/hooks/decisions-guard.sh disabled` still reads as wired, and conversely a hook invoked
|
|
through a wrapper or a constructed path reads as unwired. It catches deletion, which is the common
|
|
case; it does not catch deliberate disablement. The check does not extend to the `scripts/` half at
|
|
all.
|
|
|
|
## File populations and where they come from (ersatztv#806)
|
|
|
|
Every guard here whose members are FILES derives them from the **git index**, never a filesystem
|
|
walk. The disk is not an authoritative source: it reports build output and editor droppings and
|
|
differs per machine, so a guard derived from it asserts a different population in CI than on the
|
|
laptop of the person it is meant to stop. `scripts/tests/tracked_files.py` is the single derivation
|
|
and carries the full rationale; `scripts/tests/test_guard_populations_derive_from_git.py` proves it,
|
|
in both directions: removing EVERY member of each registered derivation from the index one at a time
|
|
and requiring it to disappear while still on disk, and watching for a directory LISTING issued while
|
|
the derivation runs (reading files stays allowed). Removal alone is blind to a source that
|
|
contributes only untracked members — an `rglob` reaching `.husky/_/` adds and never removes — and
|
|
any check phrased as "an untracked file must not enter" is itself machine-dependent, because the
|
|
untracked file has to exist. Watching for the call needs no arranged state.
|
|
|
|
That second check is a regression guard against the accidental shapes, not a boundary: what is
|
|
observed is any call that goes **through one of the spies**, whenever it happens — the spy records
|
|
into a list that outlives the patch, so a reference captured during the window and invoked after it
|
|
still counts. Whether the call goes through a spy is what decides, not when. The instance list — what
|
|
never reaches a spy at all — lives in the check's own docstring and is deliberately not restated
|
|
here, because a second copy of it drifted from the first within one commit.
|
|
|
|
One deliberate exception to "from the index" sits in the same file: the registration check lists
|
|
`scripts/tests/test_*.py` from disk on purpose, because it is a superset check over what pytest
|
|
collects — an untracked stray there makes it MORE demanding, never blind, whereas using the index
|
|
would let an unstaged new guard escape registration.
|
|
|
|
Its own limit, stated because a check described as complete stops being re-examined: it finds
|
|
derivations by PARSING each `test_*.py` for an import of the shared helper, so a module that derives
|
|
a file population some other way is invisible to it, and no mechanical check can close that (#774
|
|
reached the same conclusion about detecting filter-shaped guards by token).
|
|
|
|
The audit #806 asked for, recorded whichever way it came out, because "we looked and left it" and
|
|
"we never looked" are indistinguishable a year later:
|
|
|
|
| Guard | Population | Completeness claim over tracked files? | Outcome |
|
|
| --- | --- | --- | --- |
|
|
| `test_guard_inventory.py` | `.claude/hooks/*.sh`, `.husky/*`, `scripts/tests/test_*.py`, workflow/hook callers | **yes** — set equality against this table | converted to the index; `.husky/_/` had been excluded only because `_` is a directory, so the obvious "make it recursive" edit would have reintroduced #778's defect here |
|
|
| `test_hook_fire_log.py` | `.claude/hooks/*.sh` | **yes** — every hook must be instrumented | converted; an untracked scratch `.sh` used to demand instrumentation and redden the suite on that checkout alone |
|
|
| `test_ci_image_pin_population.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — "docker-build is the ONLY workflow pinning the toolchain image" | converted, and `*.yaml` added: Gitea accepts both spellings, so a `.yaml` workflow was structurally invisible while the test read as covering all of them |
|
|
| `test_workflow_job_guards.py` | `.gitea/workflows/*.y*ml` | **yes** — every tracked workflow's jobs must each declare a `CI_JOB_ROLE`, and the guard-declaring ones must match `## Workflow-job guards` both ways | index-derived from the start (ersatztv#786); registered in `test_guard_populations_derive_from_git.py` so the shared proof covers it |
|
|
| `test_pr_changed_files.py` | `.gitea/workflows/*.y*ml` | **yes** — "no OTHER workflow writes the review-verdict status", and since ersatztv#748 "every tracked workflow declares a `permissions:` block", with NO exemption list (the one `ci-image.yml` briefly needed was deleted when #744 landed, per #835) | converted. Not on #806's list and found by cold review on the pushed head: an untracked `.yaml` dropped in `.gitea/workflows/` reddened two guards while absent from the index — the issue's list of files to assess was a starting point, not the population |
|
|
| `test_workflow_persist_credentials.py` | `.gitea/workflows/*.yml` + `*.yaml` | **yes** — every `actions/checkout` in every workflow must drop the persisted credential | derived from the index from the start (#835). Shipped with NO exemption list, which is why it waited for #744: the one non-compliant checkout would otherwise have bought a permanent entry that keeps passing after its reason expires |
|
|
| `web/src/api/pageSizeCallSites.guard.test.ts` | `web/src/**/*.{ts,tsx,mts,cts}` via `import.meta.glob` | **yes** — an unregistered discovered site fails | DEFERRED to #819, assessed not skipped. The glob is a documented workaround: `@types/node` is deliberately out of `tsconfig.app.json`, and wiring it in was tried and reverted (it leaked Node's `setTimeout` into the app project and broke three unrelated tests), so there is no `node:child_process` to reach the index from. Over-enumerates, so it fails loudly rather than going blind |
|
|
| `test_ci_release_path_scan_job.py` | `.gitea/workflows/*.y*ml` + `scripts/**` | **no** — a fixture assembling a tmp harness, asserted about behaviour not membership | takes its file LIST from the index anyway, for hermeticity not completeness: `shutil.copytree` copied whatever was on disk, including untracked files and `scripts/__pycache__`, into a tree whose behaviour the probes then measure. Content still comes from the working tree. The copy is not a git repo, so the two files this step RUNS may not use the helper — see the fixture docstring |
|
|
| `test_ci_dropped_step_guard.py` | the parsed workflow document (its OWN subject only — the release-path `scan` job runs it in a non-git copy, so it may not derive from the index) | **no filesystem population at all** | unchanged by #806; the SCOPE residual it left open — `MARKED_JOBS` as a hand-written mirror of the required contexts on `main` — was closed by #787, which derives it from `.gitea/required-status-contexts.json` and reconciles that snapshot against the server in `scripts/check-required-contexts.sh`. The cross-workflow half of that claim lives in `test_ci_status_context_uniqueness.py`, which CAN use the index |
|
|
|
|
`test_remote_state_inventory.py` fixed its own population under #778 and kept a private copy of the
|
|
derivation; #806 folded it onto the shared one, so that module is covered by the proof above like
|
|
the rest. Across the whole change, every module that derived a file population its own way now goes
|
|
through `tracked_files.py` — one implementation of the rule instead of one per module. The
|
|
registered derivations are listed in `DERIVATIONS` in
|
|
`scripts/tests/test_guard_populations_derive_from_git.py`; this page deliberately keeps no count
|
|
of them.
|
|
|
|
**Still on filesystem walks, deliberately out of scope:** the decisions corpus
|
|
(`scripts/decisions_lib.py`'s `active_files()`, and the suites over it). Its members are `docs/`
|
|
Markdown with no generated-file pressure and a different lifecycle, and folding it in here would
|
|
have been the reflex this milestone argues against — a change with no defect behind it. It is
|
|
recorded as unexamined rather than as cleared.
|
|
|
|
## Inventory
|
|
|
|
| Guard | Blocks | Kind | Proof | Proof ref |
|
|
| --- | --- | --- | --- | --- |
|
|
| `.claude/hooks/decisions-guard.sh` | a commit | GUARD | NONE | — |
|
|
| `.claude/hooks/design-sync-reminder.sh` | the first Stop after a UI change (one-shot, then allows) | GUARD | NONE | — |
|
|
| `.claude/hooks/posttooluse-worktree-marker.sh` | nothing (writes the marker the worktree guard reads) | GUARD | MUTATION | `test_worktree_ownership_guard.py::test_MUTATION_a_marker_hook_that_stops_WRITING_makes_the_guard_go_quiet` |
|
|
| `.claude/hooks/prepush-clean-worktree-check.sh` | a push with uncommitted changes in the pushed set | GUARD | NONE | — |
|
|
| `.claude/hooks/prepush-donewhen.sh` | a direct push to `main` with unticked Done-when boxes | GUARD | NONE | — |
|
|
| `.claude/hooks/prepush-rebase-check.sh` | a push from a branch behind `origin/main` | GUARD | BEHAVIOUR-ONLY | `test_prepush_rebase_check_tag_exemption.py::test_zero_ref_lines_does_not_exempt` |
|
|
| `.claude/hooks/pretooluse-agent-model.sh` | an Agent dispatch naming no model (asks) | GUARD | NONE | — |
|
|
| `.claude/hooks/pretooluse-agent-ram.sh` | an Agent dispatch under 10% free RAM | GUARD | NONE | — |
|
|
| `.claude/hooks/pretooluse-bash-guard.sh` | a Bash call setting `ETV_UPDATE_GOLDENS` | GUARD | NONE | — |
|
|
| `.claude/hooks/pretooluse-bom-guard.sh` | a commit/push carrying a BOM in a touched `.cs` | GUARD | MUTATION | `test_bom_guard_detection.py::test_DISARMING_the_BOM_comparison_stops_detection` |
|
|
| `.claude/hooks/pretooluse-merge-consent.sh` | a PR merge without derived consent | GUARD | BEHAVIOUR-ONLY | `test_merge_consent_exemption.py::test_protected_path_on_a_LATER_page_is_still_seen` |
|
|
| `.claude/hooks/pretooluse-nav-guard.sh` | a browser navigate to a streaming URL | GUARD | NONE | — |
|
|
| `.claude/hooks/pretooluse-worktree-guard.sh` | a commit/merge in a foreign worktree | GUARD | MUTATION | `test_worktree_ownership_guard.py::test_MUTATION_disarming_the_guards_MARKER_READ_stops_the_deny` |
|
|
| `.husky/commit-msg` | a commit with no `Co-Authored-By` trailer | GUARD | NONE | — |
|
|
| `.husky/pre-commit` | a commit failing lint-staged, decisions, root-PNG or format | GUARD | NONE | — |
|
|
| `.husky/pre-push` | a push failing any pre-push hook or the SPA gate | GUARD | MUTATION | `test_prepush_unsets_git_env.py::test_MUTATION_DELETING_the_unset_lets_drift_through_silently` |
|
|
| `scripts/build_decisions_catalog.py` | the `decisions-guard` job, on a stale catalog | GUARD | MUTATION | `test_build_catalog_check_path.py::test_MUTATION_disarming_the_stale_comparison_stops_detection` |
|
|
| `scripts/check-doc-narrative.py` | nothing, by design (advisory `::warning::` only, exits 0 on every path (the error/degradation arms are defensive and unproven — see the record) — `docs.no-session-narrative` says a string predicate over prose may not be load-bearing) | TOOLING | NONE | — |
|
|
| `scripts/check-kickoff-guard.sh` | the `decisions-guard` job, on a revived #237 reference | GUARD | NONE | — |
|
|
| `scripts/check-required-contexts.sh` | the merge-consent hook, when `main`'s live required status checks no longer match `.gitea/required-status-contexts.json` (the snapshot `test_ci_dropped_step_guard.py` scopes itself to) | GUARD | MUTATION | `test_check_required_contexts.py::test_MUTATION_disarming_the_set_comparison_stops_every_drift_report` |
|
|
| `scripts/check-review-verdict.sh` | the merge-consent hook's verdict classification | GUARD | BEHAVIOUR-ONLY | `test_check_review_verdict.py::test_falseopen_token_must_be_a_whole_word` |
|
|
| `scripts/ci-detect-already-validated.sh` | nothing directly (feeds the skip gate) | GUARD | NONE | — |
|
|
| `scripts/ci-detect-docs-only.sh` | nothing directly (feeds the skip gate) | GUARD | NONE | — |
|
|
| `scripts/ci-peak-anon.sh` | nothing (samples container memory) | TOOLING | NONE | — |
|
|
| `scripts/ci-prove-ban-detects.sh` | the release path, if the delimiter ban is disarmed | GUARD | NONE | — |
|
|
| `scripts/ci-step-ran.sh` | the two required contexts, on a dropped step | GUARD | MUTATION | `test_ci_dropped_step_guard.py::test_dropping_ANY_single_step_FAILS_the_guard` |
|
|
| `scripts/ci-toolchain-image-resolves.sh` | the `toolchain-preflight` job, when the pinned CI toolchain image has been deleted from the registry | GUARD | MUTATION | `test_ci_toolchain_image_resolves.py::test_MUTATION_a_deleted_tag_is_reported_as_a_failure` |
|
|
| `scripts/decisions_validate.py` | the `decisions-guard` job, on a lifecycle fault | GUARD | MUTATION | `test_decisions_validate.py::test_main_actually_CALLS_the_wing_scan` |
|
|
| `scripts/e2e-functional.sh` | the Functional E2E job, on a failed HTTP contract assertion | GUARD | NONE | — |
|
|
| `scripts/e2e-local.sh` | nothing (boots a local instance) | TOOLING | NONE | — |
|
|
| `scripts/e2e-ui.sh` | nothing (drives the Playwright flows) | TOOLING | NONE | — |
|
|
| `scripts/hook-fire-log.sh` | nothing (records that each hook fired, and reports it) | TOOLING | NONE | — |
|
|
| `scripts/jq-preflight.sh` | the `script-tests` job, on a jq version change | GUARD | BEHAVIOUR-ONLY | `test_jq_preflight.py::test_below_the_floor_is_LOUD` |
|
|
| `scripts/post-review-verdict.sh` | nothing (writes the verdict status) | GUARD | BEHAVIOUR-ONLY | `test_post_review_verdict.py::test_never_retargets_the_verdict_at_the_new_head` |
|
|
| `scripts/pr-changed-files.sh` | the verdict exemption, on an incomplete enumeration | GUARD | BEHAVIOUR-ONLY | `test_pr_changed_files.py::test_a_SHORT_page_does_not_end_the_enumeration` |
|
|
| `scripts/prove-fix.sh` | the `prove-fix` job, on a commit whose `Proves:` trailer names a test that passes without the fix | GUARD | MUTATION | `test_prove_fix.py::test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test` |
|
|
| `scripts/update-openapi.sh` | nothing (regenerates the spec) | TOOLING | NONE | — |
|
|
| `scripts/tests/test_bom_guard_detection.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_build_catalog.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_build_catalog_check_path.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_check_required_contexts.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_check_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_check_doc_narrative.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_ci_dropped_step_guard.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_ci_image_pin_population.py` | the `script-tests` job, when a container job loses its pin | GUARD | MUTATION | `test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED` |
|
|
| `scripts/tests/test_ci_release_path_scan_job.py` | the `script-tests` job, on a weakened release-path scan job | GUARD | NONE | — |
|
|
| `scripts/tests/test_ci_status_context_uniqueness.py` | the `script-tests` job, when two CI jobs synthesize the same status-check context (which branch protection cannot tell apart) | GUARD | MUTATION | `test_ci_status_context_uniqueness.py::test_no_two_jobs_synthesize_the_SAME_status_context` |
|
|
| `scripts/tests/test_ci_toolchain_image_resolves.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_decisions_lib.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_decisions_validate.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_guard_inventory.py` | the `script-tests` job, on an unclassified guard or a stale proof ref | GUARD | MUTATION | `test_guard_inventory.py::test_the_inventory_covers_exactly_the_guards_that_exist` |
|
|
| `scripts/tests/test_guard_populations_derive_from_git.py` | the `script-tests` job, on a guard whose file population admits a file git does not track | GUARD | MUTATION | `test_guard_populations_derive_from_git.py::test_no_derivation_admits_an_untracked_file` |
|
|
| `scripts/tests/test_hook_fire_log.py` | the `script-tests` job, on a hook that stops reporting that it fired, or whose reporting changes what the harness sees | GUARD | MUTATION | `test_hook_fire_log.py::test_a_hook_that_LOSES_its_instrumentation_is_DETECTED` |
|
|
| `scripts/tests/test_jq_preflight.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_mcp_smoke.py` | the `script-tests` job, when the MCP smoke checker accepts a server it should refuse — an impostor identity, a reply carrying only an id, a missing expected tool, or an `initialize` pre-answered before the request (the `tools/list` twin is held by two mechanisms jointly and is covered behaviourally, not by the declared clause) | GUARD | MUTATION | `test_mcp_smoke.py::test_MUTATION_a_PRE_ANSWERED_id_is_refused_because_the_request_ids_are_UNGUESSABLE` |
|
|
| `scripts/tests/test_merge_consent_base_change.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_merge_consent_exemption.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_merge_consent_required_check.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_migration_equivalence.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_mutation_harness.py` | the `script-tests` job, when a `MUTATION` row's declared clause no longer reddens the test the row names | GUARD | MUTATION | `test_mutation_harness.py::test_MUTATION_disarming_the_DIAGNOSTIC_gate_accepts_a_red_for_the_wrong_reason` |
|
|
| `scripts/tests/test_post_review_verdict.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_prepush_unsets_git_env.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_pr_changed_files.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_prepush_rebase_check_tag_exemption.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_optional_request_members.py` | the `script-tests` job, on an OpenAPI request schema that can silently drop a member with no stated disposition | GUARD | MUTATION | `test_optional_request_members.py::test_every_droppable_request_schema_has_a_stated_disposition` |
|
|
| `scripts/tests/test_prove_fix.py` | the `script-tests` job | PROOF | NONE | — |
|
|
| `scripts/tests/test_review_verdict_vocabulary.py` | the `script-tests` job, when the H10 verdict vocabulary stops being shared — a word added to the one declaration that fails to reach the write side or the read side | GUARD | MUTATION | `test_review_verdict_vocabulary.py::test_a_word_added_to_the_shared_source_reaches_BOTH_sides` |
|
|
| `scripts/tests/test_remote_state_inventory.py` | the `script-tests` job, on an executable that talks to a remote service with no row in `docs/remote-state-inventory.md` | GUARD | MUTATION | `test_remote_state_inventory.py::test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file` |
|
|
| `scripts/tests/test_workflow_persist_credentials.py` | the `script-tests` job, when an `actions/checkout` step no longer sets `persist-credentials: false` | GUARD | MUTATION | `test_workflow_persist_credentials.py::test_every_actions_checkout_DROPS_the_persisted_credential` |
|
|
| `scripts/tests/test_workflow_job_guards.py` | the `script-tests` job, on a workflow JOB that is a guard with no row in the workflow-job table below, or a job with no `CI_JOB_ROLE` at all | GUARD | MUTATION | `test_workflow_job_guards.py::test_the_inventory_covers_exactly_the_guard_JOBS_that_exist` |
|
|
| `scripts/tests/test_worktree_ownership_guard.py` | the `script-tests` job | PROOF | NONE | — |
|
|
|
|
## Workflow-job guards (ersatztv#786)
|
|
|
|
The `## Inventory` table above covers guard **files**. A guard can also be a workflow **job** — its
|
|
assertion written inline in YAML, or delegated to a script that already holds a file row. Those jobs
|
|
were outside every population here until #786; `pr-checks.yml:ci-image-pin`, which stated an
|
|
invariant it did not check (#774), was sitting in exactly that gap.
|
|
|
|
**"Which jobs are guards" is a judgement the filesystem cannot supply**, so it is declared where the
|
|
job is: every job in every tracked workflow carries `env.CI_JOB_ROLE`, one of `guard`, `report-only` or
|
|
`none`. `report-only` is the third value and needs its own sentence, because "guard vs none" does not
|
|
imply it: a `report-only` job produces a **verdict it cannot enforce** — every check step carries
|
|
`continue-on-error: true`, so it reports and never fails. `docs-reminder` is the only one.
|
|
Note the limit: nothing detects a `guard` job whose checks are all `continue-on-error`, because the
|
|
checker compares the marker against the ROW, never against the job's ability to fail.
|
|
|
|
`scripts/tests/test_workflow_job_guards.py` derives the population from the parsed workflows
|
|
in the **git index** and asserts set equality against this table in both directions — a new guard job
|
|
acquires a row before the suite goes green, and a row naming a job that stopped being a guard is
|
|
reported too.
|
|
|
|
**The line is what the job PRODUCES, not whether a step can fail.** A `guard` job's output is a
|
|
**verdict** — it exists to pass or fail on some condition. A `none` job's output is an **artifact**:
|
|
an image, a set of PRs. A `none` job can still go red, but that is an ERROR in producing the thing,
|
|
not a finding about the repo. `docker-build.yml::build` publishes an image and runs a smoke test on
|
|
it, and is `none`: the smoke test failing means the build did not work, not that an invariant was
|
|
violated.
|
|
|
|
Most jobs are therefore guards, and that is the correct outcome rather than a sign the marker
|
|
distinguishes nothing: CI here is overwhelmingly checks, and a rule that excluded them would have to
|
|
draw a line no one could restate. **Which jobs are `none` is read off the markers, not restated
|
|
here** — a list in prose is a second copy of the workflow that rots on the next job added, and the
|
|
table below is the only enumeration that a check keeps honest.
|
|
|
|
**A rejected alternative, recorded so it is not re-adopted.** Drawing the line at "a guard enforces
|
|
an invariant about the REPOSITORY, so a job exercising the PRODUCT is `none`" reads as more
|
|
principled and is wrong: it puts `test` and `migrations` outside the table, and those are the two
|
|
REQUIRED status contexts on `main` — precisely the jobs where a failure to fire is fail-OPEN against
|
|
branch protection, and precisely the gap #786 exists to close. `test` also runs the C# and
|
|
TypeScript structural guards named in scope-limit item 2, so "exercises the product" was never a
|
|
clean description of it.
|
|
|
|
`Assertion` says where the logic actually lives, because that determines what a proof could even
|
|
look like: an **inline** assertion cannot be unit-tested without running the job, so most carry
|
|
`NONE` and say so, rather than borrowing credibility from a neighbouring file's proof. Two inline
|
|
rows do cite a pytest — `ci-image-pin` and `set-verdict-status` — and each names which PART of the
|
|
job it covers, because a proof reference that covers a fraction must not read as covering the job.
|
|
|
|
| Job | Blocks | Kind | Assertion | Proof |
|
|
| --- | --- | --- | --- | --- |
|
|
| `dependency-scan.yml::scan` | a weekly/dispatch run, when `dotnet list package --vulnerable` reports a vulnerable direct or transitive package | GUARD | inline — `grep -q "has the following vulnerable packages"` over the restore output | NONE |
|
|
| `docker-build.yml::toolchain-preflight` | a merge, through the COMBINED commit status the consent gate reads (#598). **Not** the other jobs: nothing `needs:` it and it is not a required context, so it reports rather than gates. It exists because a job that CONSUMED the vanished image could not run to say so (#772) | GUARD | `scripts/ci-toolchain-image-resolves.sh` | that script's row above |
|
|
| `docker-build.yml::test` | a merge — the REQUIRED context `Build ErsatzTV Image / Build & test (.NET) (pull_request)` | GUARD | the `dotnet test` / `npm test` suites, which include the C# and TypeScript structural guards named in scope-limit item 2 above | `scripts/ci-step-ran.sh` covers a dropped step — every consequential step `mark`s and the final step `assert`s, which is the mechanism `test_ci_dropped_step_guard.py` enforces on this job; the suites carry their own assertions |
|
|
| `docker-build.yml::migrations` | a merge — the REQUIRED context `Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request)` | GUARD | `dotnet ef migrations has-pending-model-changes` plus apply-to-fresh-DB, per provider | `scripts/ci-step-ran.sh` covers a dropped step — same `mark`/`assert` mechanism, enforced on this job by `test_ci_dropped_step_guard.py` |
|
|
| `docker-build.yml::functional-e2e` | a PR or a push to `main`, on a broken HTTP contract or UI flow | GUARD | `scripts/e2e-functional.sh`, which holds the HTTP contract assertions and is the only `GUARD` of the three. `scripts/e2e-ui.sh` DRIVES the Playwright specs (the UI assertions live in those specs, not in the script) and `scripts/e2e-local.sh` only BOOTS the instance they run against; both are `TOOLING` above, and naming either as the assertion would credit the coverage to a script that does not provide it | `scripts/e2e-functional.sh`'s row above (`GUARD`, `NONE`); no dropped-step guard — see the decision table below |
|
|
| `docker-build.yml::scan` | a merge and the release path, when a banned expression delimiter appears in the `run:` body of a delimiter-ban job (`DELIMITER_BAN_JOBS` = the required contexts **plus `build`**, which is wider than the marked set). It carries no `if:`, so it runs on PRs too and a red blocks through the combined status | GUARD | the ban assertion is `test_ci_dropped_step_guard.py::test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body` — note that file is graded `PROOF` above for its `ci-step-ran.sh` role, and also carries this GUARD-shaped assertion; `test_ci_release_path_scan_job.py` (`GUARD`) protects the job that runs it, and `scripts/ci-prove-ban-detects.sh` proves the ban still detects | `scripts/ci-prove-ban-detects.sh`'s row above — which itself carries `NONE`, as do the two pytest files, so this row claims no proof at all |
|
|
| `docker-build.yml::api-docs` | a PR whose committed OpenAPI spec, endpoint index or SPA client types are stale | GUARD | inline — `git diff --exit-code` after re-running `scripts/update-openapi.sh` | NONE |
|
|
| `docker-build.yml::format` | a PR whose changed `.cs` files do not conform to `.editorconfig` | GUARD | inline — `dotnet format --verify-no-changes` | NONE |
|
|
| `pr-checks.yml::ci-image-pin` | a PR whose `ersatztv-ci` pin is not the 7-char sha of the last commit to touch `docker/ci` | GUARD | inline | `test_ci_image_pin_population.py::test_a_single_job_losing_its_pin_is_DETECTED` covers the POPULATION half only; the git-history half has no test |
|
|
| `pr-checks.yml::docs-reminder` | **nothing.** Both of its check steps — and the `Set up Python` between them — carry `continue-on-error: true`, so no finding it makes can fail the job. (`Checkout` does not, so the job can still go red on infrastructure — never on a finding.) | REPORT-ONLY | inline + `scripts/check-doc-narrative.py` | n/a — it cannot go red |
|
|
| `pr-checks.yml::decisions-guard` | a PR whose decision records fail lifecycle validation, whose catalog is stale, or whose kickoff file drifted | GUARD | `scripts/decisions_validate.py`, `scripts/build_decisions_catalog.py`, `scripts/check-kickoff-guard.sh` | those scripts' rows above |
|
|
| `pr-checks.yml::prove-fix` | a PR whose `Proves:` trailer names a test that passes without the fix | GUARD | inline + `scripts/prove-fix.sh` | that script's row above |
|
|
| `pr-checks.yml::script-tests` | a PR failing ruff or the `scripts/tests` suite — this job is the RUNNER for every `scripts/tests/` row above | GUARD | inline (the ruff population guards) + pytest | the suite it runs |
|
|
| `review-verdict.yml::set-verdict-status` | a MERGE, by withholding the branch-protection-required `review-verdict/h10` status | GUARD | inline | `scripts/tests/test_pr_changed_files.py` guards its changed-file derivation |
|
|
|
|
### The four jobs with no dropped-step guard, decided per job (ersatztv#786)
|
|
|
|
`scripts/ci-step-ran.sh` turns a step the runner silently DROPPED into a red job. Its scope is
|
|
derived from `.gitea/required-status-contexts.json` (#787), so today it is enforced on `test` and
|
|
`migrations` — the two required Actions contexts, where a drop is fail-OPEN against branch
|
|
protection. `toolchain-preflight` and `scan` carry the `mark`/`assert` protocol voluntarily without
|
|
being in the enforced scope. That leaves the jobs below, which #786 listed as unexplained;
|
|
each is recorded here rather than given a marker reflexively.
|
|
|
|
| Job | Decision |
|
|
| --- | --- |
|
|
| `functional-e2e` | **No guard.** Not a required context, and it runs only on PRs and pushes to `main`. A dropped step there cannot send a required check green; it costs coverage on a run a human is already reading. |
|
|
| `api-docs` | **No guard.** Not a required context. Its single assertion is the LAST step, and a dropped `git diff --exit-code` leaves the regeneration steps' output uncommitted — visible in the job log rather than silently green. |
|
|
| `format` | **No guard.** Not a required context, and the job is three steps: a drop is the difference between a formatting red and no formatting check, on a PR whose other gates still run. |
|
|
| `build` | **Partial, and the residual is genuine.** Its delimiter class is covered — the `scan` job re-runs the ban test on the release path and is a `needs:` of `build` (#767). Per-step markers would additionally catch a drop caused by something else, and need a bucket modelling the publish-ref `if:` that the guard's always/gated buckets do not express. Stated at `.gitea/workflows/docker-build.yml`'s "Smoke + IPTV E2E" step. |
|
|
|
|
Widening `ci-step-ran.sh` beyond the required contexts is deliberately not done here:
|
|
`test_ci_dropped_step_guard.py` argues the ban is scoped to where a drop is CONSEQUENTIAL, and a
|
|
marker protocol applied everywhere costs a maintained bucket per job for jobs whose failure mode is
|
|
"a human reads a shorter log".
|
|
|
|
|
|
## The `MUTATION` column is executed, not asserted
|
|
|
|
Every row graded `MUTATION` carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`.
|
|
`scripts/tests/test_mutation_harness.py` applies each one to an isolated copy of this repository and
|
|
requires that row's own named test to go red; the manifest and the `MUTATION` rows are compared for
|
|
set equality in both directions, so the grade and the mutation cannot drift apart. Adding a row
|
|
graded `MUTATION` without declaring its clause fails the suite. Full contract and its limits:
|
|
`docs/decisions/records/testing/mutation-claims-are-executed.md`.
|
|
|
|
Two things that column still does not say. It does not say the declared clause is the ONLY thing the
|
|
guard hangs on — for three rows (measured 2026-08-22) the redness arrives through the
|
|
proof test's own "this clause has moved, RETARGET it" assertion rather than through changed
|
|
behaviour, which catches the recorded proof going stale but not much else. And one row is graded `DETECTOR` rather than `CLAUSE` in the
|
|
manifest: `instrumentation_faults` accumulates from four independent arms and a stripped hook trips
|
|
three at once, so no single-arm disarm reddens its proof. That finer mutation is carried as data and
|
|
re-run every suite, and must keep surviving — if it starts reddening, the guard has become
|
|
clause-provable and the entry is regraded.
|
|
|
|
## What the numbers say
|
|
|
|
44 guards, 6 tooling scripts, 21 proof files. **22 guards carry a mutation proof; 6 are
|
|
behaviour-only; 16 have none.** These figures are asserted against the table by
|
|
`test_the_summary_counts_match_the_table` — they were wrong in the first draft (28/4/6/3/19 against
|
|
a table holding 27/5/6/3/18), because a hand-maintained summary of a table is a second copy of it,
|
|
which is the duplication family this change argues against. Both cold reviewers found the error
|
|
independently.
|
|
|
|
**One guard was WITHDRAWN rather than shipped**, and it is worth a line here because the inventory
|
|
is where a future session will look for it. A `test_review_verdict_vocabulary_parity.py` asserted
|
|
set equality between the H10 verdict vocabulary's write side and read side by extracting both from
|
|
shell source with regexes. Six cold-review rounds each found another shell construction that either
|
|
escaped it or made it red on a correct tree. It was deleted rather than patched a seventh time: the
|
|
`testing.guard-derives-population-from-source` record says a weak detector is itself the
|
|
symptom-keyed mistake, and this was the empirical demonstration. The underlying duplication is real
|
|
and is now UNMITIGATED — tracked in #788, which fixes it by removing the second copy.
|
|
|
|
**What that file asserted, and where each part went** — enumerated because the first attempt at this
|
|
withdrawal silently dropped one and claimed otherwise, which is
|
|
`process.enumerate-workaround-behaviors-before-deleting` failing on its own removal:
|
|
|
|
| Invariant it asserted | Where it is now |
|
|
|---|---|
|
|
| Bidirectional write/read vocabulary parity | **LOST.** The reason for the withdrawal; #788 |
|
|
| No READ-side word is both positive and negative | **LOST** as a universal property; #788 |
|
|
| No WRITE-side word is both success and failure | **LOST** as a universal property; #788 |
|
|
| Strict extractor consumes every case arm | Retired with the extractor; #788 |
|
|
| `POS_RE`/`NEG_RE` each assigned exactly once | Retired with the extractor; #788 |
|
|
| Both extractors find non-empty sets (anti-vacuity) | Retired with the extractor; #788 |
|
|
| A wrong write source makes the extractors disagree | Retired with the extractor; #788 |
|
|
|
|
Two guards were ADDED in its place. They are **not** replacements for rows 2 and 3 and the table
|
|
above deliberately does not list them as such — an earlier draft did, and cold review was right that
|
|
relabelling a lost invariant as a narrower surviving one is the exact failure this table exists to
|
|
prevent, committed by the table:
|
|
|
|
| Added | What it pins |
|
|
|---|---|
|
|
| `test_check_review_verdict.py::test_each_verdict_word_retains_its_established_polarity` | the five established tokens still READ with their established polarity |
|
|
| `test_post_review_verdict.py::test_each_verdict_word_posts_its_established_polarity` | the five established tokens still POST with their established polarity |
|
|
|
|
**Polarity is strictly weaker than disjointness, and the gap is demonstrable.** Put `MERGEABLE` in
|
|
BOTH `case` arms of `post-review-verdict.sh`: the success arm wins, every polarity assertion stays
|
|
green, and the withdrawn disjointness test failed. What the two added guards do catch is the
|
|
dangerous direction — a token that a reviewer means as `BLOCKED` silently reading or posting as
|
|
approval, which writes a green `review-verdict/h10`. A token added to one script and not the other
|
|
is untested by either, which is the whole of #788.
|
|
|
|
**A guard was WITHDRAWN from `test_hook_fire_log.py` after four iterations**, recorded here because
|
|
this is where a future session will look for it. A `test_no_redirection_in_the_sink_PRECEDES_its_stderr_redirect`
|
|
generalised the `exec … 2>/dev/null` rule to any command by pattern-matching shell source. Each
|
|
iteration fixed a false positive or false negative the previous one introduced — `>/dev/null 2>&1`
|
|
flagged wrongly, then operators preceded by a digit (`exec 0<`) missed entirely, then multi-command
|
|
lines flagged wrongly — and cold review then constructed more of both (`[[ "$x" < "$y" ]] 2>/dev/null`,
|
|
a backslash continuation, a `>` inside a quoted string). Deleted rather than patched a fifth time,
|
|
on the same reasoning as the vocabulary-parity withdrawal above.
|
|
|
|
**`test_hook_fire_log.py` proves THREE clauses, and its row claims two of them.** Coverage (every
|
|
hook reports that it fired) via `test_a_hook_that_LOSES_its_instrumentation_is_DETECTED`;
|
|
transparency (the wrapper changes nothing the harness can see) via
|
|
`test_instrumentation_changes_NOTHING_the_harness_can_see` with its mutation proof
|
|
`test_DELETING_the_replay_makes_the_differential_go_RED`; and placement (`etv_hook_fire_begin` must
|
|
precede the stdin read) via `test_begin_placed_AFTER_the_stdin_read_is_DETECTED`. The row's `Blocks`
|
|
column covers the first two — "stops reporting that it fired, **or whose reporting changes what the
|
|
harness sees**" — and the `Proof ref` column holds one ref because the column holds one, not because
|
|
the second is unproven. Placement is proved and unclaimed.
|
|
|
|
**Its `test_the_suite_does_not_write_to_the_PRODUCTION_log` is narrower than its docstring**, which
|
|
says `conftest.py` "must isolate every test, not just this file's". What it checks is that the
|
|
fixture set `ETV_HOOK_FIRE_LOG_DIR` *for the test currently running* and that a hook it drives in
|
|
its own sandbox does not touch the real log. It cannot see another suite that snapshots
|
|
`os.environ` at IMPORT time — before the autouse fixture runs — and hands that stale mapping to its
|
|
subprocesses. That suite's hooks then write to `$HOME/.cache/ersatztv/hook-fire/` while every
|
|
assertion stays green, because the fire-log library is fail-open. #785 shipped exactly that defect,
|
|
inside the file added to prove those very hooks.
|
|
|
|
The reproduction is the part worth keeping: restore a module-level `{**os.environ}` snapshot — or
|
|
leave the helper correct and point a single `env=` argument at one — then run that file and count
|
|
records for its synthetic session ids. **58 per run**, measured identically on macOS/git 2.55 and
|
|
Linux/git 2.47.3. (The accumulated total observed before the fix ran to four figures across many
|
|
runs; that is an observation rather than a reproducible measurement, so check the per-run figure.)
|
|
|
|
The pin is `test_worktree_ownership_guard.py::test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir`,
|
|
which asserts the EFFECT — records land in the fixture's dir — rather than the shape of the fix. Its
|
|
predecessor asserted the helper's return value, and cold review showed that leaves the call site
|
|
unguarded: `_env()` correct, one `env=` reverted, all tests green, records still leaking. Pinning to
|
|
a hand-written revert rather than to the property is `verify-against-the-REAL-predecessor`.
|
|
|
|
It is still narrower than the property: it guards the launch path it drives. A second launcher in
|
|
the same file that passed a stale environment would leave it green — measured, 18 records — because
|
|
the hooks it drives would still log correctly. Every hook in that file goes through one helper
|
|
today, which is what makes it sufficient there. Generalising it is #809, and the reason that is hard
|
|
is that the obvious version races against a real session's hooks firing during the run.
|
|
|
|
**That unproven set carries no number here on purpose.** It restated the count, drifted the moment
|
|
the BOM guard was regraded, and `test_the_summary_counts_match_the_table` cannot see it — the parser
|
|
checks the formatted summary sentence and nothing else. A second hand-maintained copy of a number is
|
|
the duplication family this file argues against, so the copy is removed rather than corrected: the
|
|
set is the eight `.claude/hooks/` guards enumerated in the standing list below, and the count is in
|
|
the summary above. Naming them by event is what made the earlier wording wrong twice — they do not
|
|
share one event, and `design-sync-reminder.sh` is registered on **both** `PreToolUse` and `Stop` in
|
|
`.claude/settings.json`, so any "every X hook" phrasing double-counts it.
|
|
|
|
**They are now observable but still unproven, and the two words carry different weight.** Observable:
|
|
every hook records its own execution through `scripts/hook-fire-log.sh`, so "did this hook fire, and
|
|
what did it decide" is a measurement — run `scripts/hook-fire-log.sh report` (#776). Unproven:
|
|
nobody has demonstrated any of them is load-bearing — the first group in the standing list below.
|
|
Observability tells you a guard ran; only a mutation tells you it would have caught anything. The
|
|
BOM guard is the case that shows why the distinction matters — it was firing on every commit the
|
|
whole time it was fail-open.
|
|
|
|
The gaps are not uniform in cost, and the ranking that matters is *what a silent failure would let
|
|
through*, not test count. **All four ranked entries now carry clause-level mutation proofs (#785);
|
|
they are kept here with what each mutation established, because the ranking is the reusable part and
|
|
because two of them turned out to be worse than the ranking predicted.**
|
|
|
|
1. ~~`pretooluse-bom-guard.sh`~~ — **proven, and it was fail-open the whole time.** Ranked first
|
|
because the defect it guards has recurred three times (#311, #402, #405); that ranking turned out
|
|
to be right for a worse reason than intended. It detected a BOM with `xxd -p`, and `xxd` ships
|
|
with vim and is **absent on the Linux CI runner**, so the comparison never matched and every BOM
|
|
was allowed in silence. `od` now. The lesson for the rows below: an unproven guard is not merely
|
|
untested, it is a guard whose *current* behaviour nobody has established.
|
|
2. ~~`pretooluse-worktree-guard.sh` + `posttooluse-worktree-marker.sh`~~ — **proven as a pair.**
|
|
Four clauses were disarmed and witnessed red: the guard's marker read; the guard's ownership
|
|
comparison (inverted, because disarming it the other way only makes the guard deny more and every
|
|
deny assertion stays green); the `commit|merge` alternation, whose `merge` half every other case
|
|
in the file left untested while guarding the plumbing-merge path; and — the one that could not
|
|
exist while the halves were tested apart — the *marker hook's write*, asserted against the
|
|
*guard's* decision. Both hooks are
|
|
deliberately fail-open, so an absent mechanism and a working one produce the identical "commit
|
|
allowed".
|
|
3. ~~`.husky/pre-push:11`'s `unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE`~~ — **proven, and the case
|
|
is the normal one rather than an edge.** Git exports `GIT_DIR` to `pre-push` **when the push
|
|
comes from a worktree** and not from the main tree; `process.shared-tree-readonly` makes the
|
|
worktree the mandated way to work here, so every push takes the exposed path. With `GIT_DIR` set
|
|
and `GIT_WORK_TREE` unset git stops discovering the repo and treats the *current directory* as
|
|
the work tree, so `cd web && npm run check:api`'s `git diff --exit-code` compares against index
|
|
paths that do not exist and reports no diff. Both the deletion and the relocation are proved.
|
|
4. ~~`scripts/build_decisions_catalog.py`~~ — **proven, including the wiring.** The `--check`
|
|
comparison is mutated directly, and a separate subprocess case runs the command *derived from
|
|
`pr-checks.yml`* against a copied corpus. That second case is not redundant: replacing
|
|
`raise SystemExit(main())` with a bare `main()` leaves the script printing
|
|
`docs/decisions/README.md is stale` on stderr while exiting **0**, and the workflow step reads
|
|
nothing but the exit code. Only the subprocess case reddens — the #751/#719 shape.
|
|
|
|
**What a file-level `MUTATION` grade does and does not claim, because three of these four rows are
|
|
multi-purpose files.** The grade covers *the clause the cited case actually mutates*, per
|
|
`testing.guard-ships-with-mutation-proof`; it is not a statement about every line in the file.
|
|
`.husky/pre-push` runs four other things, and its row asserts only that line 11 is load-bearing —
|
|
the three hooks it invokes carry their own rows and their own grades. Reading the row as "pre-push
|
|
is tested" is the same relabelling error the withdrawn-guard table above exists to prevent.
|
|
Clause-level grading is tracked in #790.
|
|
|
|
**The remaining unproven guards, and why each is still `NONE` rather than merely unattended** — the
|
|
third `## Done-when` box of #785, answered here rather than left implicit. They fall into three
|
|
groups, and the split is the point: the first two are backlogs, the third is not.
|
|
|
|
- **The eight remaining hook guards** — genuinely unproven, and a real backlog. Four are
|
|
`PreToolUse` (`pretooluse-agent-model.sh`, `pretooluse-agent-ram.sh`, `pretooluse-bash-guard.sh`,
|
|
`pretooluse-nav-guard.sh`), two are pre-push (`prepush-clean-worktree-check.sh`,
|
|
`prepush-donewhen.sh`), one is pre-commit (`decisions-guard.sh`) and one is registered on both
|
|
`PreToolUse` and `Stop` (`design-sync-reminder.sh`); they are grouped by their status, not by
|
|
their event, because the events do not partition them. They are now
|
|
*observable* (`scripts/hook-fire-log.sh report`, #776), which is a weaker claim than proven and is
|
|
stated as such above.
|
|
- **`.husky/commit-msg` and `.husky/pre-commit`** — unproven, and each carries its own clause, so
|
|
they are a backlog too rather than "covered by what they call". `pre-commit` dispatches to
|
|
`lint-staged` and `decisions-guard.sh`, but the root-level-`*.png` refusal and the
|
|
`dotnet format whitespace --verify-no-changes` block are its own. `commit-msg` is not a dispatcher
|
|
at all: it is one `grep -q '^Co-Authored-By:'` plus a `MERGE_HEAD` exemption, and nothing tests
|
|
that the exemption fires only for merges. Neither is covered by what it calls — the trap is to read
|
|
"it dispatches to guards" as "its own clauses are guarded".
|
|
- **`ci-detect-already-validated.sh`, `ci-detect-docs-only.sh`, `ci-prove-ban-detects.sh`,
|
|
`e2e-functional.sh`, `check-kickoff-guard.sh`, `test_ci_release_path_scan_job.py`** — not
|
|
unattended either, but each needs its own harness rather than a proof written to clear a row.
|
|
`ci-prove-ban-detects.sh` is the exception noted in `testing.guard-ships-with-mutation-proof`: it
|
|
runs its own mutation at CI time, because what it proves is disarmable from inside pytest.
|
|
|
|
That list is prose and nothing checks it, which is the honest limit — a guard moving out of a group
|
|
will not redden anything. It is here so the next session inherits *why* a row is `NONE`, which is
|
|
the distinction #785 asked for; the machine-checked half remains the table.
|
|
|
|
Filling the rest is tracked rather than done in one pass, deliberately: a mutation proof written to
|
|
close a row is the kind of test that passes for the wrong reason.
|