diff --git a/docs/defect-shapes-773.md b/docs/defect-shapes-773.md index bf073c669..c54a37906 100644 --- a/docs/defect-shapes-773.md +++ b/docs/defect-shapes-773.md @@ -474,23 +474,65 @@ and 5,102 `Read` calls over the same corpus, so the search was demonstrably capa evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`). -### 5.4 We cannot currently tell whether our own hooks fire +Wiring was the strongest claim available for the hook rows when this table was written. It is no +longer the ceiling — §5.4 is now measured — so read the hook rows against +`scripts/hook-fire-log.sh report`, not against this paragraph. -The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own -evidence. In this harness version only **`Stop`** hooks emit a structured record -(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that -matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**. -What the audit could count for those hooks was filename mentions in settings dumps and prior audits, -which is not proof of execution. Only `design-sync-reminder.sh`'s Stop path and the three husky -message strings quoted in §5.3 are confirmed fires; the `.claude/hooks/` scripts are confirmed -*wired*, which is a different claim. +### 5.4 Whether our own hooks fire — now measured (#776 closed this) -So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That -is Family B (the check never ran) applied to the hook layer itself, and this project has already -paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and -#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it -rather than infer it.** A one-line append to a session-scoped log from each hook would convert every -row in that table from inference to measurement, and it is cheap. +**The finding as originally recorded.** In this harness version only `Stop` hooks emit a structured +record (`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* +guard that matters here: merge consent, worktree, BOM, agent model/RAM — leave no durable execution +trace. What this audit could count for those hooks was filename mentions in settings dumps, which is +not evidence of execution. So the guards this repo relies on most were exactly the ones whose +execution could only be inferred: Family B (the check never ran) applied to the hook layer itself, +after this project had already paid for it twice at the CI layer (#751, #756). + +**What replaced the inference.** Every hook now records its own execution through one shared sink, +`scripts/hook-fire-log.sh` (`testing.hook-reports-its-own-execution`): a `fire` record on entry and +an `exit` record carrying the status and the decision, where **the decision is parsed from the bytes +the hook actually emitted** rather than declared by its author. Read it with +`scripts/hook-fire-log.sh report [--all]`. + +**Measured 2026-08-14** — a scripted headless session (a plain Bash call, a Bash call carrying +`ETV_UPDATE_GOLDENS=1`, a `Write` to a SPA file, and one `Agent` dispatch) plus a real `git commit` +and a `git push --dry-run` in a worktree. Snapshot boundary: this is one deliberately-constructed +window, not a corpus statistic. + +| Hook | Event | Fires | Decisions observed | +|---|---|---|---| +| `pretooluse-bash-guard` | PreToolUse/Bash | 2 | `deny` 1, `no-op` 1 | +| `pretooluse-bom-guard` | PreToolUse/Bash | 2 | `no-op` 2 | +| `pretooluse-worktree-guard` | PreToolUse/Bash | 2 | `no-op` 2 | +| `posttooluse-worktree-marker` | PostToolUse/Bash | 1 | `no-op` 1 | +| `design-sync-reminder` | PreToolUse/Write + Stop | 4 | `context` 1, `block` 1, `no-op` 2 | +| `pretooluse-agent-model` | PreToolUse/Agent | 2 | `no-op` 2 | +| `pretooluse-agent-ram` | PreToolUse/Agent | 2 | `no-op` 2 | +| `decisions-guard` | git pre-commit | 2 | `pass` 2 | +| `prepush-clean-worktree-check` | git pre-push | 1 | `pass` 1 | +| `prepush-donewhen` | git pre-push | 1 | `pass` 1 | +| `prepush-rebase-check` | git pre-push | 1 | `pass` 1 | +| `pretooluse-nav-guard` | PreToolUse/navigate | **0** | not exercised — needs a live browser session | +| `pretooluse-merge-consent` | PreToolUse/PR write | **0** | not exercised — needs a real merge attempt | + +**11 of 13 hooks are confirmed firing, with the decision each reached.** The `deny` row is the load- +bearing one: `pretooluse-bash-guard` did not merely run, it *blocked* the `ETV_UPDATE_GOLDENS=1` +probe, so at least one guard in this set is demonstrably live rather than merely present. + +**A zero is two findings wearing one number** — a hook that is broken, and a hook whose trigger did +not occur — and the report says so rather than presenting a zero as a verdict. The two zeroes here +are the second kind: neither can be exercised without a browser session or a real merge, and +manufacturing a merge to observe the merge guard is a worse idea than the gap it would close. +Distinguishing the two cases stays a human judgement; what changed is that the question is now +askable at all. + +**The measurement earned its keep on its first run.** `pretooluse-agent-model` was recorded deciding +`no-op` on an `Agent` dispatch where `ask` was expected. Previously that would have been an +unanswerable suspicion about a guard nobody could observe; instead the real payload was dumped and +it carried `model: haiku`, so `no-op` was correct and there was no defect. The general form is the +standing #756 lesson — **make the system report it rather than infer it** — and note what it +displaced: an argument from the hook's source about what it "must" do, which is the reasoning shape +§3 measures going wrong repeatedly. ### 5.5 Tools we do *not* have that would address a named class @@ -533,7 +575,7 @@ This issue is analysis; it spawns implementation rather than doing it. |---|---|---|---| | #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, 27 records (39%) | high | | #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B, 9 records (13%) | high | -| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high | +| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high — **DONE**, §5.4 is measured | | #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium | | #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium | | #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |