docs(776): re-derive 773 section 5.4 from measurement, not inference
Section 5.4 said we cannot tell whether our own hooks fire, and could only count filename mentions in settings dumps. It now carries a per-hook table with the decision each hook reached: 11 of 13 confirmed firing, including a real deny from pretooluse-bash-guard blocking an ETV_UPDATE_GOLDENS probe, which is what distinguishes "this guard ran" from "this guard is present". The two zeroes are named as the not-exercised kind rather than presented as a verdict — nav-guard needs a live browser and merge-consent needs a real merge, and manufacturing a merge to watch the merge guard is worse than the gap. Refs: #776 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
+58
-16
@@ -474,23 +474,65 @@ and 5,102 `Read` calls over the same corpus, so the search was demonstrably capa
|
||||
evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message
|
||||
missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`).
|
||||
|
||||
### 5.4 We cannot currently tell whether our own hooks fire
|
||||
Wiring was the strongest claim available for the hook rows when this table was written. It is no
|
||||
longer the ceiling — §5.4 is now measured — so read the hook rows against
|
||||
`scripts/hook-fire-log.sh report`, not against this paragraph.
|
||||
|
||||
The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own
|
||||
evidence. In this harness version only **`Stop`** hooks emit a structured record
|
||||
(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that
|
||||
matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**.
|
||||
What the audit could count for those hooks was filename mentions in settings dumps and prior audits,
|
||||
which is not proof of execution. Only `design-sync-reminder.sh`'s Stop path and the three husky
|
||||
message strings quoted in §5.3 are confirmed fires; the `.claude/hooks/` scripts are confirmed
|
||||
*wired*, which is a different claim.
|
||||
### 5.4 Whether our own hooks fire — now measured (#776 closed this)
|
||||
|
||||
So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That
|
||||
is Family B (the check never ran) applied to the hook layer itself, and this project has already
|
||||
paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and
|
||||
#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it
|
||||
rather than infer it.** A one-line append to a session-scoped log from each hook would convert every
|
||||
row in that table from inference to measurement, and it is cheap.
|
||||
**The finding as originally recorded.** In this harness version only `Stop` hooks emit a structured
|
||||
record (`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every*
|
||||
guard that matters here: merge consent, worktree, BOM, agent model/RAM — leave no durable execution
|
||||
trace. What this audit could count for those hooks was filename mentions in settings dumps, which is
|
||||
not evidence of execution. So the guards this repo relies on most were exactly the ones whose
|
||||
execution could only be inferred: Family B (the check never ran) applied to the hook layer itself,
|
||||
after this project had already paid for it twice at the CI layer (#751, #756).
|
||||
|
||||
**What replaced the inference.** Every hook now records its own execution through one shared sink,
|
||||
`scripts/hook-fire-log.sh` (`testing.hook-reports-its-own-execution`): a `fire` record on entry and
|
||||
an `exit` record carrying the status and the decision, where **the decision is parsed from the bytes
|
||||
the hook actually emitted** rather than declared by its author. Read it with
|
||||
`scripts/hook-fire-log.sh report [--all]`.
|
||||
|
||||
**Measured 2026-08-14** — a scripted headless session (a plain Bash call, a Bash call carrying
|
||||
`ETV_UPDATE_GOLDENS=1`, a `Write` to a SPA file, and one `Agent` dispatch) plus a real `git commit`
|
||||
and a `git push --dry-run` in a worktree. Snapshot boundary: this is one deliberately-constructed
|
||||
window, not a corpus statistic.
|
||||
|
||||
| Hook | Event | Fires | Decisions observed |
|
||||
|---|---|---|---|
|
||||
| `pretooluse-bash-guard` | PreToolUse/Bash | 2 | `deny` 1, `no-op` 1 |
|
||||
| `pretooluse-bom-guard` | PreToolUse/Bash | 2 | `no-op` 2 |
|
||||
| `pretooluse-worktree-guard` | PreToolUse/Bash | 2 | `no-op` 2 |
|
||||
| `posttooluse-worktree-marker` | PostToolUse/Bash | 1 | `no-op` 1 |
|
||||
| `design-sync-reminder` | PreToolUse/Write + Stop | 4 | `context` 1, `block` 1, `no-op` 2 |
|
||||
| `pretooluse-agent-model` | PreToolUse/Agent | 2 | `no-op` 2 |
|
||||
| `pretooluse-agent-ram` | PreToolUse/Agent | 2 | `no-op` 2 |
|
||||
| `decisions-guard` | git pre-commit | 2 | `pass` 2 |
|
||||
| `prepush-clean-worktree-check` | git pre-push | 1 | `pass` 1 |
|
||||
| `prepush-donewhen` | git pre-push | 1 | `pass` 1 |
|
||||
| `prepush-rebase-check` | git pre-push | 1 | `pass` 1 |
|
||||
| `pretooluse-nav-guard` | PreToolUse/navigate | **0** | not exercised — needs a live browser session |
|
||||
| `pretooluse-merge-consent` | PreToolUse/PR write | **0** | not exercised — needs a real merge attempt |
|
||||
|
||||
**11 of 13 hooks are confirmed firing, with the decision each reached.** The `deny` row is the load-
|
||||
bearing one: `pretooluse-bash-guard` did not merely run, it *blocked* the `ETV_UPDATE_GOLDENS=1`
|
||||
probe, so at least one guard in this set is demonstrably live rather than merely present.
|
||||
|
||||
**A zero is two findings wearing one number** — a hook that is broken, and a hook whose trigger did
|
||||
not occur — and the report says so rather than presenting a zero as a verdict. The two zeroes here
|
||||
are the second kind: neither can be exercised without a browser session or a real merge, and
|
||||
manufacturing a merge to observe the merge guard is a worse idea than the gap it would close.
|
||||
Distinguishing the two cases stays a human judgement; what changed is that the question is now
|
||||
askable at all.
|
||||
|
||||
**The measurement earned its keep on its first run.** `pretooluse-agent-model` was recorded deciding
|
||||
`no-op` on an `Agent` dispatch where `ask` was expected. Previously that would have been an
|
||||
unanswerable suspicion about a guard nobody could observe; instead the real payload was dumped and
|
||||
it carried `model: haiku`, so `no-op` was correct and there was no defect. The general form is the
|
||||
standing #756 lesson — **make the system report it rather than infer it** — and note what it
|
||||
displaced: an argument from the hook's source about what it "must" do, which is the reasoning shape
|
||||
§3 measures going wrong repeatedly.
|
||||
|
||||
### 5.5 Tools we do *not* have that would address a named class
|
||||
|
||||
@@ -533,7 +575,7 @@ This issue is analysis; it spawns implementation rather than doing it.
|
||||
|---|---|---|---|
|
||||
| #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, 27 records (39%) | high |
|
||||
| #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B, 9 records (13%) | high |
|
||||
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high |
|
||||
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high — **DONE**, §5.4 is measured |
|
||||
| #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium |
|
||||
| #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium |
|
||||
| #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |
|
||||
|
||||
Reference in New Issue
Block a user