docs(776): re-derive 773 section 5.4 from measurement, not inference

Section 5.4 said we cannot tell whether our own hooks fire, and could only count
filename mentions in settings dumps. It now carries a per-hook table with the
decision each hook reached: 11 of 13 confirmed firing, including a real deny
from pretooluse-bash-guard blocking an ETV_UPDATE_GOLDENS probe, which is what
distinguishes "this guard ran" from "this guard is present".

The two zeroes are named as the not-exercised kind rather than presented as a
verdict — nav-guard needs a live browser and merge-consent needs a real merge,
and manufacturing a merge to watch the merge guard is worse than the gap.

Refs: #776

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-14 18:52:57 +02:00
co-authored by Claude Opus 5
parent fc65b5fbe3
commit 773151be14
+58 -16
View File
@@ -474,23 +474,65 @@ and 5,102 `Read` calls over the same corpus, so the search was demonstrably capa
evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message
missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`).
### 5.4 We cannot currently tell whether our own hooks fire
Wiring was the strongest claim available for the hook rows when this table was written. It is no
longer the ceiling — §5.4 is now measured — so read the hook rows against
`scripts/hook-fire-log.sh report`, not against this paragraph.
The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own
evidence. In this harness version only **`Stop`** hooks emit a structured record
(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that
matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**.
What the audit could count for those hooks was filename mentions in settings dumps and prior audits,
which is not proof of execution. Only `design-sync-reminder.sh`'s Stop path and the three husky
message strings quoted in §5.3 are confirmed fires; the `.claude/hooks/` scripts are confirmed
*wired*, which is a different claim.
### 5.4 Whether our own hooks fire — now measured (#776 closed this)
So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That
is Family B (the check never ran) applied to the hook layer itself, and this project has already
paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and
#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it
rather than infer it.** A one-line append to a session-scoped log from each hook would convert every
row in that table from inference to measurement, and it is cheap.
**The finding as originally recorded.** In this harness version only `Stop` hooks emit a structured
record (`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every*
guard that matters here: merge consent, worktree, BOM, agent model/RAM — leave no durable execution
trace. What this audit could count for those hooks was filename mentions in settings dumps, which is
not evidence of execution. So the guards this repo relies on most were exactly the ones whose
execution could only be inferred: Family B (the check never ran) applied to the hook layer itself,
after this project had already paid for it twice at the CI layer (#751, #756).
**What replaced the inference.** Every hook now records its own execution through one shared sink,
`scripts/hook-fire-log.sh` (`testing.hook-reports-its-own-execution`): a `fire` record on entry and
an `exit` record carrying the status and the decision, where **the decision is parsed from the bytes
the hook actually emitted** rather than declared by its author. Read it with
`scripts/hook-fire-log.sh report [--all]`.
**Measured 2026-08-14** — a scripted headless session (a plain Bash call, a Bash call carrying
`ETV_UPDATE_GOLDENS=1`, a `Write` to a SPA file, and one `Agent` dispatch) plus a real `git commit`
and a `git push --dry-run` in a worktree. Snapshot boundary: this is one deliberately-constructed
window, not a corpus statistic.
| Hook | Event | Fires | Decisions observed |
|---|---|---|---|
| `pretooluse-bash-guard` | PreToolUse/Bash | 2 | `deny` 1, `no-op` 1 |
| `pretooluse-bom-guard` | PreToolUse/Bash | 2 | `no-op` 2 |
| `pretooluse-worktree-guard` | PreToolUse/Bash | 2 | `no-op` 2 |
| `posttooluse-worktree-marker` | PostToolUse/Bash | 1 | `no-op` 1 |
| `design-sync-reminder` | PreToolUse/Write + Stop | 4 | `context` 1, `block` 1, `no-op` 2 |
| `pretooluse-agent-model` | PreToolUse/Agent | 2 | `no-op` 2 |
| `pretooluse-agent-ram` | PreToolUse/Agent | 2 | `no-op` 2 |
| `decisions-guard` | git pre-commit | 2 | `pass` 2 |
| `prepush-clean-worktree-check` | git pre-push | 1 | `pass` 1 |
| `prepush-donewhen` | git pre-push | 1 | `pass` 1 |
| `prepush-rebase-check` | git pre-push | 1 | `pass` 1 |
| `pretooluse-nav-guard` | PreToolUse/navigate | **0** | not exercised — needs a live browser session |
| `pretooluse-merge-consent` | PreToolUse/PR write | **0** | not exercised — needs a real merge attempt |
**11 of 13 hooks are confirmed firing, with the decision each reached.** The `deny` row is the load-
bearing one: `pretooluse-bash-guard` did not merely run, it *blocked* the `ETV_UPDATE_GOLDENS=1`
probe, so at least one guard in this set is demonstrably live rather than merely present.
**A zero is two findings wearing one number** — a hook that is broken, and a hook whose trigger did
not occur — and the report says so rather than presenting a zero as a verdict. The two zeroes here
are the second kind: neither can be exercised without a browser session or a real merge, and
manufacturing a merge to observe the merge guard is a worse idea than the gap it would close.
Distinguishing the two cases stays a human judgement; what changed is that the question is now
askable at all.
**The measurement earned its keep on its first run.** `pretooluse-agent-model` was recorded deciding
`no-op` on an `Agent` dispatch where `ask` was expected. Previously that would have been an
unanswerable suspicion about a guard nobody could observe; instead the real payload was dumped and
it carried `model: haiku`, so `no-op` was correct and there was no defect. The general form is the
standing #756 lesson — **make the system report it rather than infer it** — and note what it
displaced: an argument from the hook's source about what it "must" do, which is the reasoning shape
§3 measures going wrong repeatedly.
### 5.5 Tools we do *not* have that would address a named class
@@ -533,7 +575,7 @@ This issue is analysis; it spawns implementation rather than doing it.
|---|---|---|---|
| #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, 27 records (39%) | high |
| #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B, 9 records (13%) | high |
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high |
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high — **DONE**, §5.4 is measured |
| #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium |
| #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium |
| #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |