Kickoff: make per-agent model routing a hard constraint + a PreToolUse gate #583

Closed
opened 2026-07-25 11:01:34 +02:00 by timothy · 1 comment
Owner

The kickoff prompt (docs/handoffs/chicorytv-issue-queue.md) tells the orchestrator to route by capability — fast/small for bounded recon, balanced for mechanical changes, strongest tier for judgment-heavy work. On 2026-07-25 a session dispatched two implementer agents (#436, #440) with model omitted on both Agent calls, so both silently inherited the Opus orchestrator tier. #440 was a mechanical SPA slice against an already-shipped backend contract and was a plausible Sonnet candidate.

Root cause

Not forgetfulness — a structural asymmetry:

  1. Not a hard constraint. The routing rule lived only in a prose paragraph. Every bullet in HARD CONSTRAINTS (worktree-off-main, one-committing-agent-per-worktree, parallelize, local-gate-before-push, batch-pushes) was followed in that same session. The one rule in prose was the one that got defaulted.
  2. Scope gap. The low-cost-routing paragraph is written about queue selection and recon ("bounded searches, inventories, log triage, report drafting"). It never names implementers, and states no default tier for the bounded-but-not-trivial case.
  3. No forcing function, and the wrong default is silent. Every other constraint has a hook, a CI job or a script behind it. Omitting model produces no artifact — nothing in the session report revealed it; the operator had to ask.
  4. Distance from the decision point. The rule sits ~line 84; dispatch happens after orientation, claiming, the bundle scan and doc reading.

Change

  • Promote routing to a keyed HARD CONSTRAINTS bullet requiring the tier to be stated in the dispatch message (converts an invisible omission into visible output), and tighten the prose paragraph to point at the key instead of restating it — the kickoff is pasted into every session, so duplication is a per-session tax (#542).
  • Add .claude/hooks/pretooluse-agent-model.sh: PreToolUse on Agent, ask when a dispatch names no model.
  • New decision record process.per-agent-model-routing.

Design revised under review. The first cut gated only implementer-looking dispatches via a prompt-text heuristic. Independent review confirmed it both over- and under-fired — notably "author the change and open a PR" passed silently, missing the exact case it existed to catch — and that its "read-only types cannot commit" rationale was false (Explore/Plan/claude-code-guide all carry Bash). Root cause: the hook contradicted its own rule ("every dispatched agent"). Inverted to ask whenever model is absent, exempting only fork (whose model override the tool ignores). Prompt prose is not a reliable authority signal.

Done-when

  • HARD CONSTRAINTS bullet added + prose paragraph tightened (no duplicated table)
  • Hook added, wired into the existing Agent|Task PreToolUse matcher, and its decision matrix verified against real payloads (including a replay of the 2026-07-25 dispatch that missed)
  • Decision record process.per-agent-model-routing added + catalog regenerated
  • adversarial review passed
The kickoff prompt (`docs/handoffs/chicorytv-issue-queue.md`) tells the orchestrator to route by capability — fast/small for bounded recon, balanced for mechanical changes, strongest tier for judgment-heavy work. On 2026-07-25 a session dispatched two implementer agents (#436, #440) with `model` omitted on both `Agent` calls, so both silently inherited the Opus orchestrator tier. #440 was a mechanical SPA slice against an already-shipped backend contract and was a plausible Sonnet candidate. ## Root cause Not forgetfulness — a structural asymmetry: 1. **Not a hard constraint.** The routing rule lived only in a prose paragraph. Every bullet in `HARD CONSTRAINTS` (worktree-off-main, one-committing-agent-per-worktree, parallelize, local-gate-before-push, batch-pushes) *was* followed in that same session. The one rule in prose was the one that got defaulted. 2. **Scope gap.** The low-cost-routing paragraph is written about *queue selection and recon* ("bounded searches, inventories, log triage, report drafting"). It never names **implementers**, and states no default tier for the bounded-but-not-trivial case. 3. **No forcing function, and the wrong default is silent.** Every other constraint has a hook, a CI job or a script behind it. Omitting `model` produces no artifact — nothing in the session report revealed it; the operator had to ask. 4. **Distance from the decision point.** The rule sits ~line 84; dispatch happens after orientation, claiming, the bundle scan and doc reading. ## Change - Promote routing to a keyed `HARD CONSTRAINTS` bullet requiring the tier to be **stated in the dispatch message** (converts an invisible omission into visible output), and tighten the prose paragraph to point at the key instead of restating it — the kickoff is pasted into every session, so duplication is a per-session tax (#542). - Add `.claude/hooks/pretooluse-agent-model.sh`: PreToolUse on `Agent`, `ask` when a dispatch names no `model`. - New decision record `process.per-agent-model-routing`. **Design revised under review.** The first cut gated only *implementer-looking* dispatches via a prompt-text heuristic. Independent review confirmed it both over- and under-fired — notably "author the change and open a PR" passed silently, missing the exact case it existed to catch — and that its "read-only types cannot commit" rationale was false (`Explore`/`Plan`/`claude-code-guide` all carry Bash). Root cause: the hook contradicted its own rule ("every dispatched agent"). Inverted to ask whenever `model` is absent, exempting only `fork` (whose `model` override the tool ignores). Prompt prose is not a reliable authority signal. ## Done-when - [x] `HARD CONSTRAINTS` bullet added + prose paragraph tightened (no duplicated table) - [x] Hook added, wired into the existing `Agent|Task` PreToolUse matcher, and its decision matrix verified against real payloads (including a replay of the 2026-07-25 dispatch that missed) - [x] Decision record `process.per-agent-model-routing` added + catalog regenerated - [x] adversarial review passed
timothy added the ci-cdenhancementin-progresspriority: medium labels 2026-07-25 11:01:44 +02:00
Author
Owner

Closing record

Outcome: Shipped via PR #585 (merged, head 7e5d20be). Per-agent model routing is now a keyed HARD CONSTRAINTS bullet requiring the tier to be stated in the dispatch itself, backed by .claude/hooks/pretooluse-agent-model.sh — a PreToolUse gate on Agent that asks whenever model is absent (exempting only fork, whose model override the tool ignores). The prose paragraph was tightened to point at the key rather than restate the table.

Root cause: Not forgetfulness — a structural asymmetry. Every bullet in HARD CONSTRAINTS was followed in the same session that missed this (worktree off origin/main, one committing agent per worktree, parallelize, local gate before push, batch pushes). Routing was the one rule living only in a prose paragraph. Three aggravating factors: the low-cost-routing paragraph is written entirely about queue selection and recon and never names implementers, so the largest-cost dispatch fell in a scope gap; omitting model is the silent path, producing no artifact (nothing in the session report revealed it — the operator had to ask); and the rule sits ~line 84 while dispatch happens much later, after orientation/claiming/bundle-scan.

Decisions/conventions changed: Added process.per-agent-model-routing (2026-07-25) in docs/decisions/workflow-process.md. No supersession.

Reusable knowledge:

  1. In a doc pasted into every session, the bulleted imperative list is what actually functions as the checklist; prose above it reads as background. A rule you want followed belongs in the list, keyed, or it is advisory in practice. This is the generalizable lesson, independent of routing.
  2. Prompt prose is not a reliable signal for an agent's authority. The first cut gated only dispatches whose text matched implementer signals (git commit, worktree, fixes #). Review confirmed it both over-fired (a read-only recon brief mentioning "worktree") and under-fired — "author the change and open a PR", "land this on the branch" and "make the changes and commit them" all passed silently, missing exactly the case it existed to catch. A gate with an unreliable catch rate is worse than none, because it gets trusted.
  3. A narrow gate can contradict the rule it enforces. The HARD CONSTRAINT says "every dispatched agent"; the hook gated only implementer-looking ones. Inverting dissolved all three review findings at once rather than patching a regex.
  4. Explore, Plan and claude-code-guide all carry Bash — the "read-only agents cannot commit" premise behind the original exemption list was simply false.
  5. Answer a noise objection with an escape hatch, not with scoping. Naming a tier costs one parameter and the hook never fires, so the prompt is self-eliminating for anyone following the rule. And routing matters most for cheap dispatches — bounded recon is exactly what should be explicitly routed down, which the exemption list had backwards.

Verification: Decision matrix exercised against 11 crafted payloads including a verbatim replay of the 2026-07-25 dispatch that missed, all three review findings, the invariants (model named → never fires; fork exempt; non-Agent passes) and robustness (malformed JSON, empty payload, missing tool_input, embedded backticks/quotes/newlines, 2MB prompt, $(...) injection probe on the label interpolation). Never crashes, never blocks, never emits malformed JSON (jq -n --arg throughout). Two independent review rounds (initial + re-review of the fix commit); decisions-validate: OK; CI green on 7e5d20be.

Deferred: One accepted non-blocking nit — the broad design fires on every model-less dispatch, so an operator could habituate to reflex-approving without reading the reason, eroding the gate without formally disabling it (harder to detect than a kill-switch). The record does not document a fallback posture. Accepted deliberately; if noise materializes in practice, a fallback (e.g. a cost threshold below which dispatches free-pass) gets its own issue. Watch post-merge.

Docs updated: docs/handoffs/chicorytv-issue-queue.md (HARD CONSTRAINTS bullet + tightened prose), docs/decisions/workflow-process.md (new record), docs/decisions/README.md (regenerated), .claude/settings.json + .claude/hooks/pretooluse-agent-model.sh (new hook).

## Closing record **Outcome:** Shipped via PR #585 (merged, head `7e5d20be`). Per-agent model routing is now a keyed `HARD CONSTRAINTS` bullet requiring the tier to be **stated in the dispatch itself**, backed by `.claude/hooks/pretooluse-agent-model.sh` — a PreToolUse gate on `Agent` that `ask`s whenever `model` is absent (exempting only `fork`, whose `model` override the tool ignores). The prose paragraph was tightened to point at the key rather than restate the table. **Root cause:** Not forgetfulness — a structural asymmetry. **Every bullet in `HARD CONSTRAINTS` was followed in the same session that missed this** (worktree off `origin/main`, one committing agent per worktree, parallelize, local gate before push, batch pushes). Routing was the one rule living only in a prose paragraph. Three aggravating factors: the low-cost-routing paragraph is written entirely about queue selection and recon and never names *implementers*, so the largest-cost dispatch fell in a scope gap; omitting `model` is the **silent** path, producing no artifact (nothing in the session report revealed it — the operator had to ask); and the rule sits ~line 84 while dispatch happens much later, after orientation/claiming/bundle-scan. **Decisions/conventions changed:** Added `process.per-agent-model-routing` (2026-07-25) in `docs/decisions/workflow-process.md`. No supersession. **Reusable knowledge:** 1. **In a doc pasted into every session, the bulleted imperative list is what actually functions as the checklist; prose above it reads as background.** A rule you want followed belongs in the list, keyed, or it is advisory in practice. This is the generalizable lesson, independent of routing. 2. **Prompt prose is not a reliable signal for an agent's authority.** The first cut gated only dispatches whose text matched implementer signals (`git commit`, `worktree`, `fixes #`). Review confirmed it both over-fired (a read-only recon brief *mentioning* "worktree") and under-fired — `"author the change and open a PR"`, `"land this on the branch"` and `"make the changes and commit them"` all passed **silently**, missing exactly the case it existed to catch. A gate with an unreliable catch rate is worse than none, because it gets trusted. 3. **A narrow gate can contradict the rule it enforces.** The HARD CONSTRAINT says "every dispatched agent"; the hook gated only implementer-looking ones. Inverting dissolved all three review findings at once rather than patching a regex. 4. **`Explore`, `Plan` and `claude-code-guide` all carry Bash** — the "read-only agents cannot commit" premise behind the original exemption list was simply false. 5. **Answer a noise objection with an escape hatch, not with scoping.** Naming a tier costs one parameter and the hook never fires, so the prompt is self-eliminating for anyone following the rule. And routing matters *most* for cheap dispatches — bounded recon is exactly what should be explicitly routed *down*, which the exemption list had backwards. **Verification:** Decision matrix exercised against 11 crafted payloads including a verbatim replay of the 2026-07-25 dispatch that missed, all three review findings, the invariants (model named → never fires; `fork` exempt; non-`Agent` passes) and robustness (malformed JSON, empty payload, missing `tool_input`, embedded backticks/quotes/newlines, 2MB prompt, `$(...)` injection probe on the `label` interpolation). Never crashes, never blocks, never emits malformed JSON (`jq -n --arg` throughout). Two independent review rounds (initial + re-review of the fix commit); `decisions-validate: OK`; CI green on `7e5d20be`. **Deferred:** One accepted non-blocking nit — the broad design fires on every model-less dispatch, so an operator could habituate to reflex-approving without reading the reason, eroding the gate without formally disabling it (harder to detect than a kill-switch). The record does not document a fallback posture. Accepted deliberately; if noise materializes in practice, a fallback (e.g. a cost threshold below which dispatches free-pass) gets its own issue. Watch post-merge. **Docs updated:** `docs/handoffs/chicorytv-issue-queue.md` (HARD CONSTRAINTS bullet + tightened prose), `docs/decisions/workflow-process.md` (new record), `docs/decisions/README.md` (regenerated), `.claude/settings.json` + `.claude/hooks/pretooluse-agent-model.sh` (new hook).
timothy removed the in-progress label 2026-07-25 11:49:25 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#583