chore(583): make per-agent model routing a hard constraint + PreToolUse gate #585

Merged
timothy merged 2 commits from chore/agent-model-routing into main 2026-07-25 11:47:44 +02:00
Owner

fixes #583

The kickoff tells the orchestrator to route agents by capability, but that rule lived only in a prose paragraph. On 2026-07-25 a session dispatched two implementers (#436, #440) with model omitted on both Agent calls — both silently inherited the Opus orchestrator tier, and nothing in the session report revealed it.

Root cause

Not forgetfulness. Every bullet in HARD CONSTRAINTS was followed in that same session — worktree off origin/main, one committing agent per worktree, parallelize on disjoint slices, local gate before push, batch pushes. Routing was the one instruction living only in prose, and it was the one that got defaulted. In a doc pasted into every session, the bulleted imperative list is what actually functions as the checklist; prose above it reads as background.

Three aggravating factors: the low-cost-routing paragraph is written entirely about queue selection and recon and never names implementers (so the largest-cost dispatch fell in a scope gap); omitting model is the silent path, producing no artifact; and the rule sits ~line 84 while dispatch happens much later.

Change

  • HARD CONSTRAINTS gains a keyed routing bullet requiring the tier to be stated in the dispatch itself — converting an invisible omission into visible output. The prose paragraph is tightened to point at the key rather than restate the table (#542: the kickoff is pasted into every session, so duplication is a per-session tax).
  • .claude/hooks/pretooluse-agent-model.sh — PreToolUse on Agent, wired into the existing Agent|Task matcher alongside the RAM gate. asks whenever model is absent.
  • New decision record process.per-agent-model-routing; catalog regenerated.

The design changed under review — read this part

The first cut (d221a065) was narrow: it fired only when the prompt text matched implementer signals (git commit, worktree, fixes #), reasoning that a gate firing on every fan-out trains one-shot dismissal.

Independent review confirmed that heuristic both over- and under-fired:

  1. A read-only recon brief merely mentioning "worktree" nagged.
  2. "author the change and open a PR", "land this on the branch" and "make the changes and commit them" all passed silently — missing exactly the case the gate exists to catch.
  3. The exemption list justified itself as "these types cannot commit", but Explore, Plan and claude-code-guide all carry Bash. The stated rationale was simply false.

All three share one root cause: the hook contradicted its own rule. The HARD CONSTRAINT says "every dispatched agent"; the hook gated only implementer-looking dispatches. So the fix (7e5d20be) inverts the design rather than patching the regex — ask whenever model is absent, exempting only fork (whose model override the tool ignores, so a prompt could not be acted on). That dissolves all three findings instead of narrowing them: no phrasing dependence, no false-positive concept, no false "cannot commit" claim.

Prompt prose is not a reliable signal for authority, and a gate with an unreliable catch rate is worse than none, because it gets trusted.

The noise objection is answered by the escape hatch, not by scoping: naming a tier costs one parameter and the hook never fires. Self-eliminating for anyone following the rule. And routing matters most for the cheap cases — bounded recon is precisely what should be explicitly routed down to a fast tier, which the old exemption list had backwards.

Verification (fix head 7e5d20be)

All three review findings re-tested verbatim, plus invariants and robustness — 11 payloads:

Case Decision
F1 recon mentioning "worktree" ask (now by design)
F2 "author the change and open a PR" / "…and commit them" ask (was silent)
F3 Explore (carries Bash) ask (was exempt on a false premise)
model named, any subagent_type pass — never fires
fork pass (tool ignores its model)
non-Agent tool / malformed JSON / empty payload pass (fail-open)
embedded backticks, quotes, newlines in prompt ask, valid JSON

Never crashes, never blocks, never emits malformed JSON (jq -n --arg throughout — no manual interpolation). decisions-validate: OK. settings.json valid, 2 hooks on the Agent|Task matcher.

ask, never deny: routing is a judgment call with no derivable right answer, unlike the H6/H10 merge gate which derives verifiable state.

fixes #583 The kickoff tells the orchestrator to route agents by capability, but that rule lived only in a prose paragraph. On 2026-07-25 a session dispatched two implementers (#436, #440) with `model` omitted on both `Agent` calls — both silently inherited the Opus orchestrator tier, and nothing in the session report revealed it. ## Root cause Not forgetfulness. **Every bullet in `HARD CONSTRAINTS` was followed in that same session** — worktree off `origin/main`, one committing agent per worktree, parallelize on disjoint slices, local gate before push, batch pushes. Routing was the one instruction living only in prose, and it was the one that got defaulted. In a doc pasted into every session, the bulleted imperative list is what actually functions as the checklist; prose above it reads as background. Three aggravating factors: the low-cost-routing paragraph is written entirely about *queue selection and recon* and never names implementers (so the largest-cost dispatch fell in a scope gap); omitting `model` is the **silent** path, producing no artifact; and the rule sits ~line 84 while dispatch happens much later. ## Change - **`HARD CONSTRAINTS`** gains a keyed routing bullet requiring the tier to be **stated in the dispatch itself** — converting an invisible omission into visible output. The prose paragraph is tightened to point at the key rather than restate the table (#542: the kickoff is pasted into every session, so duplication is a per-session tax). - **`.claude/hooks/pretooluse-agent-model.sh`** — PreToolUse on `Agent`, wired into the existing `Agent|Task` matcher alongside the RAM gate. `ask`s whenever `model` is absent. - New decision record `process.per-agent-model-routing`; catalog regenerated. ## The design changed under review — read this part The **first cut (`d221a065`) was narrow**: it fired only when the prompt *text* matched implementer signals (`git commit`, `worktree`, `fixes #`), reasoning that a gate firing on every fan-out trains one-shot dismissal. Independent review confirmed that heuristic both over- and under-fired: 1. A read-only recon brief merely *mentioning* "worktree" nagged. 2. `"author the change and open a PR"`, `"land this on the branch"` and `"make the changes and commit them"` all passed **silently** — missing exactly the case the gate exists to catch. 3. The exemption list justified itself as "these types cannot commit", but `Explore`, `Plan` and `claude-code-guide` all carry Bash. The stated rationale was simply false. All three share one root cause: **the hook contradicted its own rule.** The HARD CONSTRAINT says "every dispatched agent"; the hook gated only implementer-*looking* dispatches. So the fix (`7e5d20be`) inverts the design rather than patching the regex — ask whenever `model` is absent, exempting only `fork` (whose `model` override the tool ignores, so a prompt could not be acted on). That dissolves all three findings instead of narrowing them: no phrasing dependence, no false-positive concept, no false "cannot commit" claim. Prompt prose is not a reliable signal for authority, and a gate with an unreliable catch rate is worse than none, because it gets trusted. **The noise objection is answered by the escape hatch, not by scoping:** naming a tier costs one parameter and the hook never fires. Self-eliminating for anyone following the rule. And routing matters *most* for the cheap cases — bounded recon is precisely what should be explicitly routed down to a fast tier, which the old exemption list had backwards. ## Verification (fix head `7e5d20be`) All three review findings re-tested verbatim, plus invariants and robustness — 11 payloads: | Case | Decision | |---|---| | F1 recon mentioning "worktree" | `ask` (now by design) | | F2 "author the change and open a PR" / "…and commit them" | `ask` (was silent) | | F3 `Explore` (carries Bash) | `ask` (was exempt on a false premise) | | `model` named, any subagent_type | pass — never fires | | `fork` | pass (tool ignores its `model`) | | non-`Agent` tool / malformed JSON / empty payload | pass (fail-open) | | embedded backticks, quotes, newlines in prompt | `ask`, valid JSON | Never crashes, never blocks, never emits malformed JSON (`jq -n --arg` throughout — no manual interpolation). `decisions-validate: OK`. `settings.json` valid, 2 hooks on the `Agent|Task` matcher. `ask`, never `deny`: routing is a judgment call with no derivable right answer, unlike the H6/H10 merge gate which derives verifiable state.
timothy added 1 commit 2026-07-25 11:13:10 +02:00
chore(583): make per-agent model routing a hard constraint + PreToolUse gate
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 9s
PR Gates / Docs update reminder (pull_request) Successful in 12s
PR Gates / decisions lifecycle (pull_request) Successful in 14s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 5m42s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 21m53s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 15m2s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
d221a06522
The kickoff tells the orchestrator to route agents by capability, but that rule
lived only in a prose paragraph. On 2026-07-25 a session dispatched two
implementers (#436, #440) with `model` omitted, silently inheriting the
orchestrator tier — while following every bullet in HARD CONSTRAINTS in the same
session. The bulleted list is what functions as the checklist; prose above it
reads as background.

- HARD CONSTRAINTS: keyed routing bullet requiring the tier to be stated in the
  dispatch itself, so an invisible omission becomes visible output. Prose
  paragraph tightened to point at the key rather than restate the table (the
  kickoff is pasted into every session — duplication is a per-session tax, #542).
- .claude/hooks/pretooluse-agent-model.sh: PreToolUse on Agent, `ask` when a
  committing agent is dispatched with no explicit `model`. Narrow by design —
  passes through read-only/recon types, `fork` (model override ignored by the
  tool), and any dispatch already naming a tier, because a gate that fires on
  every fan-out trains one-shot dismissal. `ask` not `deny`: routing is a
  judgment call with no derivable right answer, unlike the H6/H10 merge gate.
- New decision record `process.per-agent-model-routing`; catalog regenerated.

Decision matrix verified against 9 payloads incl. a replay of the dispatch that
missed. decisions-validate: OK.

fixes #583
timothy added 1 commit 2026-07-25 11:19:36 +02:00
fix(583): gate every model-less dispatch, not just implementer-looking ones
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 14s
PR Gates / Docs update reminder (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Successful in 17s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 7m37s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 55s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 17s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m23s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 16m22s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
7e5d20be98
Independent review of d221a065 found the prompt-text heuristic both over- and
under-fired. Confirmed repros: a read-only recon brief merely mentioning
"worktree" nagged, while "author the change and open a PR", "land this on the
branch" and "make the changes and commit them" all passed silently — the gate
missed exactly the case it existed to catch.

Root cause behind all three findings: the hook contradicted its own rule. The
HARD CONSTRAINT says "every dispatched agent"; the hook gated only dispatches
whose prose looked like an implementer. Prompt text is not a reliable signal for
authority.

Inverted: ask whenever `model` is absent, exempting only `fork` (the tool ignores
a fork's model override, so a prompt could not be acted on). This dissolves all
three findings rather than patching the regex — no phrasing dependence, no
false-positive concept, and no "cannot commit" claim to be wrong about. The
reviewer was right that Explore/Plan/claude-code-guide all carry Bash, so the old
exemption rationale was false.

The noise objection is answered by the escape hatch, not by scoping: naming a
tier costs one parameter and the hook never fires. Self-eliminating for anyone
following the rule.

Decision record updated to record the rejected narrow design and why.

Re-verified: 11 payloads incl. all three findings, the invariants (model named ->
never fires; fork exempt; non-Agent passes), and robustness (malformed JSON,
empty payload, missing tool_input, embedded backticks/quotes/newlines) — never
crashes, never emits malformed JSON. decisions-validate: OK.
Author
Owner

Independent re-review of the FIX COMMIT 7e5d20be (not the original diff) — cold review-only brief, no implementation role.

All three findings from the review of d221a065 confirmed resolved:

  • F1 (false positive on "worktree" in a read-only brief) — now asks by design; the hook no longer inspects prompt text at all, only model/subagent_type.
  • F2 (false negatives: "author the change and open a PR", "land this on the branch", "make the changes and commit them" all silently passed) — re-run verbatim, all three now ask. The grep heuristic block is deleted entirely.
  • F3 (false "cannot commit" rationale) — exemption list gone; all seven previously-exempted subagent types now ask when model is omitted.

Regression hunt on the inverted logic found nothing: missing tool_input, absent subagent_type, non-Agent tool, empty stdin, non-JSON stdin, model:""/null (correctly treated as absent), model:123 (stringifies non-empty → treated as an explicit choice), unicode and embedded-newline subagent_type, a 2MB prompt (0.075s — the prompt is never read), and a $(...)/backtick subagent_type probed for command injection through label="${subagent:-general-purpose}" — no injection; bash does not re-evaluate command substitution already captured in a variable. All outputs valid JSON via jq -n --arg.

fork confirmed as the sole correct exemption per the Agent tool's documented semantics ("always runs on your model — a model override is ignored"); no other type has model-ignoring behavior. Escape hatch confirmed real: any non-empty model suppresses the prompt for every subagent type tested. Docs/record coherent with shipped behavior; decisions-validate: OK.

Accepted non-blocking nit (PLAUSIBLE, watch post-merge): the broad design fires on every model-less dispatch, so an operator could habituate to reflex-approving without reading the reason — eroding the gate without formally disabling it, which is harder to detect than a kill-switch. The record does not document a fallback posture if it proves too noisy in practice. Accepted deliberately: the escape hatch is real and cheap, and routing matters most for the cheap dispatches the narrow version exempted. Not fixing in this PR to avoid superseding an in-flight uncancellable CI run for a one-sentence doc change; if noise materializes, the fallback (e.g. a cost/tokens threshold below which dispatches free-pass) gets its own issue.

Review-verdict: MERGEABLE @ 7e5d20be98

Independent re-review of the FIX COMMIT `7e5d20be` (not the original diff) — cold review-only brief, no implementation role. All three findings from the review of `d221a065` confirmed resolved: - F1 (false positive on "worktree" in a read-only brief) — now asks by design; the hook no longer inspects prompt text at all, only `model`/`subagent_type`. - F2 (false negatives: "author the change and open a PR", "land this on the branch", "make the changes and commit them" all silently passed) — re-run verbatim, all three now ask. The grep heuristic block is deleted entirely. - F3 (false "cannot commit" rationale) — exemption list gone; all seven previously-exempted subagent types now ask when `model` is omitted. Regression hunt on the inverted logic found nothing: missing `tool_input`, absent `subagent_type`, non-Agent tool, empty stdin, non-JSON stdin, `model:""`/`null` (correctly treated as absent), `model:123` (stringifies non-empty → treated as an explicit choice), unicode and embedded-newline `subagent_type`, a 2MB prompt (0.075s — the prompt is never read), and a `$(...)`/backtick `subagent_type` probed for command injection through `label="${subagent:-general-purpose}"` — no injection; bash does not re-evaluate command substitution already captured in a variable. All outputs valid JSON via `jq -n --arg`. `fork` confirmed as the sole correct exemption per the Agent tool's documented semantics ("always runs on your model — a `model` override is ignored"); no other type has model-ignoring behavior. Escape hatch confirmed real: any non-empty `model` suppresses the prompt for every subagent type tested. Docs/record coherent with shipped behavior; `decisions-validate: OK`. Accepted non-blocking nit (PLAUSIBLE, watch post-merge): the broad design fires on every model-less dispatch, so an operator could habituate to reflex-approving without reading the reason — eroding the gate without formally disabling it, which is harder to detect than a kill-switch. The record does not document a fallback posture if it proves too noisy in practice. Accepted deliberately: the escape hatch is real and cheap, and routing matters most for the cheap dispatches the narrow version exempted. Not fixing in this PR to avoid superseding an in-flight uncancellable CI run for a one-sentence doc change; if noise materializes, the fallback (e.g. a cost/tokens threshold below which dispatches free-pass) gets its own issue. Review-verdict: MERGEABLE @ 7e5d20be9892f2787eadfa9879032ffe18953e1d
timothy merged commit 8664220aa5 into main 2026-07-25 11:47:44 +02:00
timothy deleted branch chore/agent-model-routing 2026-07-25 11:47:45 +02:00
Sign in to join this conversation.