Files
ersatztv/docs/handoffs/chicorytv-issue-queue.md
T
timothyandClaude Opus 4.8 708e602197
Build ErsatzTV Image / CI image pin matches docker/ci (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 9s
Build ErsatzTV Image / Docs update reminder (pull_request) Successful in 10s
Build ErsatzTV Image / decisions.md append-only (pull_request) Successful in 16s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m23s
Build ErsatzTV Image / Functional E2E (curl contracts) (pull_request) Successful in 14m52s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 18m23s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 18m48s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
feat(queue): deterministic scripts/select-queue.sh — stop re-deriving the selector by hand
The kickoff dispatches a cheap model to rank the backlog, and the lore then made the
orchestrator re-derive its mechanical claims (deps, milestone tiering, priority order,
in-progress state) because a small model kept getting them wrong. This pays that tax down:
the mechanical checks are now CODE — GET /dependencies exclusion, LOCAL
.milestone.state/review/priority tiering (never the no-op ?milestones= filter),
(tier,priority,issue#) ordering, in-progress/parked/PR exclusion — leaving only two
JUDGMENT flags (CLAIM?, UMBRELLA?) for a human/model to resolve.

- scripts/select-queue.sh: fail-open on no-creds/unreachable; ~1.3s; shellcheck clean;
  UMBRELLA? computed in the jq pass (no per-candidate body fetch).
- handoff kickoff: run the script FIRST; trust its deps/tiering/ordering, recheck only flags.
- handoff lore: the three 'cheap selector unreliable → re-derive by hand' bullets kept as the
  EVIDENCE for why the script exists; the prescription is redirected to 'run the script'.

Operator-requested this session: 'rather than have the lore make us redo the selector's work,
improve the selector.'

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-19 01:09:57 +02:00

590 lines
51 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# ChicoryTV issue-queue handoff (client-neutral kickoff + workflow lore)
> **PROTOCOL CHANGED 2026-07-11** (decisions.md entry of same date). Queue state lives in the
> **pinned Gitea tracker [ersatztv#237](http://192.168.1.95:3000/timothy/ersatztv/issues/237)**,
> not in this file. Do **NOT** write session state, queue order, or next-session prompts here —
> sessions ending under the old protocol should append their session comment to #237 instead.
> This file holds only stable operating guidance: the standing kickoff prompt and the workflow lore.
> Historical per-session state: `git log` of this file (last state-bearing revision: 8b77d5e7).
---
## Current phase — maintenance / backlog mode (terminal transition, 2026-07-13)
The ChicoryTV rewrite + go-live arc is **COMPLETE**: the pinned arc tracker
[ersatztv#237](http://192.168.1.95:3000/timothy/ersatztv/issues/237) is **CLOSED** and all six
numbered arc items shipped. The project is now in **maintenance / backlog mode**. This is a
*terminal* fact (the arc will not re-open), so it belongs in this standing file — unlike any
"current frontier issue", which must never be written here (read it live; see the queue-drift
lesson at the end of this file).
So a fresh session stops re-discovering this every time:
- The **arc / gate / open-`review` tiers are normally EMPTY now.** An empty arc frontier is the
*expected steady state* — not an anomaly to investigate, not a reason to stop, and not a reason
to ask the user what to work on. Fall straight through to the `priority:`-label cascade and
claim the top-ranked eligible backlog issue.
- **Selection is priority-label-driven**: OPEN-milestoned → `priority: high``priority: medium`
`priority: low`; within a tier, lowest issue number. **Exclude** `in-progress`, pull requests,
and **any issue with an open `blocked-by` dependency** — Gitea tracks these (`GET
/issues/{n}/dependencies`) and auto-clears the block when the blocker closes, so a blocked issue
becomes eligible automatically once its blocker lands. Do not pick an issue ahead of its open
blocker (e.g. #70 waits on #380).
- **`parked` → closed, not parked** (convention 2026-07-17): `parked` means "no concrete plans," and
the decision was to **close** such issues rather than let them linger (reopen instantly if revived).
So `parked` should normally not appear; if it does, treat it as excluded and surface it to the user.
- **Bundles (single-session sweeps)**: related issues are grouped two ways — (a) an **active-arc
milestone** (e.g. "Scheduling: refactor + distribution", "Auto-Tune DetailPanel", "Design system
(Claude Design ↔ prod)"), or (b) a **cross-reference comment** on backlog clusters (e.g. "🔗 Session
bundle — security hardening: #293, #376"). When you claim an issue, check its **milestone** AND its
**cross-references / backlinks** for co-workable siblings, and sweep the whole disjoint cluster in one
session. Active-arc milestones auto-sort above the flat priority backlog; leave individually-important
backlog bugs unmilestoned so their `priority:` label still orders them.
- A **new gate re-appears only** when a future adversarial-review Blocker/High finding is slotted
as a priority pickup, or a new OPEN milestone re-establishes an arc. Until then, backlog
features (ordered by milestone/`priority:` label) are the normal, expected work.
---
## Orchestrator launch profile (operator-facing; do not paste)
As of 2026-07-14:
- **Codex:** use **GPT-5.6 Sol, `ultra`** for the orchestrator. `ultra` is the Codex
multi-agent orchestration setting; if it is unavailable, use **GPT-5.6 Sol, `max` effort**
(`xhigh` only when you deliberately want to trade some capability for quota/latency).
- **Claude Code:** use **Opus 4.8, `xhigh` effort** (or **ultracode**, which is the Claude Code
orchestration preset built on `xhigh`). Reserve **Fable 5, `high` effort** for the frontier
escalations listed below; use `xhigh` there only when the decision remains genuinely difficult.
- **Queue preflight:** when the user has not named an issue, use the client's cheapest suitable
fast/small model at **low** effort. Queue selection is tool-bearing, so do not use `minimal`:
Codex validates the full enabled tool profile, and enabled `web_search` can reject `minimal` even
when the selector intends to call only Gitea MCP. Route it as a subagent when possible; otherwise
run a separate low-cost selector session and pass its compact selection packet to the orchestrator.
Never spend the orchestrator tier on queue ranking.
- If those names are unavailable, choose the client's strongest long-horizon coding/agentic model
at its high extended-reasoning setting. Do not weaken the orchestrator to a fast/mini model; use
smaller models for bounded delegated slices, with a separate low-cost queue preflight when the
client cannot select a cheaper model per agent.
The kickoff below deliberately describes capability tiers instead of model names so it survives
model rollovers. The operator-facing recommendations above may be updated without changing the
queue protocol.
---
# STANDING KICKOFF PROMPT (paste into a fresh Codex or Claude Code session unchanged)
You are the ORCHESTRATOR in the main **Codex or Claude Code** session. First identify your client,
active model, effort/reasoning level, available delegation mechanisms, and concurrency limit.
Orchestration means: decompose, delegate independent slices, integrate their results, verify the
whole, and keep canonical issue state accurate. Use the client's native agent/subagent tools; never
assume a named tool, command, plugin, model-routing feature, or fork mechanism exists.
Route by capability when the client supports per-agent model selection: fast/small for bounded recon,
balanced for mechanical changes, and the strongest coding/agentic tier for judgment-heavy work.
Otherwise use the active model for every slice except the mandatory queue preflight below. Independent
review MUST start from a cold, review-only brief; prefer a different model family/client when one is
available, otherwise use a fresh agent with no implementation role.
**Low-cost routing applies throughout the session, not only to queue selection.** Before any batch of
bounded searches, inventories, log triage, URL/status sampling, or report drafting, dispatch the cheapest
suitable fast/small model. **If any tool is enabled or required, start at `low`; this includes enabled
`web_search` even when the prompt names only Gitea MCP. Reserve `minimal` for explicitly tool-free synthesis
over already-supplied bounded evidence.** Never retry a known tool-bearing profile at `minimal`. Strictly
cap its allowed files, queries, log window, result/evidence count, and output; instruct it to stop and return
partial evidence at a cap instead of broadening scope. The orchestrator receives a compact evidence packet
and performs only the minimum focused live recheck. Same-tier agents add parallelism, not cost savings,
while direct tool-only mechanical checks may stay inline. If a tool-bearing launch is rejected at `minimal`,
retry once with the same model and unchanged scope at `low`. If a cheap worker lacks a required tool, give it
already-collected bounded evidence for explicitly tool-free synthesis or report the limitation instead of
silently expanding orchestrator reconnaissance. Global Codex hook enforcement is tracked in
`timothy/server-management#592`; the wider Claude-hook port is tracked in `timothy/server-management#593`.
**When the user has not named an issue, queue selection is mechanical fast/small work, never orchestrator
work.** FIRST run the deterministic selector — `ETV_GITEA_BASICAUTH=user:pass scripts/select-queue.sh [N]`
— which does the parts a cheap model kept getting wrong IN CODE: it excludes `in-progress`/`parked`/PRs,
resolves `GET /issues/{n}/dependencies` on every candidate (dropping any with an OPEN blocker), tiers by
LOCAL `.milestone.state`/`review`/`priority:` filters (never the server `?milestones=` name filter, which
no-ops on `:`/`+`), and orders by (tier, priority, issue#). **Its DEPS/tiering/ordering are deterministic —
trust them, do not re-derive them by hand.** It also raises two JUDGMENT flags it deliberately does not
decide — `CLAIM?` (a claim can precede the label) and `UMBRELLA?` (an epic whose children are the real
pickups); the orchestrator (or a cheap model) resolves ONLY those by reading the flagged issue's
comments/body. The script does not rank the arc tier (arc order is #237 prose, currently CLOSED) — read
#237 if an arc re-opens. If the script is unavailable (no creds / Gitea down), fall back to the cheap-model
path below. Dispatch exactly one selector on the cheapest suitable model at `low` effort. Give it #237's
body and last ~6 comments, then have it query live candidates as a cascade in this exact tier order:
**arc → OPEN issues assigned to OPEN milestones → open `review` → unmilestoned/unreviewed
`priority: high``priority: medium``priority: low`**. Every candidate row represents an issue
(or eligible reviewer audit): milestone records provide tier metadata and are NEVER pickup candidates.
An OPEN milestone with zero eligible OPEN issues contributes zero candidates. Deduplicate candidates,
apply the live eligibility exclusions at each tier (including automatic exclusion of `parked` issues and any issue with an open `blocked-by` dependency),
and stop as soon as five eligible issues have been accumulated (or all tiers are exhausted). It returns
only that ranked top-five shortlist, with one-line rationales and live-state evidence; a small result
packet is the desired behavior, but an empty or undersized higher tier must fall through to the next tier.
The orchestrator receives only that compact packet, then performs a focused live recheck of the winner
before claiming. If the cheap worker lacks a repository-scoped issue-list tool, it must immediately return
the literal result `TOOL_LIMITATION`; it must not rank issue IDs mentioned in tracker prose, turn milestone
records into candidates, or infer that any tier is empty. The orchestrator then performs only the mechanical
tier queries (repository + state + milestone/label), passes at most the bounded raw ISSUE rows back to the
cheap worker, and leaves all filtering/ranking to that worker. Never substitute an owner-wide/global search
or treat that tool limitation as an empty tier.
Do not load implementation docs or issue bodies into the selector. If the client cannot route a
cheaper subagent, run the selector in a separate low-cost session before starting or resuming the orchestrator
and pass in its packet. **Do not fall back to inline sorting or an equally expensive selector.** If no cheaper
route or session is available, pause and request the selector packet rather than consuming orchestrator tokens
on queue ranking. If the user names an issue, skip selection and only verify that issue's live claimability;
this explicit user choice is the sole path by which a `parked` issue may be worked.
FIRST read `AGENTS.md` and `CLAUDE.md` when present, then `docs/README.md`, the convention docs it
indexes, and the Lessons below. Apply both client instruction files; where they differ, follow the
stricter safety/completion requirement unless a higher-priority instruction resolves the conflict.
FRONTIER ESCALATION — these moments go to the strongest available reasoning model. In Claude Code,
prefer Fable; in Codex, prefer the strongest frontier model at extended-high reasoning. If you are
already running that profile, handle them inline. Otherwise delegate with a SELF-CONTAINED brief
(scope, exact question, evidence, file paths, and doc pointers), or ask the user to switch models when
the client cannot route the escalation. Do not block merely because one brand-specific model is absent:
- Architecture/API design decisions (e.g. #202's media-source API shape) and anything
`[PLAN-MODE]`-tagged.
- Parity verdicts for blazor-route-parity.md (per-capability diff sign-off) and #91b
removal go/no-go.
- Reconciling contradictory independent-review findings and the final
pre-merge judgment on gate PRs.
- Stuck after two genuinely different approaches, or evidence contradicts the issue/docs.
Everything else (claiming, worktrees, dispatching implementers, CI monitoring, protocol
bookkeeping, routine merges of green reviewed PRs with user consent) stays at your level.
Then work the queue:
1. Unless the user named an issue, dispatch or obtain the mandatory low-cost selector packet. The
**selector**, not the orchestrator, reads the pinned tracker **ersatztv#237** — body = goal + ordered
arc + session protocol — and its **last ~6 session comments** (newest-first; the full comment
payload is large, so stop at ~6).
**SOURCE OF TRUTH = live Gitea issue state, NEVER the prose.** The arc body carries ORDER + goal
only; a session comment's "Recommended next" is a forward *guess* written before the next session
acted. Both go stale the instant an item closes (especially under parallel sessions narrating each
other's work as "the gate that unblocks X"). So derive the candidate set from LIVE state, not from
any inline marker or a prior comment's "next": the **current gate = the lowest-numbered OPEN arc
item in #237's arc list**; its open children are the gate cluster (query them by the `review`
label). Cross-check every arc / "recommended" item's real open/closed state (issue **and**
milestone) before trusting it — do NOT hardcode which issue is the frontier; read it. Candidate
discovery is a bounded cascade, not a full inventory: query **arc → OPEN issues assigned to OPEN
milestones → open `review` → unmilestoned/unreviewed `priority: high``priority: medium`
`priority: low`**, carrying eligible unique results forward until the shortlist contains five issues
or all tiers are exhausted. Milestone records are tier metadata, NEVER pickup candidates; an OPEN
milestone with zero eligible OPEN issues contributes zero candidates.
This priority-label fall-through is mandatory: an empty arc/milestone/review pool is never evidence
that the queue is empty. Each priority tier queries all OPEN `timothy/ersatztv` issues carrying that
label, not only IDs mentioned in #237 or recent comments, and excludes pull requests. Treat an
umbrella/epic as a container rather than a pickup when #237 names its eligible children. In the
review tier, ALSO list
open `ersatztv`-labeled issues in **timothy/adversarial-reviewer** — unclaimed audits there are
pickup candidates too (read-only, parallel-safe; see the tracker's "Pending adversarial reviews"
section). Reviewer audits are claimed by comment: a claim remains active until a later comment
explicitly releases or abandons it, and a posted audit/review deliverable is completed work even
when its issue stays open for implementer replies. If the Gitea MCP is down, hit the REST API
directly (use credentials supplied by the environment or your global client instructions; never
print them):
`curl -u <user>:<pass> http://192.168.1.95:3000/api/v1/repos/timothy/ersatztv/issues/237`.
**The authoritative pickup + ranking protocol lives in #237's "Session protocol" section — this is a summary; if the two ever disagree, #237 wins.**
2. The selector returns the highest-ranked eligible candidate plus up to four fallbacks. Apply this
strict tier order: **(1)** open arc items in #237 arc order; **(2)** OPEN issues assigned to an OPEN
milestone;
**(3)** OPEN `review`-labeled issues and eligible reviewer-repo audits; **(4)** remaining
unmilestoned/unreviewed `priority: high` issues; **(5)** `priority: medium`; **(6)** `priority: low`.
Within milestone and review tiers, order by `priority: high` > `medium` > `low`; for otherwise equal
candidates preserve #237's explicit eligible order, then use lowest issue number. At every tier exclude
pull requests, CLOSED issues, `in-progress` claims, `parked` issues, issues blocked by a CLOSED milestone,
**any issue with an open `blocked-by` dependency** (`GET /issues/{n}/dependencies`; Gitea auto-clears the
block when the blocker closes — do not pick an issue ahead of its open blocker), containers whose children
are the pickups, and reviewer audits that are already claimed or have a posted deliverable. Note the
2026-07-17 convention: `parked` (no concrete plans) issues are **closed**, not left parked, so `parked`
should rarely appear; if it does, treat it as excluded and surface it to the user.
Query the next tier only while fewer than five eligible unique candidates have been collected; do not
enumerate the remainder after the shortlist is full.
The orchestrator then makes one focused Gitea read to confirm the proposed winner is still OPEN,
unclaimed, and not blocked by a closed milestone; if it changed, check the next supplied fallback.
**An empty arc frontier is not a stopping condition.** If the selector returns any eligible
candidate, claim its top-ranked winner; do not ask the user to choose merely because candidates
belong to different workstreams.
Do not reread the full tracker or comments for selection. If the prose says "recommended next / now
unblocked" but the issue (or its milestone) is already CLOSED, it is done — skip it and fix the
stale line in your session comment. Prose lags live state; live state wins; tier + `priority:` labels
decide order, not the prose. Never invent a fix-size, recency, or perceived-relevance tiebreaker.
(This mirrors #237's Session-protocol ranking — #237 is canonical.)
3. **Claim it**: add the `in-progress` label + a "claiming" comment on the issue(s);
reviewer-repo audits are claimed by comment only. Treat that claim as live until a later comment
explicitly releases or abandons it, and exclude audits with a posted deliverable even while the
issue remains open for implementer replies.
4. Read the issue bodies (they carry the task context/evidence) and work the item under the
HARD CONSTRAINTS below.
5. Finish by following the session-end protocol in #237: run the **H12 qualification audit**
(`ETV_GITEA_BASICAUTH=user:pass scripts/issue-qualification-audit.sh`) and add a `priority:`
label to anything it lists (every issue you filed this session included); then ONE session
comment on the tracker (template in the tracker body, incl. triage verdicts for any new issues),
remove your `in-progress` labels, and complete the per-issue Task Completion Protocol from the
applicable `AGENTS.md` / `CLAUDE.md` instructions (including the `done` workflow when required).
HARD CONSTRAINTS:
- Work in worktrees off `origin/main` — ALWAYS, including for a one-line docs commit. `/Users/timothy/ersatztv`
is a **shared mutable tree, not a `main` mirror**: never commit in it, and never read its HEAD/`git log`
to learn anything about `main` (you'll get another session's branch). Its only sanctioned use is
**read-only**: copying `web/node_modules` out of it. `origin/main` is the source of truth — `git fetch
origin main` and reference it explicitly. See the lore bullet for the two sessions this bit on 2026-07-17.
- PARALLELIZE BY DEFAULT when there are genuinely disjoint slices: use up to the client's concurrency
limit, normally 23 delegated agents plus the orchestrator. Fan out bounded recon early, but do not
invent artificial slices or exceed a runtime-imposed agent cap. At most 34 concurrent
dotnet/npm builds are fine on this Mac (M4, 10 cores, 16 GB); the go/no-go signal is FREE
RAM, not CPU load (`memory_pressure -Q`: <20% free → don't launch more build agents; <10% →
pause/stagger). CPU load spikes during builds are benign. Never 5+ builds (the historic
crash was RAM starvation from an 89-way fan-out). NEVER set ETV_UPDATE_GOLDENS.
- Never two committing agents on ONE worktree — give each parallel slice its own worktree
branched off the feature branch and merge back. Sequence only where a slice genuinely depends
on another's output (backend-first narrow, SPA-wide after worked well for #216).
- **BATCH YOUR PUSHES — you cannot cancel a CI run** (operator-requested, 2026-07-17). Hold review
fixes, doc corrections and format fixes locally, verify, and push **once**; each push spawns a run
that orphans the previous one on a 4-slot runner. Cancellation is impossible from the agent side on
this Gitea (**1.25.4**): `POST /api/v1/.../actions/runs/{id}/cancel`**404**, MCP
`actions_run_write` `cancel_run`**404**, and the web-UI route needs a session+CSRF that does not
script (login 303s, no session cookie). **Only the operator can cancel, in the browser** — so if you
must supersede a live run, SAY SO explicitly instead of leaving it burning. (`dispatch_workflow` is
a different route and still works for re-triggering a **main** run.) Corrects the older "superseded
runs drain on their own" note below: they do finish, but they hold a slot while doing it.
- **Do NOT gate or throttle your push on host health — trust the Gitea build queue.** The runners were
retuned for stability (operator, 2026-07-17); queueing is the queue's job, not yours. Don't SSH to
bumblebee to sample load/RAM before pushing, and don't hand-schedule around other sessions' runs. Batch
your pushes (above) because orphaned runs can't be cancelled — not because the host needs protecting.
- **An infra-shaped red under host pressure is not a code failure — check the host before diagnosing the
diff** (2026-07-17). While bumblebee was saturated, two unrelated jobs died in ways that look like code
bugs: `EF migration integrity` failed in ~11ms inside **`Setup .NET`** with
`remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and main's
`Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after ~210s of
retries. Both re-ran green once the host recovered; the migration job had no model change to test in the
first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file a CI bug
off one sample under pressure — you'll send the next session chasing a phantom (this nearly got filed as
#390 lane-rebalance fallout, which the evidence did not support).
- **Before any push touching `.cs`, check the touched set for a UTF-8 BOM.** The #311 gate is
fix-as-you-touch: a legacy file's inherited BOM becomes *your* problem the moment you touch it, and
`charset=utf-8` means no BOM.
`for f in $(git diff --name-only origin/main...HEAD -- '*.cs'); do head -c3 "$f" | xxd -p | grep -q "^efbbbf" && echo "BOM: $f"; done`
Strip any hit (1-line diff, zero content change). Bit **two sessions on 2026-07-17** — PR #405 (6
files) and #70/PR #402 (19 files, where Python `io.open(..., encoding='utf-8-sig')` wrote BOMs back
while "preserving existing style"). **Verify your detector**: an `od -An -c | grep '357 273 277'`
check reported all-clean while 19 files were dirty — a detector that can only say "ok" is worse than
none. Use the `xxd` form or read bytes directly.
**`dotnet format --include` DOES work here** (corrected 2026-07-17, #70/PR #402 — an earlier note
claiming it silently no-ops was WRONG). The apparent no-op was **the shell**: CI's recipe uses
`mapfile`, which is **bash-only**, and the default shell here is **zsh** → empty array → zero files
→ exit 0. Run it under `bash -c`:
```bash
bash -c 'mapfile -t files < <(git diff --name-only --diff-filter=ACM origin/main...HEAD -- "*.cs")
dotnet format ErsatzTV.sln --no-restore --verify-no-changes --include "${files[@]}"' # exit 0 = gate passes
```
- Merge consent is DERIVED, not asked: the H6/H10 gate (`pretooluse-merge-consent.sh`) auto-grants a
merge when the linked issue's `## Done-when` boxes are all ticked, a fresh positive `Review-verdict`
references the current head, and CI is green — no conversational confirmation on that satisfied path
(ersatztv#314). Confirm in-conversation only when the gate ASKS (state not derivable: no creds, no
linked issue, no Done-when/verdict, Gitea down). Prior pre-approvals still do NOT carry over.
- **Local gate + cold review run BEFORE the push to Gitea CI, not after** (Timothy, 2026-07-17):
run the local build/test gate AND a cold-context adversarial review agent (SCOPED "review only")
over the diff, fold any fixes, and only THEN push / open the PR. Gitea runners should only ever
see already-validated work — CI is a scarce shared slow resource and a run cannot be cancelled on
this Gitea, so spending a run to surface a lint error or a review-caught bug is pure waste + queue
pressure. CI (armed at PR-open, below) *confirms*; it is not the first line of defence.
- Arm a CI monitor on the PR head sha AT PR-OPEN (commit-status endpoint), not at the end.
- Live-E2E via scripts/e2e-local.sh (also before the push for write-path / UI changes); NEVER
exercise download endpoints via browser tabs (curl them).
---
# Lessons / workflow lore (STANDING workflow/orchestration rules only)
> **Scope**: this section is *how we work* (orchestration, CI, review, cross-session hygiene) —
> not engineering decisions. **Every "why the code does X" decision lives in `docs/decisions.md`**
> (+ `api-conventions.md` §7a/b/c for the concurrency/side-effect mechanics, `spa-conventions.md`,
> `e2e-local.md`). Do NOT re-record a code/API/SPA decision here — it will duplicate and drift.
> Pruned 2026-07-12: issue-specific engineering bullets now covered in those docs, and dead
> Blazor-parity process notes, were removed (git history has them).
- READ docs/README.md → the convention docs FIRST; point recon/implementer agents at specific
doc sections. Only recon the task-specific delta.
- **The mechanical selection checks now live in `scripts/select-queue.sh` — run it instead of
re-deriving them by hand (2026-07-19).** The three bullets below are the *evidence* for why the
script exists: a cheap model repeatedly reported blocked issues as `deps:clear`, mis-tiered on the
no-op `?milestones=` filter, and ranked by issue number instead of priority-within-tier — so the
lore grew to "re-derive the whole contested tier every session," which is exactly the tax the
script pays down. The script does the deterministic work in code (dependency exclusion via
`GET /dependencies`, LOCAL `.milestone.state`/`review`/`priority:` tiering, (tier, priority, issue#)
ordering, `in-progress`/`parked`/PR exclusion) and only *flags* `CLAIM?`/`UMBRELLA?` for judgment.
**Trust its DEPS/tiering/ordering; do the focused recheck only on the flagged rows.** Keep the
evidence bullets below (they explain the failure modes the script encodes), but the *prescription*
is now "run the script," not "hand-re-derive." If you extend the tiers/filters, update the script
first, then these bullets.
- **Gitea milestone-name issue filter silently NO-OPS on names with special chars (2026-07-17,
#77 selection)**: `GET /issues?milestones=<name>` returns the WHOLE open-issue list — not a
filtered set — when the milestone title contains `:` or `+` (e.g. `Scheduling: refactor +
distribution`), because the raw name doesn't round-trip through Gitea's filter. A selector that
trusts that response will mis-tier issues (it made #72 look like a milestone-10 member when it's
unmilestoned, and hid #77's true sibling set). **Robust recipe: fetch all open issues once and
filter LOCALLY on each issue's `.milestone.title`** (`?state=open&type=issues&limit=50`, then
`[i for i in issues if (i['milestone'] or {}).get('title')==NAME]`) — do not rely on the
server-side `milestones=` name filter. If a count looks suspiciously like "all open issues," that
filter silently failed; re-derive membership locally. (Dependencies API is fine: `POST
/issues/{n}/dependencies` with `{"owner","repo","index"}` sets blocked-by; the bare `{index}`
form 201s but may not attach — verify with the GET.)
- **Confirmed again + widened (2026-07-17, #72/#164 selection): treat EVERY mechanical claim in the
selector packet as unverified — not just deps and priority.** That run's dependency data was
correct, yet it still ranked **#383 first while missing that #384 was `in-progress`** (claimed 40
minutes earlier) **and that #383 is that bundle's umbrella** (its body: "Follow-up arc to #69" with
#384#386 as deferred sub-work). Claiming its #1 pick would have collided head-on with a live
parallel session. The packet does not reliably see: `blocked-by`, priority-within-tier,
**`in-progress` claim state**, or umbrella-vs-child. Before claiming, check the candidate's **last
comments** for a live claim (a claim stays live until explicitly released) and read its **body** for
umbrella framing. An empty-ish tier is normal in maintenance mode — three of the four open
milestone issues that day were legitimately blocked or claimed, which is the expected steady state,
not a reason to relax the checks.
- **The cheap selector's dependency + priority data is NOT trustworthy — re-derive the whole contested tier,
not just the winner (2026-07-17, #73 selection)**: the selector packet reported **#385/#386 as `deps:clear`
when both were blocked** (#385 by open #70; #386 by open #385+#384) — it never called
`GET /issues/{n}/dependencies` — and **missed #383 entirely** (the only `priority: medium` in tier 1),
ranking by issue number and ignoring the priority-within-tier rule. Either error alone produces a bad
pickup: working a blocked issue, or skipping the top-priority one. The kickoff's "orchestrator makes one
focused recheck of the **winner**" is **insufficient** whenever a tier has >1 candidate — a wrong winner is
invisible to a winner-only recheck. **Recipe: fetch all open issues once, filter locally on
`.milestone.title`/labels (the `?milestones=` filter no-ops on `:`/`+` names — see the bullet above; both
current open milestones have them), then run `GET /issues/{n}/dependencies` yourself on every candidate you'd
plausibly claim.** That's ~3 cheap calls and it is the only thing standing between the packet and a wasted
session. Keep delegating selection (it's still not orchestrator work) — just verify its two mechanical
outputs (deps, priority order) rather than its ranking prose.
- **Codex cheap-worker launch (tested 2026-07-14)**: the native `spawn_agent` interface currently
has no model/effort selector, so it provides parallelism but not cost savings. For bounded,
tool-bearing selector/recon work, launch a separate worker with
`codex exec --ephemeral --color never -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only
-C <repo> "<hard-capped scope contract>"`. GPT-5.4-Mini is the current supported small/cost-efficient
profile on this ChatGPT-authenticated host; the older `gpt-5.1-codex-mini` guess fails with HTTP 400.
Keep `low` whenever any shell/MCP/web tool is exposed; reserve `minimal` for explicitly tool-free
synthesis over already-supplied bounded evidence. If the model rolls over, inspect the visible entries
in `~/.codex/models_cache.json` instead of guessing names, then update this dated note once verified.
- Keep the MAIN checkout's `web/node_modules` fresh (`npm install` after pulling a PR that
adds a dep) — worktrees copy it, and a stale copy broke typecheck in a #198 worktree.
- **Playwright-MCP E2E: never open tabs/window.open for file-download endpoints** — curl them.
If browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session.
- **THERE IS NO "main checkout". `/Users/timothy/ersatztv` is a shared mutable working tree whose HEAD
is whatever the last session left there — it is NOT a `main` mirror, and its name lies.** Treat it as
**read-only, and never as a source of truth**:
- **Never commit in it** — not even a one-line docs fix. Need a commit on `main`? Make a worktree off
`origin/main` (`git worktree add -b <branch> <path> origin/main`) and commit there.
- **Never read `git log` / `git status` / `HEAD` there and infer anything about `main`** — you will be
reading some other session's branch.
- **`origin/main` is the only source of truth.** `git fetch origin main` and reference `origin/main`
explicitly. Do not trust the local `main` ref either (it lags; nobody is obliged to update it).
- Its **one sanctioned use is read-only**: copying `web/node_modules` into your worktree (keep it
fresh — a stale copy broke typecheck in a #198 worktree).
**This is a design flaw, not a discipline failure — do not "fix" it by telling sessions to check
harder.** A check does not stay true. Worktree discipline is healthy (10 feature worktrees on
2026-07-17: Claude under `.claude/worktrees/`, Codex under `.worktrees/`, some in `/private/tmp/`);
the shared tree is the single place the model breaks, and it broke for **two different sessions on the
same day, both doing the obvious thing**:
- Session A assumed it was on `main` (true at 12:46 when it pulled — **correct when made**, stale by
14:17) and committed → the commit landed on the #604/CI session's `ci/604-lane-rebalance`, **24
seconds after that session's own commit at 14:17:28**. Two writers, one tree. Nothing tangled, by luck.
- Session B ran `git log` there, read `ci/604-lane-rebalance`'s HEAD, and concluded `main` was "4
behind origin" — a phantom.
A session-start `git status` *appears to confirm* the false assumption, which makes it worse than a
plainly wrong claim. (Supersedes the old "the user's main checkout stays ON main and pulled each
session" note.) Same family as the sibling bullet below: never commit inside a checkout or worktree
you didn't create.
- Subagent connection drops = laptop sleep/transient; re-resume via SendMessage (work survives).
- Build/verify (and review) locally, then trust it; CI (VM 127) confirms (~9 min warm; seconds for
a docs-only PR since ersatztv#416). Watch by commit
status: `/api/v1/repos/timothy/ersatztv/commits/{sha}/status`.
- PR routine (works): worktree off origin/main → implement (strongest tier for judgment-heavy /
balanced tier for mechanical work when model routing is available; doc pointers + exact facts;
use the available C# language tooling) → API changes: build app project
FIRST then `./scripts/update-openapi.sh` then `npm run generate:api` → **full local test pass +
cold-context adversarial review (SCOPED "review only") + live-E2E for UI/write-path changes
(`scripts/e2e-local.sh`), ALL BEFORE the push** — fold pre-push review fixes into the branch
(amend/reword is fine while nothing is pushed) → push, PR (arm the CI monitor at open) → CI
*confirms* → any further review fixes (a post-push cross-model pass, or CI-surfaced issues) as
follow-up COMMITS (never amend/force-push a pushed branch) → CI green → merge (derived consent
gate) → structured close comments per the applicable `AGENTS.md` / `CLAUDE.md` protocol.
- Never run TWO committing agents on one worktree concurrently. Read-only review agents are fine.
- **A KILLED job reports `conclusion: failure` — the conclusion field is not enough. Read the log tail.**
(2026-07-17.) When the operator restarts the runners (a retune, a reboot), in-flight jobs die and Gitea
marks them **`failure`**, not `cancelled`. Tell: the log **stops mid-step with no error and no
`❌ Failure - Main …` marker** — a real failure always leaves a marker. Run 1006's `EF migration
integrity` + `Functional E2E` "failed" on a **BOM-removal-only** diff (6 files, 1 line each, zero
content change) that could not possibly break them; the logs simply truncated mid-`dotnet build` at
12:08 **UTC**, and both runners showed `Up About an hour` — the retune had killed them. **Log
timestamps are UTC; the host is UTC+2** — convert before correlating, or the restart looks 2h off and
the theory gets wrongly discarded.
Corollary: **a diff that cannot cause a failure is evidence the failure isn't yours** — when a job that
passed on the previous head fails on a semantically-null delta, suspect the environment and go read the
log, don't re-litigate the diff. Also beware log **warnings that look fatal**: `Unable to pull
refs/heads/v4: …` is act refreshing its `/root/.cache/act` action cache and is followed by `Cloned …`
— it is NOISE, not a cause. Grep for the failure marker, not for the word "error".
- **`cancelled` is NOT `failure` — never read a cancel as a CI verdict** (2026-07-17, operator-surfaced).
The operator cancels runs by hand (they're the only one who can — see the batching constraint), and a
run-level `conclusion: cancelled` means **no verdict**, not a pass and not a fail. Two traps: (1) a
run whose overall state is `failure` may hold a *genuine* job failure that happened **before** the
cancel — check job-level `conclusion` + timestamps, don't attribute it to the cancel; (2) a
`cancelled` run tells you nothing about your diff, so don't claim green on it. Report FAILED and
CANCELLED counts **separately** in any CI monitor:
`fail=[j for j in jobs if j['conclusion']=='failure']; canc=[j for j in jobs if j['conclusion']=='cancelled']`
— a monitor that only asks "is state != pending" will call a cancelled run a failure and send you
debugging a phantom.
- **An infra-shaped red under host load is not a code failure — check `uptime` on bumblebee BEFORE
diagnosing the diff** (2026-07-17). At load average **243**, two unrelated jobs died in ways that
look like code bugs but aren't: `EF migration integrity` failed in ~11ms inside **`Setup .NET`**
with `remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and
main's `Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after
~210s of retries. Both re-ran green at normal load; the migration job had no model change to test in
the first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file
a CI bug off one sample under load — you'll send the next session chasing a phantom (this nearly got
filed as #390 lane-rebalance fallout, which the evidence did not support). A game server
(`FactoryServer-L`) shares that host with the runners, so load isn't always CI's doing.
- **Arm a CI monitor the moment the PR opens** (commit-status endpoint), not at the end — in the
#216 session three PR runs sat red ~an hour on a CI-only flake while review/E2E ran. (The old
MySQL host-port 3306 collision is fixed on main, `ef8915f1`; a lone MySQL-apply red now = the
known infra flake, rerun — see MEMORY / `ci-migration-job-mysql-flake`.)
- **"Make X consistent with Y" review findings deserve new-code scrutiny** — a #216 review fix
propagated MediaBrowseScreen's mount-debounce selection-wipe race to SearchScreen instead of
catching it. For any timer/effect, ask "when does this fire?" (incl. mount).
- Live-E2E seeding recipe now lives in `docs/e2e-local.md` → "Seeding a local TV library for E2E"
(tiny ffmpeg testsrc MKVs + `LibraryPath` SQL rows + scan). Live-E2E is a **stated requirement**
for write-path handler changes — see that doc's "When live-E2E is required" + the decisions.md
entry; it's the only net for the lazy-enumeration / reload-through-read-path 500 class (#229).
- **Parallel sessions (2026-07-11 protocol)**: claim before working (`in-progress` label — the
tiny read→claim race window is accepted; later claimant backs off). Claiming prevents
duplicate pickup, NOT overlapping code changes — check the tracker's dependency notes
("#234 after #231", "coordinate with #215") before touching shared surfaces. Editing THIS
lore: prune covered/stale bullets too (not append-only — git keeps history), `git pull --rebase`
before commit. Two runners (ci-runner VM 127 + bumblebee-runner, 4 slots); **no cancel route exists
on this Gitea version — verified 1.25.4: REST and MCP `cancel_run` both 404** (see the batching
constraint above). Superseded runs do finish on their own, but they hold a runner slot while doing
it, so with 4 parallel sessions they are a real cost — batch pushes rather than relying on them to
drain.
- **Two sessions touching one machine**: a branch may be checked out in ANOTHER session's
worktree — never commit/merge inside a worktree you didn't create. To land a merge on such a
branch without touching their checkout: plumbing merge (`git read-tree -m base ours theirs`
into a temp GIT_INDEX_FILE → `write-tree` → `commit-tree -p ours -p theirs` → push the
commit to the branch ref); the owning session then `git pull`s.
- **Cross-model independent review rotation** (2026-07-11): use the other client/model family when
it is available — Codex reviews Claude-authored changes; Claude reviews Codex-authored changes.
Use whatever native integration is actually installed; client-specific helpers such as
`codex:rescue` are optional accelerators, not protocol dependencies. On quota/tool exhaustion,
use a cold-context review-only agent in the current client and retry cross-model review next window.
Independent reviewers do review and stuck-diagnosis work, not implementation on the same change.
**Skip rubric (self-exemption is an auditable claim, not a vibe)**: an *independent* review pass
(preferably cross-model; otherwise a cold agent) is **mandatory** for any diff touching
locks/concurrency, auth/security, API write-path handlers, or DB migrations, or larger than
~150 changed C# lines. It is **skippable** only for a pure-SPA/docs leaf with no server-state
effect — and when you skip it, **say so and why** in the PR or close comment (a stated exemption,
not silence). This is the correlated-blindspot net: independent model families miss *different* classes
(see the #235 "enumerate every producer" and #250 lessons), so a single reviewer on a risky diff
is the gap.
- **Force enumeration of lazy LanguageExt returns in tests** (2026-07-11, PR #229): a write-path 500
survived a green fixed-point test because the handler returned a lazy `Map` the test never enumerated.
Enumerate handler return values in tests; live E2E remains the only net for this class. (Write-path
reload-through-read-path mechanics: api-conventions §7.)
- **Keep a PR branch current by REBASING on origin/main — do NOT merge main in** (ersatztv#311,
enforced by the H11 pre-push hook: a push from a branch behind origin/main is blocked → `git fetch
origin main && git rebase origin/main`). A *merge* commit pulls in every file main changed, incl.
files you never touched (e.g. the ~2500 legacy-BOM `.cs`), which then trip the format hook/CI on code
that isn't yours (the #309 session). Rebasing keeps your diff to exactly what you changed. After a
rebase that hits conflicts in **generated artifacts** (v1.json, v1.d.ts, endpoint-index), REGENERATE
them (`./scripts/update-openapi.sh` + `npm run generate:api`) — never hand-resolve; git text-merges
them plausibly-but-wrong and `npm run check:api` is the guard. Escape hatch for a deliberate
non-rebased push: `ETV_SKIP_REBASE_CHECK=1 git push`.
- **H12 issue-qualification audit** (ersatztv#312, `scripts/issue-qualification-audit.sh`): a
session-end check that lists OPEN issues missing a `priority:` label — the #237 ranking keys off
`priority:`/gate labels, so an unlabeled issue is invisible to it. "Fully qualified" = has a
`priority: {high,medium,low}` label (that signals triage ran; gate-vs-backlog is then derivable
from the `review` label / milestone, and a milestone is NOT required — backlog is unmilestoned).
Run it at session end and label anything it flags (esp. issues you filed this session). Fail-open
without creds; advisory (exit 1 when any are unqualified). Sibling to H11 (both #311/#312).
- **CI VM test timeouts**: heavy-render web tests (100+ item grids) need explicit vitest
timeouts (e.g. 15s) — the CI VM hit the 5s default on a test that runs in ~1s locally
(run 686). Bump per-test, don't raise the global default.
- **Concurrency tests need a negative control + a stress loop, not a single round** (2026-07-11,
#231): a one-shot `Barrier(N)` + `Task.WhenAll` "exactly one winner" test did NOT catch a
deliberately non-atomic check-then-set flag on this Mac — the load→store window is too narrow to
collide when threads release once. It green-lit broken code. Fix: hammer the race over many rounds
(N worker threads × ~20k rounds, two Barriers per round: acquire → validate one-winner → reset),
and ALWAYS prove non-vacuous by temporarily breaking the primitive and watching the test fail.
Cheap (sub-second) and deterministic-enough. Same discipline as #12's negative control.
- **Before deleting a "workaround", check it isn't serving a SECOND purpose** (2026-07-11, #232):
an issue said "remove the SPA `PENDING_GRACE_TICKS` grace-tick workaround now the API is honest".
The grace ticks did double duty — compensating the lying-200 (now genuinely obsolete) AND
absorbing the inherent queue→observed-active lag / fast-completion race (still needed). Removing
wholesale wedged the scan button disabled-until-reload for short scans; both fork + Codex caught
it. Keep a bounded version, re-scoped honestly. When an issue says "delete X", enumerate every
behavior X provided before deleting.
- **Re-review the FIX COMMIT, not just the initial PR diff, before closing** (2026-07-11, #242,
user-surfaced): the pre-fix review is only half the loop — the fix commit is new code no
independent reviewer has seen. Here the first fix (gate Edit on `saving`) only PARTIALLY closed
the in-flight-save clobber class; a Codex re-review of the fix commit caught that **Delete** was
the one schedule-switch path (`applySwitch`) still missing the guard. Fold a scoped re-review
(cross-model if quota, else a cold review agent) over the fix commit into the close protocol; loop until a clean
MERGEABLE verdict or an explicit acceptable-defer (file a tracked issue for the deferral, as #248
here). Cheap relative to shipping a half-fix that reopens the issue.
- **Negative-controlling a lock/flag fix: invert the condition, don't stub `if (true)`** (2026-07-11,
#250): to prove a lock-release test is non-vacuous I temporarily broke the fix. First attempt made
the guard `if (true)` — but that left the ownership flag "assigned but never read" → **CS0219 under
warnings-as-errors → the Application build silently FAILED → `dotnet test --no-build` ran the STALE
(fixed) dll → false PASS** that looked like the control had failed to catch the bug. The tell: build
output must show `Build succeeded` / `0 Error(s)`, not just a truncated tail. Break the primitive by
**inverting** the real condition (`if (!lockAcquired)`) so the flag stays read and the code compiles;
always grep the build for `error CS` before trusting a `--no-build` test result. (Codex-found High
this session: an outer `catch` that unlocked unconditionally cross-released another session's lock on
a pre-acquisition exception — the fix was a Handle-scoped ownership flag gating the catch.)
- **A lock/channel-ownership "no cross-release" verdict must enumerate EVERY producer of that message
type, not just the enqueuers visible in the PR diff** (2026-07-11, #235/#267). The cold fork cleared the
new `scan-collections` per-provider lock as sound — "only the acquirer enqueues, so only its message
triggers the scanner unlock" — because it checked the three new controllers but never grepped for ALL
`Synchronize{X}Collections(` senders. Codex did: `SchedulerService` also enqueues them *periodically and
unlocked*, and `ScannerService`'s `finally` released the lock whenever held → a scheduled scan
cross-releases an API scan's lock (#250 class). Rule: before trusting any "single owner / no double
release / no cross-release" claim, `grep` the whole host project for every writer of that channel message
(or acquirer of that lock) — the background scheduler/worker is the usual missing producer. This is why the
independent Codex pass earns its keep alongside the fork even when the fork returns MERGEABLE.
- **Independent-review disagreement on a gate PR → frontier judge; judge the fix, not just the verdict**
(2026-07-11, #235): the reconciliation isn't only "who's right" — Fable historically ratified Codex's High AND caught
that the orchestrator's proposed *fix* (per-source lock-or-skip) would deterministically starve the 2nd+
media source (fixed loop order), prescribing the shipped library-scan §C7 pattern instead (lock once per
provider, batch-enqueue, release on the `Unlock: isLast` message). Escalate the FIX approach too, not just
the finding.
- **Harden a soon-to-grow feature with a runtime posture/toggle, not a hardcoded clamp** (2026-07-12, #289):
the MCP's final design (#58) is read+write, so a GET-clamp would be ripped out — shipped a runtime read-only
posture (`ERSATZTV_ALLOW_WRITES`, default false) instead. When a security fix constrains a capability the
roadmap will later want, make the safe state the *default of a switch*, not a wall; check the feature's issue
for its end-state first.
- **Fixing onto an unmerged feature branch: branch off it, land by FF-push, never touch a sibling worktree that
has it checked out** (2026-07-12, #289 → PR#76). Gotcha: recon by `Read`ing a *sibling* worktree leaked its
paths into my later `Edit`s → they landed there, not the fresh worktree (`dotnet` built the unchanged tree →
"0 new tests" tipped it off). After creating a worktree, do the first Edit/Read against ITS absolute paths and
`git status` it before building.
- **`HttpCompletionOption.ResponseHeadersRead` moves the body read OUTSIDE `HttpClient.Timeout`** (2026-07-12,
#289): with a `None` read token a slow-drip upstream hangs forever. Fix = per-request linked
`CancellationTokenSource.CancelAfter(timeout)` into BOTH `SendAsync` and the stream reads +
`HttpClient.Timeout = InfiniteTimeSpan`; catch transport/timeout exceptions and turn them into a response.
(Not yet in decisions.md — #289 landed on PR#76, not main.)
- **Queue-drift root cause + standing rule (2026-07-12, user-surfaced)**: pickups repeatedly re-picked
already-done work — #91b was framed "recommended next / now unblocked" for two sessions *after* it had
merged (2026-07-11), and the #251/#252 priority-pickups sat listed "open" after closing — requiring a
reactive body correction (tracker comment 16:00). ROOT CAUSE: DONE/OPEN status was read from **prose** (the
arc body's inline "DONE" markers + each session comment's "Recommended next"), which is append-only and
hand-edited, so it lags real issue state — worst under parallel sessions that narrate each other's merges as
"the gate that unblocks X" (#271 got cast as the last gate for an already-merged #91b). It is a *structural*
bug, not a stale-writer bug: any status embedded in prose will drift. STANDING FIX (don't just re-patch the
body next time): **live Gitea state is the ONLY source of truth for status.** The arc body carries order+goal;
a comment's "Recommended next" is a *candidate* that MUST be re-verified OPEN (issue **and** milestone) at
pickup. When prose disagrees with live state, live state wins — correct the prose in your session comment,
never propagate it. STRUCTURAL CURE (2026-07-12 review, Fable): #237's arc no longer carries inline `DONE`
markers — closed items move to a "Done (history)" section, so status lives ONLY in live Gitea state and can't
drift; and the gate/frontier is defined **structurally** (lowest-numbered open arc item), never hardcoded to
an issue number. Beware: the *first* pass at this fix re-planted the very bug by hardcoding "#197 cluster" /
"#91b milestone CLOSED" into the kickoff — if you name today's frontier issue in this standing file, you are
writing the next drift. Gate cluster = open `review`-labeled issues serving the current open arc item.