docs(lore): batching, no "main checkout", trust the queue, diagnosing CI reds #417

Merged
timothy merged 1 commits from docs/lore-batching-rules into main 2026-07-17 18:43:34 +02:00
+121 -5
View File
@@ -215,7 +215,11 @@ Then work the queue:
applicable `AGENTS.md` / `CLAUDE.md` instructions (including the `done` workflow when required).
HARD CONSTRAINTS:
- Work in worktrees off origin/main. Copy web/node_modules from the main checkout.
- Work in worktrees off `origin/main` — ALWAYS, including for a one-line docs commit. `/Users/timothy/ersatztv`
is a **shared mutable tree, not a `main` mirror**: never commit in it, and never read its HEAD/`git log`
to learn anything about `main` (you'll get another session's branch). Its only sanctioned use is
**read-only**: copying `web/node_modules` out of it. `origin/main` is the source of truth — `git fetch
origin main` and reference it explicitly. See the lore bullet for the two sessions this bit on 2026-07-17.
- PARALLELIZE BY DEFAULT when there are genuinely disjoint slices: use up to the client's concurrency
limit, normally 2–3 delegated agents plus the orchestrator. Fan out bounded recon early, but do not
invent artificial slices or exceed a runtime-imposed agent cap. At most 3–4 concurrent
@@ -226,6 +230,45 @@ HARD CONSTRAINTS:
- Never two committing agents on ONE worktree — give each parallel slice its own worktree
branched off the feature branch and merge back. Sequence only where a slice genuinely depends
on another's output (backend-first narrow, SPA-wide after worked well for #216).
- **BATCH YOUR PUSHES — you cannot cancel a CI run** (operator-requested, 2026-07-17). Hold review
fixes, doc corrections and format fixes locally, verify, and push **once**; each push spawns a run
that orphans the previous one on a 4-slot runner. Cancellation is impossible from the agent side on
this Gitea (**1.25.4**): `POST /api/v1/.../actions/runs/{id}/cancel` → **404**, MCP
`actions_run_write` `cancel_run` → **404**, and the web-UI route needs a session+CSRF that does not
script (login 303s, no session cookie). **Only the operator can cancel, in the browser** — so if you
must supersede a live run, SAY SO explicitly instead of leaving it burning. (`dispatch_workflow` is
a different route and still works for re-triggering a **main** run.) Corrects the older "superseded
runs drain on their own" note below: they do finish, but they hold a slot while doing it.
- **Do NOT gate or throttle your push on host health — trust the Gitea build queue.** The runners were
retuned for stability (operator, 2026-07-17); queueing is the queue's job, not yours. Don't SSH to
bumblebee to sample load/RAM before pushing, and don't hand-schedule around other sessions' runs. Batch
your pushes (above) because orphaned runs can't be cancelled — not because the host needs protecting.
- **An infra-shaped red under host pressure is not a code failure — check the host before diagnosing the
diff** (2026-07-17). While bumblebee was saturated, two unrelated jobs died in ways that look like code
bugs: `EF migration integrity` failed in ~11ms inside **`Setup .NET`** with
`remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and main's
`Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after ~210s of
retries. Both re-ran green once the host recovered; the migration job had no model change to test in the
first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file a CI bug
off one sample under pressure — you'll send the next session chasing a phantom (this nearly got filed as
#390 lane-rebalance fallout, which the evidence did not support).
- **Before any push touching `.cs`, check the touched set for a UTF-8 BOM.** The #311 gate is
fix-as-you-touch: a legacy file's inherited BOM becomes *your* problem the moment you touch it, and
`charset=utf-8` means no BOM.
`for f in $(git diff --name-only origin/main...HEAD -- '*.cs'); do head -c3 "$f" | xxd -p | grep -q "^efbbbf" && echo "BOM: $f"; done`
Strip any hit (1-line diff, zero content change). Bit **two sessions on 2026-07-17** — PR #405 (6
files) and #70/PR #402 (19 files, where Python `io.open(..., encoding='utf-8-sig')` wrote BOMs back
while "preserving existing style"). **Verify your detector**: an `od -An -c | grep '357 273 277'`
check reported all-clean while 19 files were dirty — a detector that can only say "ok" is worse than
none. Use the `xxd` form or read bytes directly.
**`dotnet format --include` DOES work here** (corrected 2026-07-17, #70/PR #402 — an earlier note
claiming it silently no-ops was WRONG). The apparent no-op was **the shell**: CI's recipe uses
`mapfile`, which is **bash-only**, and the default shell here is **zsh** → empty array → zero files
→ exit 0. Run it under `bash -c`:
```bash
bash -c 'mapfile -t files < <(git diff --name-only --diff-filter=ACM origin/main...HEAD -- "*.cs")
dotnet format ErsatzTV.sln --no-restore --verify-no-changes --include "${files[@]}"' # exit 0 = gate passes
```
- Merge consent is DERIVED, not asked: the H6/H10 gate (`pretooluse-merge-consent.sh`) auto-grants a
merge when the linked issue's `## Done-when` boxes are all ticked, a fresh positive `Review-verdict`
references the current head, and CI is green — no conversational confirmation on that satisfied path
@@ -260,6 +303,17 @@ HARD CONSTRAINTS:
filter silently failed; re-derive membership locally. (Dependencies API is fine: `POST
/issues/{n}/dependencies` with `{"owner","repo","index"}` sets blocked-by; the bare `{index}`
form 201s but may not attach — verify with the GET.)
- **Confirmed again + widened (2026-07-17, #72/#164 selection): treat EVERY mechanical claim in the
selector packet as unverified — not just deps and priority.** That run's dependency data was
correct, yet it still ranked **#383 first while missing that #384 was `in-progress`** (claimed 40
minutes earlier) **and that #383 is that bundle's umbrella** (its body: "Follow-up arc to #69" with
#384–#386 as deferred sub-work). Claiming its #1 pick would have collided head-on with a live
parallel session. The packet does not reliably see: `blocked-by`, priority-within-tier,
**`in-progress` claim state**, or umbrella-vs-child. Before claiming, check the candidate's **last
comments** for a live claim (a claim stays live until explicitly released) and read its **body** for
umbrella framing. An empty-ish tier is normal in maintenance mode — three of the four open
milestone issues that day were legitimately blocked or claimed, which is the expected steady state,
not a reason to relax the checks.
- **The cheap selector's dependency + priority data is NOT trustworthy — re-derive the whole contested tier,
not just the winner (2026-07-17, #73 selection)**: the selector packet reported **#385/#386 as `deps:clear`
when both were blocked** (#385 by open #70; #386 by open #385+#384) — it never called
@@ -286,8 +340,33 @@ HARD CONSTRAINTS:
adds a dep) — worktrees copy it, and a stale copy broke typecheck in a #198 worktree.
- **Playwright-MCP E2E: never open tabs/window.open for file-download endpoints** — curl them.
If browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session.
- The user's main checkout (/Users/timothy/ersatztv) stays ON main and pulled each session.
Feature work in worktrees off origin/main; copy `web/node_modules` from the main checkout.
- **THERE IS NO "main checkout". `/Users/timothy/ersatztv` is a shared mutable working tree whose HEAD
is whatever the last session left there — it is NOT a `main` mirror, and its name lies.** Treat it as
**read-only, and never as a source of truth**:
- **Never commit in it** — not even a one-line docs fix. Need a commit on `main`? Make a worktree off
`origin/main` (`git worktree add -b <branch> <path> origin/main`) and commit there.
- **Never read `git log` / `git status` / `HEAD` there and infer anything about `main`** — you will be
reading some other session's branch.
- **`origin/main` is the only source of truth.** `git fetch origin main` and reference `origin/main`
explicitly. Do not trust the local `main` ref either (it lags; nobody is obliged to update it).
- Its **one sanctioned use is read-only**: copying `web/node_modules` into your worktree (keep it
fresh — a stale copy broke typecheck in a #198 worktree).
**This is a design flaw, not a discipline failure — do not "fix" it by telling sessions to check
harder.** A check does not stay true. Worktree discipline is healthy (10 feature worktrees on
2026-07-17: Claude under `.claude/worktrees/`, Codex under `.worktrees/`, some in `/private/tmp/`);
the shared tree is the single place the model breaks, and it broke for **two different sessions on the
same day, both doing the obvious thing**:
- Session A assumed it was on `main` (true at 12:46 when it pulled — **correct when made**, stale by
14:17) and committed → the commit landed on the #604/CI session's `ci/604-lane-rebalance`, **24
seconds after that session's own commit at 14:17:28**. Two writers, one tree. Nothing tangled, by luck.
- Session B ran `git log` there, read `ci/604-lane-rebalance`'s HEAD, and concluded `main` was "4
behind origin" — a phantom.
A session-start `git status` *appears to confirm* the false assumption, which makes it worse than a
plainly wrong claim. (Supersedes the old "the user's main checkout stays ON main and pulled each
session" note.) Same family as the sibling bullet below: never commit inside a checkout or worktree
you didn't create.
- Subagent connection drops = laptop sleep/transient; re-resume via SendMessage (work survives).
- Build/verify locally, then trust it; CI (VM 127) confirms (~9 min warm). Watch by commit
status: `/api/v1/repos/timothy/ersatztv/commits/{sha}/status`.
@@ -300,6 +379,40 @@ HARD CONSTRAINTS:
green → merge (derived consent gate) → structured close comments per the applicable
`AGENTS.md` / `CLAUDE.md` protocol.
- Never run TWO committing agents on one worktree concurrently. Read-only review agents are fine.
- **A KILLED job reports `conclusion: failure` — the conclusion field is not enough. Read the log tail.**
(2026-07-17.) When the operator restarts the runners (a retune, a reboot), in-flight jobs die and Gitea
marks them **`failure`**, not `cancelled`. Tell: the log **stops mid-step with no error and no
`❌ Failure - Main …` marker** — a real failure always leaves a marker. Run 1006's `EF migration
integrity` + `Functional E2E` "failed" on a **BOM-removal-only** diff (6 files, 1 line each, zero
content change) that could not possibly break them; the logs simply truncated mid-`dotnet build` at
12:08 **UTC**, and both runners showed `Up About an hour` — the retune had killed them. **Log
timestamps are UTC; the host is UTC+2** — convert before correlating, or the restart looks 2h off and
the theory gets wrongly discarded.
Corollary: **a diff that cannot cause a failure is evidence the failure isn't yours** — when a job that
passed on the previous head fails on a semantically-null delta, suspect the environment and go read the
log, don't re-litigate the diff. Also beware log **warnings that look fatal**: `Unable to pull
refs/heads/v4: …` is act refreshing its `/root/.cache/act` action cache and is followed by `Cloned …`
— it is NOISE, not a cause. Grep for the failure marker, not for the word "error".
- **`cancelled` is NOT `failure` — never read a cancel as a CI verdict** (2026-07-17, operator-surfaced).
The operator cancels runs by hand (they're the only one who can — see the batching constraint), and a
run-level `conclusion: cancelled` means **no verdict**, not a pass and not a fail. Two traps: (1) a
run whose overall state is `failure` may hold a *genuine* job failure that happened **before** the
cancel — check job-level `conclusion` + timestamps, don't attribute it to the cancel; (2) a
`cancelled` run tells you nothing about your diff, so don't claim green on it. Report FAILED and
CANCELLED counts **separately** in any CI monitor:
`fail=[j for j in jobs if j['conclusion']=='failure']; canc=[j for j in jobs if j['conclusion']=='cancelled']`
— a monitor that only asks "is state != pending" will call a cancelled run a failure and send you
debugging a phantom.
- **An infra-shaped red under host load is not a code failure — check `uptime` on bumblebee BEFORE
diagnosing the diff** (2026-07-17). At load average **243**, two unrelated jobs died in ways that
look like code bugs but aren't: `EF migration integrity` failed in ~11ms inside **`Setup .NET`**
with `remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and
main's `Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after
~210s of retries. Both re-ran green at normal load; the migration job had no model change to test in
the first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file
a CI bug off one sample under load — you'll send the next session chasing a phantom (this nearly got
filed as #390 lane-rebalance fallout, which the evidence did not support). A game server
(`FactoryServer-L`) shares that host with the runners, so load isn't always CI's doing.
- **Arm a CI monitor the moment the PR opens** (commit-status endpoint), not at the end — in the
#216 session three PR runs sat red ~an hour on a CI-only flake while review/E2E ran. (The old
MySQL host-port 3306 collision is fixed on main, `ef8915f1`; a lone MySQL-apply red now = the
@@ -316,8 +429,11 @@ HARD CONSTRAINTS:
duplicate pickup, NOT overlapping code changes — check the tracker's dependency notes
("#234 after #231", "coordinate with #215") before touching shared surfaces. Editing THIS
lore: prune covered/stale bullets too (not append-only — git keeps history), `git pull --rebase`
before commit. Two runners (ci-runner VM 127 + bumblebee-runner, 4 slots) → superseded runs
drain on their own; no cancel route exists on this Gitea version.
before commit. Two runners (ci-runner VM 127 + bumblebee-runner, 4 slots); **no cancel route exists
on this Gitea version — verified 1.25.4: REST and MCP `cancel_run` both 404** (see the batching
constraint above). Superseded runs do finish on their own, but they hold a runner slot while doing
it, so with 4 parallel sessions they are a real cost — batch pushes rather than relying on them to
drain.
- **Two sessions touching one machine**: a branch may be checked out in ANOTHER session's
worktree — never commit/merge inside a worktree you didn't create. To land a merge on such a
branch without touching their checkout: plumbing merge (`git read-tree -m base ours theirs`