feat(610): split the decision corpus into one YAML-frontmatter file per record
168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key, so one-active-record-per-key becomes a filesystem property rather than a validator check, and supersession becomes a `git mv`. WHY: the monolith was a concurrency problem before an aesthetic one. A 3,900-line append target made parallel sessions collide -- PR #605 and PR #614 both hit append-vs-append conflicts during routine rebases, and hand-resolving those inside the corpus is exactly the operation the rationale-rewrite guard exists to police. HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness does not rest on reading it. The parser was taught BOTH formats first, so the body-diff guard parses the old form at the merge-base and the new form at head -- the migration validates itself, no bypass. The proof is a field-level equivalence harness: 168 records before and after, zero lost, zero gained, zero field mismatches, zero rationale bodies differing. Reviewers should scrutinise the harness; it is the actual evidence. What measuring caught that reading would not have: - ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each topic file's preamble, mostly the only copy. Source files are kept and stripped, never deleted. They also cannot be filed per-area: topic files hold several areas and 4 of 23 areas span several files. - Archive discovery was a non-recursive glob; after the split it found ZERO archived records, surfacing as four bogus "supersedes points to unknown key" errors rather than an obvious failure. - ~32 live docs point into the corpus BY DATE, which the split dangles. Each stripped file now ends with a generated "Records formerly in this file" index, which also rescues the identical breadcrumbs in old issue comments. - decisions.md's "In this file:" list was 97 same-file anchor bullets that the split makes WRONG, not merely stale. Dropped; the generated index replaces them with links that resolve. The equivalence harness now runs against a checked-in FIXTURE, not the live corpus. The earlier version migrated the real tree, which made it a one-shot: the moment the migration landed there was nothing left to move and the tests failed for reasons unrelated to the code. A fixture keeps them testing the SCRIPT rather than the repo's current state. Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain two directories for one concept. Renaming a key is not a move -- it changes identity, breaks the equivalence proof, and invalidates MemPalace's per-key drawers. Taxonomy normalisation is separate work. refs #610
This commit is contained in:
@@ -14,681 +14,111 @@ Related: `release-ci-governance.md` (merge/release gates and hooks), `startup.pa
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-21 — A killed CI job reports `conclusion: failure`; read the log tail before diagnosing the diff (#542)
|
||||
|
||||
`key: ci.killed-job-triage` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never trust a job's `conclusion` field alone — read the log tail and require an `❌ Failure - Main …` marker before treating a red as a real failure.
|
||||
**Signals:** killed job · runner restart · conclusion failure · log truncated mid-step · `❌ Failure - Main` marker · `Unable to pull refs/heads/v4` · act cache noise · UTC vs UTC+2 · semantically-null diff · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions job logs + runner container uptime on the runner host.
|
||||
|
||||
When the operator restarts the runners (a retune, a reboot), in-flight jobs die and Gitea marks them
|
||||
**`failure`**, not `cancelled`. The tell is that the log **stops mid-step with no error and no
|
||||
`❌ Failure - Main …` marker** — a real failure always leaves that marker. On run 1006, `EF migration
|
||||
integrity` and `Functional E2E` both "failed" on a **BOM-removal-only** diff (6 files, one line each,
|
||||
zero content change) that could not possibly break them; the logs simply truncated mid-`dotnet build`
|
||||
at 12:08 UTC and both runners showed `Up About an hour` — the retune had killed them.
|
||||
|
||||
**Log timestamps are UTC; the host is UTC+2.** Convert before correlating, or the restart looks two
|
||||
hours off and a correct theory gets wrongly discarded.
|
||||
|
||||
Corollary: a diff that *cannot* cause a failure is evidence the failure isn't yours — when a job that
|
||||
passed on the previous head fails on a semantically-null delta, suspect the environment and go read
|
||||
the log rather than re-litigating the diff. Beware warnings that look fatal: `Unable to pull
|
||||
refs/heads/v4: …` is act refreshing its `/root/.cache/act` action cache and is followed by `Cloned …`
|
||||
— it is noise, not a cause. Grep for the failure marker, not for the word "error".
|
||||
|
||||
## 2026-07-21 — `cancelled` is not `failure`; a cancelled run is no verdict (#542)
|
||||
|
||||
`key: ci.cancelled-is-not-a-verdict` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat a `cancelled` conclusion as "no verdict" — never as pass or fail — and report FAILED and CANCELLED counts separately in any CI monitor.
|
||||
**Signals:** conclusion cancelled · run-level vs job-level conclusion · pre-cancel genuine failure · CI monitor state != pending · phantom failure · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions run/job API; monitor logic, e.g. `fail=[j for j in jobs if j['conclusion']=='failure']; canc=[j for j in jobs if j['conclusion']=='cancelled']`.
|
||||
|
||||
The operator cancels runs by hand — they are the only party who can (see
|
||||
`ci.batch-pushes-no-cancel-route`) — and a run-level `conclusion: cancelled` means the run produced
|
||||
**no verdict** about your diff. Two traps follow. First, a run whose *overall* state is `failure` may
|
||||
hold a **genuine job failure that happened before the cancel**: check job-level `conclusion` plus
|
||||
timestamps rather than attributing the red to the cancel. Second, a cancelled run tells you nothing,
|
||||
so never claim green on one.
|
||||
|
||||
A monitor that only asks "is state != pending" will report a cancelled run as a failure and send the
|
||||
next session debugging a phantom. Split the two counts explicitly.
|
||||
|
||||
## 2026-07-21 — An infra-shaped red under host load is not a code failure (#542)
|
||||
|
||||
`key: ci.infra-shaped-red-under-load` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none` · `stale-after: 2027-02-15`
|
||||
**Rule:** When a job dies inside a setup/cache step before your code compiles, check the runner host's load before diagnosing the diff, and never file a CI bug off one sample under pressure.
|
||||
**Signals:** load average 243 · `Setup .NET` 11ms failure · `remove /root/.cache/act/…/.gitignore: no such file or directory` · buildx `DeadlineExceeded: failed to compute cache key` · `Build & push image` · `EF migration integrity` · `FactoryServer-L` game server · phantom CI bug · paths: n/a · issues: #542
|
||||
**Mechanics:** `uptime` on the runner host (bumblebee); Gitea Actions job step logs.
|
||||
**Sources:** 2026-07-17, bumblebee at load average 243 — `EF migration integrity` dead in ~11ms inside `Setup .NET`, main's `Build & push image` dead on buildx `DeadlineExceeded`; both green on re-run at normal load (#542)
|
||||
|
||||
On 2026-07-17, with bumblebee at load average **243**, two unrelated jobs died in ways that look like
|
||||
code bugs but weren't: `EF migration integrity` failed in ~11ms inside **`Setup .NET`** with
|
||||
`remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and main's
|
||||
`Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after ~210s of
|
||||
retries. Both re-ran green at normal load, and the migration job had no model change to test in the
|
||||
first place.
|
||||
|
||||
The diagnostic tell is the **location** of the failure: a setup or cache step, before your code
|
||||
compiles. One sample under load is not evidence of a systemic problem — this nearly got filed as #390
|
||||
lane-rebalance fallout, which the evidence did not support, and would have sent the next session
|
||||
chasing a phantom. Note also that a game server (`FactoryServer-L`) shares that host with the
|
||||
runners, so high load is not always CI's own doing.
|
||||
|
||||
## 2026-07-21 — Batch your pushes: there is no agent-side cancel route on Gitea 1.25.4 (#542)
|
||||
|
||||
`key: ci.batch-pushes-no-cancel-route` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Hold review fixes, doc corrections and format fixes locally and push **once** — a superseded run cannot be cancelled from the agent side and holds a runner slot until it finishes.
|
||||
**Signals:** cancel run 404 · Gitea 1.25.4 · `POST /api/v1/.../actions/runs/{id}/cancel` · MCP `actions_run_write` `cancel_run` · session+CSRF · `dispatch_workflow` · orphaned run · 4-slot runner · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions REST + MCP `actions_run_write`; operator-only cancel in the browser UI.
|
||||
|
||||
Cancellation is impossible from the agent side on this Gitea (**1.25.4**):
|
||||
`POST /api/v1/.../actions/runs/{id}/cancel` returns **404**, MCP `actions_run_write`'s `cancel_run`
|
||||
returns **404**, and the web-UI route needs a session + CSRF that does not script (login 303s with no
|
||||
session cookie). **Only the operator can cancel, in the browser** — so if you must supersede a live
|
||||
run, say so explicitly instead of leaving it burning. (`dispatch_workflow` is a different route and
|
||||
still works for re-triggering a **main** run.)
|
||||
|
||||
This corrects the older "superseded runs drain on their own" framing: they do finish, but they hold
|
||||
one of the 4 runner slots while doing it, which is a real cost with several parallel sessions.
|
||||
|
||||
## 2026-07-21 — Do not gate or throttle pushes on host health (#542)
|
||||
|
||||
`key: ci.no-host-health-gating` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs.
|
||||
**Signals:** trust the build queue · runner retune · don't sample load before pushing · queueing is the queue's job · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea build queue (two runners, 4 slots).
|
||||
|
||||
The runners were retuned for stability (operator, 2026-07-17); queueing is the queue's job, not the
|
||||
agent's. Sampling host load before a push is both unreliable and unnecessary, and hand-scheduling
|
||||
around other sessions produces coordination that no one can verify.
|
||||
|
||||
Batch your pushes (see `ci.batch-pushes-no-cancel-route`) because orphaned runs cannot be cancelled —
|
||||
**not** because the host needs protecting. The two rules have the same action and different reasons;
|
||||
conflating them leads to load-watching behaviour that this record forbids.
|
||||
|
||||
## 2026-07-21 — A lone `decisions lifecycle` red is a known infra flake: do nothing (#542)
|
||||
|
||||
`key: ci.decisions-lifecycle-flake` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When `decisions lifecycle` is the **only** red job, do not investigate and do not create a new run to clear it — no rebase, no `--amend`, no no-op push; the operator reruns that single job from the Gitea UI.
|
||||
**Signals:** decisions lifecycle red · append-only gate · known flake · operator selective rerun · no-op push · convincing local explanation · paths: `docs/decisions.md` · issues: #542, #473, PR #479
|
||||
**Mechanics:** Gitea Actions `decisions lifecycle` job; operator-driven single-job rerun in the web UI.
|
||||
|
||||
Operator-stated, 2026-07-19. Report it as a known flake and carry on; only if *other* jobs are red too
|
||||
does the run deserve diagnosis. Same family as the killed-job rule — a spurious single-job red is
|
||||
cleared by the operator's selective rerun, never by pushing, and pushes cannot be cancelled anyway.
|
||||
|
||||
**The trap is that a convincing local explanation is always available.** On #473/PR #479 the job went
|
||||
red just after `main` landed its own `decisions.md` entry, so "mine is no longer at EOF, I must
|
||||
rebase" looked airtight. The rebase happened — and the job went red **again** on a head whose diff was
|
||||
a verified pure EOF append with zero deleted lines. A rebase that provably satisfied the gate's stated
|
||||
rule did not turn it green, which is the proof that the red was never about the diff. Check this rule
|
||||
*before* theorising about a cause.
|
||||
|
||||
## 2026-07-21 — Arm the CI monitor at PR-open, via the commit-status endpoint (#542)
|
||||
|
||||
`key: ci.monitor-armed-at-pr-open` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work.
|
||||
**Signals:** arm monitor at PR open · commit status endpoint · head sha · red run sat unnoticed · MySQL-apply flake · `skipped` misread as red · `Build & push image (amd64)` skipped · monitor classification · `!= "success"` filter · combined `.state` · paths: n/a · issues: #542, #216, #583
|
||||
**Mechanics:** `GET /api/v1/repos/timothy/ersatztv/commits/{sha}/status` for the combined verdict (`.state`); `…/statuses?limit=50` only when you need per-context detail.
|
||||
|
||||
CI runs concurrently with the review and E2E work that follows a push, so the cost of a late-armed
|
||||
monitor is entirely wasted wall-clock. In the #216 session three PR runs sat red for roughly an hour
|
||||
on a CI-only flake while review and E2E ran to completion — the reds were only discovered afterwards.
|
||||
|
||||
**Classify per-context states correctly, or prefer the combined `.state`.** A monitor that enumerates
|
||||
contexts and treats anything `!= "success"` as red is WRONG on this repo: `Build & push image (amd64)`
|
||||
is `if:`-gated at the JOB level on `github.event_name != 'pull_request'`, so it reports **`skipped` on
|
||||
every PR**, by design and regardless of content (images are built only on push-to-main and tags — see
|
||||
`ci.docs-only-skip-steps`, which records the same fact from the branch-protection angle: "`build` is
|
||||
`skipped` on every PR").
|
||||
Such a monitor cries "NOT all green" on a perfectly green PR. Note this is *not* the docs-only skip:
|
||||
the docs-only mechanism deliberately gates individual STEPS so required jobs still report `success` in
|
||||
seconds — misattributing the image job's skip to docs-only is a plausible-sounding wrong diagnosis
|
||||
(#583 session, 2026-07-25).
|
||||
|
||||
Three distinct non-`success` states, three meanings — do not collapse them:
|
||||
- **`skipped`** — deliberately not applicable. Settled, and *not* red. Gitea's combined `.state` already
|
||||
treats it as non-blocking (a PR with a skipped `build` reports `overall=success`), which is why the
|
||||
combined endpoint is the safer thing to gate on.
|
||||
- **`failure`** — a real red; diagnose it (but first check `ci.killed-job-triage` and
|
||||
`ci.infra-shaped-red-under-load`).
|
||||
- **`cancelled`** — no verdict at all; see `ci.cancelled-is-not-a-verdict`.
|
||||
|
||||
Working filter when you do enumerate — verified silent on a green PR carrying a skipped `build`, and
|
||||
verified to still report a genuinely unfinished run (i.e. proven able to go dirty, per
|
||||
`process.bom-format-detection-recipe`'s "verify your detector" rule):
|
||||
|
||||
```bash
|
||||
curl -s -u "$ETV_GITEA_BASICAUTH" ".../commits/$SHA/statuses?limit=50" \
|
||||
| jq -r '[.[]|{c:.context,st:.status}]|group_by(.c)|map(.[0])|.[]
|
||||
|select(.st!="success" and .st!="skipped")|"NOT-GREEN: \(.c) = \(.st)"'
|
||||
```
|
||||
|
||||
**Mind the renamed key.** The first draft of this snippet said `select(.status != …)` after the
|
||||
pipeline had already renamed `.status` to `.st`, so the comparison ran against `null`, passed
|
||||
*everything*, and reported a fully green PR as nine failures. Report `failure` and `cancelled` in
|
||||
separate counts.
|
||||
|
||||
Related context for interpreting an early red: the old MySQL host-port 3306 collision is fixed on main
|
||||
(`ef8915f1`), so a lone MySQL-apply red now indicates the known infra flake and warrants a rerun
|
||||
rather than diagnosis.
|
||||
|
||||
## 2026-07-21 — Heavy-render web tests need explicit per-test vitest timeouts on the CI VM (#542)
|
||||
|
||||
`key: ci.web-test-per-test-timeouts` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Give heavy-render web tests an explicit per-test vitest timeout (e.g. 15s); never raise the global default to fix one slow test.
|
||||
**Signals:** vitest timeout · 5s default · 100+ item grid · CI VM slower than local · run 686 · paths: `web/` tests · issues: #542
|
||||
**Mechanics:** per-test timeout argument in the vitest test declaration.
|
||||
|
||||
The CI VM is materially slower than a dev Mac for render-heavy work. A web test rendering a 100+ item
|
||||
grid runs in about 1s locally but hit the **5s** vitest default on the CI VM (run 686) and went red
|
||||
for no code reason.
|
||||
|
||||
Bump the timeout on the specific test. Raising the global default hides genuine hangs across the whole
|
||||
suite in exchange for fixing one known-slow case.
|
||||
|
||||
## 2026-07-21 — Build and verify locally, then trust it; CI confirms (#542)
|
||||
|
||||
`key: ci.verify-locally-ci-confirms` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat the local build/verify/review pass as the decision point and CI as confirmation — don't idle waiting on a run you have no reason to doubt.
|
||||
**Signals:** ~9 min warm CI · docs-only PR seconds · CI VM 127 · commit status watch · local gate first · paths: n/a · issues: #542, #416
|
||||
**Mechanics:** `GET /api/v1/repos/timothy/ersatztv/commits/{sha}/status`; CI on VM 127.
|
||||
|
||||
A warm full CI run takes roughly **9 minutes**; a docs-only PR completes in seconds since ersatztv#416's
|
||||
docs-only skip. Neither duration justifies blocking on the run when the same gates already passed
|
||||
locally.
|
||||
|
||||
Watch the run by commit status rather than by polling the UI, and continue working. CI *confirms* the
|
||||
local verdict; it is not the first line of defence.
|
||||
|
||||
## 2026-07-21 — Independent cross-model review is mandatory on risky diffs; a skip is a stated, auditable exemption (#542)
|
||||
|
||||
`key: process.independent-review-rubric` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run an independent review pass — preferably a different model family, otherwise a cold-context review-only agent — on any diff touching locks/concurrency, auth/security, API write-path handlers, or DB migrations, or larger than ~150 changed C# lines; skip only for a pure-SPA/docs leaf with no server-state effect, and state the skip and its reason in the PR or close comment.
|
||||
**Signals:** cross-model review rotation · correlated blindspots · skip rubric · self-exemption · review-only agent · quota exhaustion · reviewer does not implement · paths: n/a · issues: #542, #235, #250
|
||||
**Mechanics:** rotation — Codex reviews Claude-authored changes, Claude reviews Codex-authored changes; use whatever native integration is installed (client helpers like `codex:rescue` are optional accelerators, not protocol dependencies). On quota/tool exhaustion, fall back to a cold-context review-only agent in the current client and retry cross-model next window. Independent reviewers do review and stuck-diagnosis work only — never implementation on the same change they review.
|
||||
|
||||
The justification is correlated blindspots: independent model families miss *different* classes of
|
||||
defect, so a single reviewer on a risky diff is the gap, not a saving. #235 (a reviewer clearing a lock
|
||||
as sound because it never grepped for every producer) and #250 (a vacuous negative control that
|
||||
silently ran a stale dll) are both cases the second, differently-blind pass caught.
|
||||
|
||||
The skip clause is deliberately narrow and deliberately loud. A self-exemption is an auditable claim,
|
||||
not a vibe — silence is indistinguishable from forgetting, so the exemption must be written down where
|
||||
the merge record can be re-read later.
|
||||
|
||||
## 2026-07-21 — Local build/test gate + cold-context review run BEFORE the push, not after (#542)
|
||||
|
||||
`key: process.local-gate-before-push` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run the local build/test gate and a cold-context, scoped "review only" adversarial review over the diff, fold the fixes, and only then push or open the PR.
|
||||
**Signals:** local gate before push · cold review before CI · CI is not the first line of defence · uncancellable run · queue pressure · paths: n/a · issues: #542
|
||||
**Mechanics:** local build/test gate + a scoped review-only agent over `origin/main...HEAD`; CI (armed on the PR head sha at PR-open) *confirms* the result.
|
||||
|
||||
Gitea runners should only ever see already-validated work. CI here is a scarce shared slow resource and
|
||||
a run cannot be cancelled on this Gitea version, so spending a run to surface a lint error or a
|
||||
review-catchable bug is pure waste plus queue pressure on every other in-flight session.
|
||||
|
||||
CI confirms; it is not the first line of defence. Treating it as the first line inverts the cost: the
|
||||
cheap fast private check is skipped in favour of the expensive slow shared one.
|
||||
|
||||
## 2026-07-21 — Review disagreement on a gate PR escalates to the frontier judge — and the proposed FIX escalates with it (#542)
|
||||
|
||||
`key: process.review-disagreement-frontier-judge` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When independent reviews disagree on a gate PR, escalate to the frontier judge, and put the proposed fix approach in front of it — not just the disputed finding.
|
||||
**Signals:** review disagreement · frontier judge · escalate the fix not just the verdict · gate PR reconciliation · paths: n/a · issues: #542, #235
|
||||
**Mechanics:** n/a — escalation is a routing decision by the orchestrator.
|
||||
|
||||
The reconciliation is not only "who is right about the finding". On #235 the frontier model ratified
|
||||
the disputed High *and* caught that the orchestrator's own proposed fix (a per-source lock-or-skip over
|
||||
a fixed loop order) would deterministically starve the 2nd and later media source — prescribing instead
|
||||
the pattern that shipped.
|
||||
|
||||
That is the whole argument: a judge given only the verdict question would have returned "the High is
|
||||
real" and let a starving fix ship. Send the fix approach up with the finding.
|
||||
|
||||
## 2026-07-21 — "Make X consistent with Y" review findings are new code and get new-code scrutiny (#542)
|
||||
|
||||
`key: process.consistency-fix-new-code-scrutiny` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Review a "make X consistent with Y" change as new code, not as a mechanical copy — and for any timer or effect involved, ask explicitly "when does this fire?", including on mount.
|
||||
**Signals:** consistency fix · propagating the pattern propagates the bug · mount-debounce · timer/effect fire timing · paths: n/a · issues: #542, #216
|
||||
**Mechanics:** n/a — a review-attention rule.
|
||||
|
||||
Propagating a pattern propagates its bugs. A #216 review fix copied MediaBrowseScreen's mount-debounce
|
||||
selection-wipe race into SearchScreen instead of catching it — the finding said "be consistent", and
|
||||
consistency was achieved, including with the defect.
|
||||
|
||||
The mount question is the cheap general form of the check: most of this class hides in an effect or
|
||||
timer that fires once more than the author pictured.
|
||||
|
||||
## 2026-07-21 — A lock/channel "no cross-release" verdict must enumerate every producer via grep (#542)
|
||||
|
||||
`key: process.lock-ownership-enumerate-producers` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Before trusting any "single owner / no double release / no cross-release" claim, grep the whole host project for every writer of that channel message (or acquirer of that lock) — the background scheduler/worker is the usual missing producer.
|
||||
**Signals:** cross-release · lock ownership · channel message producers · scheduler enqueues too · grep the host project · paths: n/a · issues: #542, #235, #267, #250
|
||||
**Mechanics:** grep for every sender of the message type across the host project (e.g. all `Synchronize{X}Collections(` call sites), not just the controllers present in the PR diff. General form of "re-derive from a grep rather than enumerating from a list": see `sched.weighted-shuffle`.
|
||||
|
||||
On #235/#267 a cold fork cleared a new `scan-collections` per-provider lock as sound — "only the
|
||||
acquirer enqueues, so only its message triggers the scanner unlock" — having checked the three new
|
||||
controllers in the diff and stopped there. Codex grepped: `SchedulerService` also enqueues those
|
||||
messages periodically and unlocked, and `ScannerService`'s `finally` released the lock whenever held,
|
||||
so a scheduled scan cross-releases an API scan's lock (the #250 class).
|
||||
|
||||
Corollary: the independent pass earns its keep even when the first reviewer returns MERGEABLE. The
|
||||
first verdict was not careless — it was complete over the evidence it looked at, which is exactly the
|
||||
failure mode a second, differently-scoped reviewer catches.
|
||||
|
||||
## 2026-07-21 — Before deleting a "workaround", enumerate every behavior it provided (#542)
|
||||
|
||||
`key: process.enumerate-workaround-behaviors-before-deleting` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When an issue says "delete X", enumerate every behavior X provided before removing it — a workaround often serves a second purpose that outlives the first.
|
||||
**Signals:** remove the workaround · double duty · second purpose · bounded version re-scoped · PENDING_GRACE_TICKS · paths: n/a · issues: #542, #232
|
||||
**Mechanics:** n/a — a scoping rule applied when reading a removal issue.
|
||||
|
||||
#232 said "remove the SPA `PENDING_GRACE_TICKS` grace-tick workaround now the API is honest". The grace
|
||||
ticks did double duty: compensating the lying-200 (genuinely obsolete) *and* absorbing the inherent
|
||||
queue→observed-active lag and fast-completion race (still needed). Removing them wholesale wedged the
|
||||
scan button disabled-until-reload for short scans; both the fork and Codex caught it.
|
||||
|
||||
The landing move is usually not "delete" or "keep" but a bounded version re-scoped to the purpose that
|
||||
still holds.
|
||||
|
||||
## 2026-07-21 — The shared tree at /Users/timothy/ersatztv is read-only and tells you nothing about `main` (#542)
|
||||
|
||||
`key: process.shared-tree-readonly` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never commit in `/Users/timothy/ersatztv` and never read its `git log`/`git status`/HEAD to infer anything about `main` — work in a worktree off `origin/main`, which is the only source of truth.
|
||||
**Signals:** shared checkout · "main checkout" · worktree discipline · stale HEAD · origin/main is truth · node_modules copy · paths: `/Users/timothy/ersatztv`, `.claude/worktrees/`, `.worktrees/` · issues: #542
|
||||
**Mechanics:** `git worktree add -b <branch> <path> origin/main`; `git fetch origin main` and reference `origin/main` explicitly — the local `main` ref lags and nobody is obliged to update it. The one sanctioned use of the shared tree is read-only: copying `web/node_modules` out of it (keep that fresh — a stale copy broke typecheck in a #198 worktree).
|
||||
|
||||
There is no "main checkout". `/Users/timothy/ersatztv` is a shared mutable working tree whose HEAD is
|
||||
whatever the last session left there; its name lies. The rule holds even for a one-line docs fix — make
|
||||
a worktree.
|
||||
|
||||
This is a design flaw, not a discipline failure: do not "fix" it by telling sessions to check harder,
|
||||
because a check does not stay true. It broke for two different sessions on the same day, 2026-07-17,
|
||||
both doing the obvious thing. Session A assumed it was on `main` — correct when it pulled at 12:46,
|
||||
stale by 14:17 — and committed; the commit landed on the CI session's `ci/604-lane-rebalance`, 24
|
||||
seconds after that session's own commit at 14:17:28. Two writers, one tree; nothing tangled, by luck.
|
||||
Session B ran `git log` there, read that same branch's HEAD, and concluded `main` was "4 behind origin"
|
||||
— a phantom. A session-start `git status` *appears to confirm* the false assumption, which makes it
|
||||
worse than a plainly wrong claim.
|
||||
|
||||
Cross-reference `session.shared-checkout-refresh`, which covers the complementary failure mode: any
|
||||
FILE (including a pasted kickoff prompt) read out of that tree may be stale, and the session-end
|
||||
refresh script that mitigates it. This record covers the git-state failure mode only.
|
||||
|
||||
## 2026-07-21 — One worktree, one committing agent (#542)
|
||||
|
||||
`key: process.one-worktree-one-committing-agent` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never run two committing agents concurrently on one worktree — give each parallel slice its own worktree branched off the feature branch and merge back.
|
||||
**Signals:** parallel agents · concurrent commits · worktree per slice · read-only review agent · paths: n/a · issues: #542
|
||||
**Mechanics:** `git worktree add` a fresh tree per slice off the feature branch; merge the slices back into the feature branch when they land.
|
||||
|
||||
Read-only review agents sharing a tree are fine — the constraint is on *writers*. Sequence slices only
|
||||
where one genuinely depends on another's output; "backend-first narrow, SPA-wide after" worked well for
|
||||
#216.
|
||||
|
||||
## 2026-07-21 — Never commit or merge inside a worktree you did not create (#542)
|
||||
|
||||
`key: process.foreign-worktree-plumbing-merge` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never commit or merge inside a worktree another session created; land the merge with git plumbing against the branch ref instead.
|
||||
**Signals:** sibling session worktree · branch already checked out · plumbing merge · read-tree · commit-tree · GIT_INDEX_FILE · paths: n/a · issues: #542
|
||||
**Mechanics:** `git read-tree -m <base> <ours> <theirs>` into a temporary `GIT_INDEX_FILE` → `git write-tree` → `git commit-tree -p <ours> -p <theirs>` → push the resulting commit to the branch ref. The owning session then `git pull`s it.
|
||||
|
||||
With several sessions on one machine, any given branch may already be checked out in a sibling
|
||||
session's worktree. Touching that checkout writes into another agent's working state; the plumbing
|
||||
route mutates only the ref, so the merge lands without disturbing the owner's tree.
|
||||
|
||||
## 2026-07-21 — Fix onto an unmerged feature branch by branching off it and FF-pushing (#542)
|
||||
|
||||
`key: process.branch-off-feature-branch` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** To fix work on an unmerged feature branch, branch off that branch and land by fast-forward push — and after creating a worktree, drive the first Edit/Read from ITS absolute paths and `git status` it before building.
|
||||
**Signals:** unmerged feature branch · FF-push · path leak · sibling worktree recon · wrong tree edits · "0 new tests" · paths: n/a · issues: #542
|
||||
**Mechanics:** `git worktree add -b <fix-branch> <path> <feature-branch>`; land with a fast-forward push to the feature branch. Never touch a sibling worktree that has the branch checked out (see `process.foreign-worktree-plumbing-merge`).
|
||||
|
||||
The path-leak trap, hit on 2026-07-12 (#289 → PR#76): reconnaissance by `Read`ing a *sibling* worktree
|
||||
leaked that tree's absolute paths into subsequent `Edit`s, so the edits landed there rather than in the
|
||||
fresh worktree. `dotnet` then built the unchanged tree, and the tell was a build reporting "0 new
|
||||
tests". A `git status` in the new worktree before the first build catches this in one command.
|
||||
|
||||
## 2026-07-21 — Bound parallel builds by FREE RAM, never regenerate goldens (#542)
|
||||
|
||||
`key: process.build-concurrency-limits` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run at most 3–4 concurrent dotnet/npm builds on this Mac, gate launches on FREE RAM rather than CPU load, and never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS`.
|
||||
**Signals:** build concurrency · memory_pressure · RAM starvation · fan-out cap · golden regeneration · paths: n/a · issues: #542
|
||||
**Mechanics:** `memory_pressure -Q` — under 20% free, do not launch more build agents; under 10%, pause or stagger. Host is an M4, 10 cores, 16 GB.
|
||||
|
||||
Parallelize by default when slices are genuinely disjoint: use up to the client's concurrency limit,
|
||||
normally 2–3 delegated agents plus the orchestrator. Fan out bounded recon early, but do not invent
|
||||
artificial slices or exceed a runtime-imposed agent cap. CPU load spikes during builds are benign and
|
||||
are not a stop signal; the resource that actually runs out is memory — the historic crash was RAM
|
||||
starvation from an 8–9-way fan-out, which is why 5+ concurrent builds is never acceptable.
|
||||
|
||||
Separately: never set the golden-regeneration env vars. Regenerating goldens to make a diff disappear
|
||||
hides exactly the change the golden net exists to surface.
|
||||
|
||||
## 2026-07-21 — A dropped subagent connection is transient; resume, don't restart (#542)
|
||||
|
||||
`key: process.subagent-drop-resume` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat a subagent connection drop as laptop sleep or transient network and re-resume via SendMessage — the work survives.
|
||||
**Signals:** subagent disconnect · connection drop · laptop sleep · SendMessage resume · paths: n/a · issues: #542
|
||||
**Mechanics:** n/a — client-side resume.
|
||||
|
||||
Restarting the agent from scratch discards context the agent still holds; the drop is at the transport,
|
||||
not the agent.
|
||||
|
||||
## 2026-07-21 — Every open issue carries a `priority:` label — run the H12 audit at session end (#542)
|
||||
|
||||
`key: process.issue-qualification-audit` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run `scripts/issue-qualification-audit.sh` at session end and label everything it flags, including issues you filed that session.
|
||||
**Signals:** issue qualification · priority label · queue tiering · unqualified issues · H12 · session-end check · paths: `scripts/issue-qualification-audit.sh` · issues: #542, #312
|
||||
**Mechanics:** `scripts/issue-qualification-audit.sh` (H12, ersatztv#312) lists OPEN issues missing a `priority:` label. It is fail-open without Gitea creds and advisory — exit 1 when any issue is unqualified. Sibling to H11 (both #311/#312).
|
||||
|
||||
Queue tiering keys off `priority:` and gate labels, so an issue without one is simply invisible to the
|
||||
selector — it is filed but will never be picked. "Fully qualified" means the issue has a
|
||||
`priority: {high,medium,low}` label, which signals triage ran; gate-vs-backlog is then derivable from
|
||||
the `review` label and milestone. A milestone is **not** required — backlog is unmilestoned.
|
||||
|
||||
## 2026-07-21 — Claim with `in-progress` before working; claiming is not collision safety (#542)
|
||||
|
||||
`key: process.parallel-session-claim` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Apply the `in-progress` label before starting an issue, and still read its dependency notes before touching shared surfaces — a claim prevents duplicate pickup, not overlapping code changes.
|
||||
**Signals:** parallel sessions · in-progress label · claim race · dependency notes · shared surfaces · lore pruning · paths: `docs/handoffs/chicorytv-issue-queue.md` · issues: #542
|
||||
**Mechanics:** `in-progress` label on the Gitea issue. The tiny read→claim race window is accepted; the later claimant backs off. Runner topology: two runners (ci-runner VM 127 + bumblebee-runner), 4 slots total.
|
||||
|
||||
The critical nuance is what a claim does *not* buy. Two sessions on non-overlapping issues can still
|
||||
collide in the same files. Before touching a shared surface, read the issue's dependency notes ("#234
|
||||
after #231", "coordinate with #215").
|
||||
|
||||
When editing the standing lore/handoff doc, prune covered and stale bullets rather than appending — it
|
||||
is not append-only, and git keeps the history. `git pull --rebase` before committing it, since it is
|
||||
the single most contended file across parallel sessions.
|
||||
|
||||
## 2026-07-21 — BOM/format pre-push detection: use the `xxd` loop, and run `dotnet format --include` under `bash -c` (#542)
|
||||
|
||||
`key: process.bom-format-detection-recipe` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Before any push touching `.cs`, detect BOMs with the `xxd` byte check and verify the format gate with `dotnet format --include` run under `bash -c`, never bare zsh.
|
||||
**Signals:** UTF-8 BOM · `efbbbf` · `dotnet format --verify-no-changes` · `--include` · `mapfile` · zsh vs bash · #311 format gate · detector verification · paths: `ErsatzTV.sln`, `.editorconfig` · issues: #542, #311, #70, PR #402, PR #405
|
||||
**Mechanics:** Local pre-push shell; CI's format job recipe (`ci.format-gate-folder-mode`). The BOM *policy* is `release.format-as-you-touch-rebase`; this record is the mechanics only.
|
||||
|
||||
Detection loop, verbatim:
|
||||
|
||||
```bash
|
||||
for f in $(git diff --name-only origin/main...HEAD -- '*.cs'); do head -c3 "$f" | xxd -p | grep -q "^efbbbf" && echo "BOM: $f"; done
|
||||
```
|
||||
|
||||
**Verify your detector.** An `od -An -c | grep '357 273 277'` check reported all-clean while 19 files
|
||||
were actually dirty. A detector that can only say "ok" is worse than none — use the `xxd` form or read
|
||||
the bytes directly, and distrust a clean result you didn't prove can go dirty.
|
||||
|
||||
**`dotnet format --include` DOES work here** — an earlier note claiming it silently no-ops was WRONG.
|
||||
The apparent no-op was **the shell**: CI's recipe uses `mapfile`, which is bash-only, and the default
|
||||
shell here is zsh → empty array → zero files → exit 0. Working form:
|
||||
|
||||
```bash
|
||||
bash -c 'mapfile -t files < <(git diff --name-only --diff-filter=ACM origin/main...HEAD -- "*.cs")
|
||||
dotnet format ErsatzTV.sln --no-restore --verify-no-changes --include "${files[@]}"' # exit 0 = gate passes
|
||||
```
|
||||
|
||||
## 2026-07-21 — Run `scripts/e2e-local.sh` against a FRESH config dir; a reused one hangs the readiness probe (#542)
|
||||
|
||||
`key: testing.e2e-local-fresh-config-dir` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Always point `scripts/e2e-local.sh` at a fresh config dir — leftover channels/schedules/DB rows bleed state between runs and corrupt assertions. (The *readiness-probe hang* this record was originally written about was fixed in #533; the fresh-dir rule stands on state-bleed grounds alone.)
|
||||
**Signals:** fresh config dir · state bleed between e2e runs · `READY_REGEX` · `Done migrating search index` · `Search index is already version` · readiness probe · e2e-local hang (historical, fixed #533) · paths: `scripts/e2e-local.sh`, `docs/e2e-local.md` · issues: #542, #533, #502, #524
|
||||
**Mechanics:** Since #533 the script greps a `READY_REGEX` **alternation** over both ready lines, so a reused dir no longer hangs. `scripts/e2e-ui.sh` additionally *requires* a fresh dir, because its first spec asserts the one-shot Setup gate.
|
||||
|
||||
⚠️ **The heading is historical** — as of #533 a reused dir no longer hangs the probe; read the Rule,
|
||||
not the heading. **Amended rather than superseded**, for two reasons: the operative Rule never reversed
|
||||
(only its rationale moved from "the probe hangs" to "state bleed"), and this `key:` is cited from
|
||||
`docs/handoffs/chicorytv-issue-queue.md` and two `docs/superpowers/` files — superseding would aim those
|
||||
at an archived, stale-labelled record instead of live guidance. (Note the heading being unrenameable is
|
||||
*not* an argument for amending: a supersession relocates to `archive/` with its heading intact — see
|
||||
`archive/api.md`. The two are independent.)
|
||||
|
||||
Originally: the probe waited for `Done migrating search index`, which only a **fresh** config emits; a
|
||||
reused one logs `Search index is already version {Version}`, so the line never arrived, the probe timed
|
||||
out after 120s, and the script killed a healthy server — reading as a broken build. Surfaced in #502 and
|
||||
recovered from the old tracker by the #524 triage.
|
||||
|
||||
**#533 fixed the mechanism, not the guidance.** The probe now matches either line (the handler's
|
||||
`if`/`else` is exhaustive). The original advice survives for a *different* reason — a reused dir still
|
||||
bleeds state — and `scripts/e2e-ui.sh` (#445, `ci.ui-e2e-harness`) hard-requires a fresh one, since a
|
||||
claimed local admin makes its Setup-gate spec unreachable.
|
||||
|
||||
## 2026-07-21 — Channel branding is NOT testable through the troubleshooting-playback API (#542)
|
||||
|
||||
`key: testing.troubleshoot-path-cannot-test-branding` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Verify logo/watermark/bug changes through a real channel playout — a green troubleshoot run proves nothing about branding.
|
||||
**Signals:** `PrepareTroubleshootingPlaybackHandler` · synthetic channel · `Artwork = []` · `ChannelId > 0` branch · watermark · channel logo · bug overlay · paths: `ErsatzTV.Application` troubleshooting handlers · issues: #542
|
||||
**Mechanics:** n/a — a property of the handler, not a toggle.
|
||||
|
||||
`PrepareTroubleshootingPlaybackHandler` builds a **synthetic** channel with `Artwork = []`, and its
|
||||
`ChannelId > 0` branch ignores the media item. Both together mean no branding state ever reaches the
|
||||
pipeline through that path.
|
||||
|
||||
So the troubleshoot endpoint is a false witness for this class: it exercises the transcode pipeline but
|
||||
not the overlay inputs. Route branding verification through a real channel playout instead.
|
||||
|
||||
## 2026-07-21 — Playwright-MCP: curl download endpoints, never open a tab or `window.open` (#542)
|
||||
|
||||
`key: testing.playwright-mcp-download-and-recovery` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** In Playwright-MCP E2E, fetch file-download endpoints with curl — never a browser tab or `window.open` — and if browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session.
|
||||
**Signals:** Playwright MCP · file download endpoint · window.open · browser tab stall · `ms-playwright-mcp` · paths: n/a · issues: #542
|
||||
**Mechanics:** `pkill -f ms-playwright-mcp` then re-drive from a fresh MCP session.
|
||||
|
||||
Download endpoints hand the browser a file rather than a page; the tab has nothing to snapshot and the
|
||||
tool waits on navigation that never completes. curl gives you the bytes and the status code directly.
|
||||
|
||||
Repeated stalls are a wedged MCP browser process, not a slow app — kill it and restart rather than
|
||||
retrying the same call.
|
||||
|
||||
## 2026-07-21 — Launch Codex cheap workers via `codex exec`, not `spawn_agent` (#542)
|
||||
|
||||
`key: process.codex-cheap-worker-launch` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** For bounded tool-bearing selector/recon work, launch a Codex worker with `codex exec -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only`; `spawn_agent` buys parallelism but no cost savings.
|
||||
**Signals:** codex exec · `--ephemeral` · `spawn_agent` · `gpt-5.4-mini` · `gpt-5.1-codex-mini` HTTP 400 · `model_reasoning_effort` · `~/.codex/models_cache.json` · paths: `~/.codex/models_cache.json` · issues: #542
|
||||
**Mechanics:** Local `codex` CLI on this ChatGPT-authenticated host. Tested 2026-07-14.
|
||||
|
||||
The native `spawn_agent` interface currently exposes no model or effort selector, so it parallelises at
|
||||
full cost. Launch a separate worker instead:
|
||||
|
||||
```
|
||||
codex exec --ephemeral --color never -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only -C <repo> "<hard-capped scope contract>"
|
||||
```
|
||||
|
||||
GPT-5.4-Mini is the current supported small/cost-efficient profile on this host; the older
|
||||
`gpt-5.1-codex-mini` guess fails with **HTTP 400**. Keep `low` whenever any shell/MCP/web tool is
|
||||
exposed; reserve `minimal` for explicitly tool-free synthesis over already-supplied bounded evidence.
|
||||
On model rollover, inspect the visible entries in `~/.codex/models_cache.json` rather than guessing
|
||||
names, then update this record once verified.
|
||||
|
||||
## 2026-07-21 — Live-E2E runs BEFORE the push, and downloads are curled, not browsed (#542)
|
||||
|
||||
`key: testing.live-e2e-prepush-timing` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab.
|
||||
**Signals:** live E2E timing · pre-push validation · `scripts/e2e-local.sh` · download endpoint · browser tab · paths: `scripts/e2e-local.sh`, `docs/e2e-local.md` · issues: #542
|
||||
**Mechanics:** Seeding recipe: `docs/e2e-local.md` → "Seeding a local TV library for E2E" (tiny ffmpeg testsrc MKVs + `LibraryPath` SQL rows + scan). The *requirement* itself is `release.live-e2e-required`; this record adds only the timing and the download rule.
|
||||
|
||||
Live-E2E after the push wastes a CI run to learn something the local harness would have said first —
|
||||
and on this Gitea a run cannot be cancelled. Fold E2E findings into the branch while nothing is pushed.
|
||||
|
||||
Download endpoints must be curled for the same reason as the Playwright rule: a tab has no page to
|
||||
settle on and the browser tool hangs.
|
||||
|
||||
## 2026-07-21 — The PR routine is a fixed sequence: validate locally, then push, then only follow-up commits (#542)
|
||||
|
||||
`key: process.pr-routine-sequence` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Worktree off origin/main → implement → regenerate API artifacts → full local tests + cold review + live-E2E ALL before the push → push, open PR, arm the CI monitor at open → fixes after the push are follow-up commits, never amend/force-push.
|
||||
**Signals:** PR routine · worktree off origin/main · `./scripts/update-openapi.sh` · `npm run generate:api` · cold-context review · CI monitor at PR open · no force-push on a pushed branch · derived-consent merge gate · paths: `scripts/update-openapi.sh`, `scripts/e2e-local.sh` · issues: #542
|
||||
**Mechanics:** CI watch by commit status: `/api/v1/repos/timothy/ersatztv/commits/{sha}/status`. Merge via the derived-consent gate (`pretooluse-merge-consent.sh`).
|
||||
|
||||
The sequence: create a worktree off `origin/main`; implement; for **API changes build the app project
|
||||
FIRST**, then `./scripts/update-openapi.sh`, then `npm run generate:api` (order matters — the generator
|
||||
reads the built app's spec). Then run the full local test pass, a cold-context adversarial review
|
||||
scoped to "review only", and live-E2E for UI/write-path changes — **all before the push**. Fold
|
||||
pre-push fixes into the branch; amend/reword is fine while nothing is pushed.
|
||||
|
||||
Then push, open the PR, and arm the CI monitor **at open**, not at the end. Once the branch is pushed,
|
||||
every further fix — a post-push cross-model pass, or a CI-surfaced issue — lands as a **follow-up
|
||||
commit**; never amend or force-push a pushed branch. On green, merge through the derived-consent gate
|
||||
and write the structured close comments the applicable `AGENTS.md` / `CLAUDE.md` protocol requires.
|
||||
|
||||
## 2026-07-21 — Harden a soon-to-grow feature with a runtime posture, not a hardcoded clamp (#542)
|
||||
|
||||
`key: process.harden-with-runtime-posture-not-clamp` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When a security fix constrains a capability the roadmap will later want, make the safe state the DEFAULT OF A SWITCH rather than a wall — and read the feature's own issue for its end-state first.
|
||||
**Signals:** runtime posture · feature toggle · `ERSATZTV_ALLOW_WRITES` · read-only default · hardcoded clamp · roadmap end-state · paths: n/a · issues: #542, #289, #58
|
||||
**Mechanics:** `ERSATZTV_ALLOW_WRITES` env var, default `false`. See `mcp.server-foundation`.
|
||||
|
||||
The MCP hardening in #289 could have shipped a GET-only clamp. But the MCP's final design (#58) is read
|
||||
**and** write, so that clamp would have been ripped out within a release — churn plus a window where
|
||||
the removal, not the original fix, is the risky change.
|
||||
|
||||
A runtime read-only posture (`ERSATZTV_ALLOW_WRITES`, default false) gives the same safety today and
|
||||
grows into the intended design without a revert. The generalisable step is the cheap one: check the
|
||||
feature's issue for its end-state *before* choosing between a switch and a wall.
|
||||
|
||||
## 2026-07-21 — Gitea's `?milestones=` issue filter silently no-ops on names containing `:` or `+` (#542)
|
||||
|
||||
`key: ci.gitea-milestone-filter-noop` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never filter issues with the server-side `?milestones=<name>` parameter — fetch all open issues once and filter LOCALLY on each issue's `.milestone.title`.
|
||||
**Signals:** `GET /issues?milestones=` · milestone name filter · returns the whole open-issue list · `:` or `+` in a milestone title · mis-tiered issues · count looks like "all open issues" · paths: `scripts/select-queue.sh` · issues: #542, #77, #72
|
||||
**Mechanics:** `?state=open&type=issues&limit=50`, then filter in the client, e.g. `[i for i in issues if (i['milestone'] or {}).get('title') == NAME]`. `scripts/select-queue.sh` already does this; the trap is for anyone writing a *new* query.
|
||||
|
||||
The filter does not error on a name it cannot round-trip — it returns the **entire open-issue list** as
|
||||
though every issue matched. Milestone titles here routinely contain the offending characters (e.g.
|
||||
`Scheduling: refactor + distribution`), so the failure is the normal case, not an edge case.
|
||||
|
||||
Anything trusting that response mis-tiers issues: during #77 selection it made unmilestoned #72 look
|
||||
like a member of the milestone and hid #77's true sibling set. **The tell is a result count suspiciously
|
||||
equal to the total number of open issues** — if a filtered query returns everything, the filter silently
|
||||
failed; re-derive membership locally rather than reasoning about the result.
|
||||
|
||||
The dependencies API is unaffected: `POST /issues/{n}/dependencies` with `{"owner","repo","index"}` sets
|
||||
blocked-by correctly, though the bare `{index}` form returns 201 without reliably attaching — verify
|
||||
with the GET. (That last sentence is carried forward from an earlier revision of the handoff doc,
|
||||
commit `f93458c7`, where it was dropped by a later prune rather than disproved.)
|
||||
|
||||
## 2026-07-25 — Name the model tier for every dispatched agent; a PreToolUse gate makes the silent default visible (#583)
|
||||
|
||||
`key: process.per-agent-model-routing` · `status: active` · `since: 2026-07-25` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** State the model tier (and effort, where the client exposes it) in the dispatch itself for every delegated agent — bounded recon → cheapest fast tier at `low`; mechanical slice against a documented contract → mid tier; judgment-heavy work → orchestrator tier; independent review → a different model family than the implementer.
|
||||
**Signals:** subagent model routing · `model` omitted · silent tier inheritance · orchestrator tier for a mechanical slice · capability routing · prose rule vs HARD CONSTRAINT · dispatch-time checkpoint · paths: `.claude/hooks/pretooluse-agent-model.sh`, `docs/handoffs/chicorytv-issue-queue.md` · issues: #583, #436, #440
|
||||
**Mechanics:** `Agent` tool `model` parameter; PreToolUse hook on the existing `Agent|Task` matcher in `.claude/settings.json`.
|
||||
|
||||
On 2026-07-25 a session dispatched two implementers (#436, #440) with `model` omitted on both calls;
|
||||
both silently inherited the Opus orchestrator tier. #440 was a mechanical SPA slice against an
|
||||
already-shipped backend contract — a plausible mid-tier candidate.
|
||||
|
||||
The interesting part is *why*, because it wasn't forgetfulness. **Every rule in `HARD CONSTRAINTS` was
|
||||
followed in that same session** — worktree off `origin/main`, one committing agent per worktree,
|
||||
parallelize on disjoint slices, local gate before push. Routing was the one instruction living only in
|
||||
a prose paragraph, and it was the one that got defaulted. Treat that as the general lesson: in a
|
||||
kickoff doc that is pasted into every session, **the bulleted imperative list is what actually
|
||||
functions as the checklist**, and prose above it is read as background. A rule you want followed
|
||||
belongs in the list, keyed, or it is advisory in practice.
|
||||
|
||||
Three aggravating factors, all worth checking when writing any future rule here:
|
||||
|
||||
- **Scope gap.** The low-cost-routing paragraph is written entirely about queue selection and recon
|
||||
("bounded searches, inventories, log triage, report drafting"). It never named *implementers*, and
|
||||
gave no default for the bounded-but-not-trivial case — so the largest-cost dispatch fell in a gap.
|
||||
- **The wrong default is the silent one.** Omitting `model` produces no artifact. Nothing in the
|
||||
session report revealed the tier; the operator had to ask. Contrast the BOM trap
|
||||
(`process.bom-format-detection-recipe`), where a hook fires because a memory describing the trap
|
||||
demonstrably failed to prevent it twice in one day.
|
||||
- **Distance from the decision point.** The rule sits ~line 84 of the kickoff; dispatch happens after
|
||||
orientation, claiming, the bundle scan and doc reading.
|
||||
|
||||
Hence the two-part fix: a keyed HARD CONSTRAINT that requires the tier to be **stated out loud in the
|
||||
dispatch** (a self-correcting mechanism — it turns an invisible omission into visible output), plus
|
||||
`.claude/hooks/pretooluse-agent-model.sh`, which `ask`s whenever an agent is dispatched with no
|
||||
explicit `model`. It exempts only `fork`, whose `model` override the tool ignores by design, so a
|
||||
prompt there could not be acted on. It is `ask`, never `deny`: routing is a judgment call with no
|
||||
derivable right answer, unlike the H6/H10 merge gate (`release.merge-consent-autogrant`), which
|
||||
derives a verifiable state and can therefore grant or refuse outright.
|
||||
|
||||
**The first cut was narrower, and review killed it** — worth recording, because the reasoning was
|
||||
seductive. It fired only when the prompt *text* matched implementer signals (`git commit`, `worktree`,
|
||||
`fixes #`), on the theory that a gate firing on every fan-out trains one-shot dismissal. Independent
|
||||
review confirmed the heuristic both over- and under-fired: a read-only recon brief merely *mentioning*
|
||||
"worktree" nagged, while "author the change and open a PR", "land this on the branch" and "make the
|
||||
changes and commit them" all passed silently — it missed precisely the case it existed to catch.
|
||||
**Prompt prose is not a reliable signal for authority**, and a gate with an unreliable catch rate is
|
||||
worse than none, because it gets trusted.
|
||||
|
||||
Two further reasons the broad form is right, both of which the narrow version had backwards:
|
||||
|
||||
- **It now matches the rule it enforces.** The HARD CONSTRAINT says "every dispatched agent"; a hook
|
||||
gating only implementer-looking dispatches contradicted its own rule.
|
||||
- **Routing matters MOST for the cheap cases.** The old exemption list justified itself as
|
||||
"read-only, so routing barely matters" — but bounded recon is exactly what should be explicitly
|
||||
routed *down* to a fast tier. The premise was also false: `Explore`, `Plan` and `claude-code-guide`
|
||||
all carry Bash, so none of them provably "cannot commit".
|
||||
|
||||
The noise objection is answered by the escape hatch rather than by scoping: **naming a tier costs one
|
||||
parameter and the hook never fires again**. The prompt is self-eliminating for anyone following the
|
||||
rule, which is the habit being built.
|
||||
|
||||
## 2026-07-25 — E2E cleanup kills only the PIDs it started; never a `pkill -f` pattern (#586)
|
||||
|
||||
`key: testing.e2e-cleanup-scope-by-pid` · `status: active` · `since: 2026-07-25` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** An E2E harness or agent may only kill processes whose PIDs it captured at launch — capture the PID; whoever owns the lifecycle releases it from a `trap ... EXIT INT TERM`. Never `pkill -f "dotnet ErsatzTV.dll"` (or any pattern that can match a process this run did not start). A foreign listener is reported, not reaped.
|
||||
**Signals:** `pkill -f` · `pkill -f "dotnet ErsatzTV.dll"` · E2E teardown · shared-host reap · port 8409 · port 8410 · port 8420 · `lsof -ti :$PORT` · `trap cleanup EXIT INT TERM` · truncated E2E output · delegated-brief omission · paths: `scripts/e2e-local.sh`, `scripts/e2e-ui.sh`, `docs/e2e-local.md` · issues: #586, #596, #445, #533, #440, #406
|
||||
**Mechanics:** `PID=$!` at launch, `trap cleanup EXIT INT TERM` in the lifecycle-owning script; to identify a foreign listener use `lsof -ti :"$PORT"` and fail with its PID, or re-run under `ETV_UI_PORT=<other>`.
|
||||
|
||||
A delegated agent running the #440 live-E2E found a `dotnet ErsatzTV.dll` already on the default port
|
||||
8409 that wasn't its own (mismatched `api.key`). It moved its own run to 8420 — then during cleanup ran
|
||||
a broad `pkill -f "dotnet ErsatzTV.dll"`, which would have reaped the parallel #445/#533 session's
|
||||
instance too. No damage that time: `scripts/e2e-ui.sh` owns its instance lifecycle and kills its server
|
||||
from an EXIT/INT/TERM trap, so a free 8409 was the expected post-run state, and all four determinism
|
||||
runs had already printed explicit `3 passed` summaries.
|
||||
|
||||
(That 8420 run could only have succeeded because the other instance had already released 8409 — as the
|
||||
port section below explains, at the time `ETV_UI_PORT=8420` alone still bound 8409 as well, so it would
|
||||
have failed outright against a live holder. The incident report's "correctly moved to 8420" reads as a
|
||||
clean sidestep; it was closer to a coincidence. That is itself part of why the escape hatch needed
|
||||
fixing rather than just documenting.)
|
||||
|
||||
**The near-miss is the point.** A mid-run kill of an E2E harness does not fail loudly — it truncates
|
||||
output into plausible-looking-but-wrong data, which is exactly the failure class that survives review.
|
||||
Same shape as the #406 peak-anon incident: reaping a shared-host resource you did not create.
|
||||
|
||||
**Port separation is not mitigation — and there is less of it than the incident report assumed.** A
|
||||
`pkill -f "dotnet ErsatzTV.dll"` matches the *command line*, so it hits every instance regardless of
|
||||
which port each one chose. Worse, the ports are not actually separated: the CI `functional-e2e` step
|
||||
exports `ETV_UI_PORT=8409` (`docker-build.yml`), the *same* port local runs use — 8410 is the
|
||||
`ersatztv-test` container on jazz, a different thing that the incident report conflated with it. (The
|
||||
ui-E2E step added by #445 does use 8410.) Scope by PID, not by pattern and not by port.
|
||||
|
||||
**And "just use another port" is not a complete escape either.** `Program.cs` binds *two* listeners —
|
||||
`ListenAnyIP(UiPort)` plus `ListenAnyIP(StreamingPort)` when they differ — and `SystemEnvironment.cs`
|
||||
defaults `StreamingPort` to **8409 independently of `ETV_UI_PORT`**. So `ETV_UI_PORT=8420` alone still
|
||||
binds 8409 and dies at startup against a foreign holder. An agent that took the obvious escape hatch
|
||||
would land right back in the confusion that produced the pattern kill. `scripts/e2e-local.sh` now
|
||||
defaults `ETV_STREAMING_PORT` to the port it was given, and probes both before launching.
|
||||
|
||||
**Root cause is a delegation gap, not agent error.** The brief specified a fresh config dir (per
|
||||
`testing.e2e-local-fresh-config-dir`) but said nothing about process cleanup, so the agent invented a
|
||||
reasonable-looking pattern kill. An omitted rule is not an unenforced rule — it is a rule replaced by
|
||||
whatever plausible default the agent reaches for. Hence the constraint lives in `docs/e2e-local.md`,
|
||||
where a brief-writer reads it, and not only here.
|
||||
|
||||
**Known tension with `testing.playwright-mcp-download-and-recovery`,** which prescribes
|
||||
`pkill -f ms-playwright-mcp` as the stall-recovery move. That is the same pattern-kill hazard this
|
||||
record forbids — on a shared machine it reaps every session's MCP server, not just the stalled one —
|
||||
and the rule above is deliberately general ("any pattern that can match a process this run did not
|
||||
start"), so it covers that case too. That record is not line-edited here (records are superseded, not
|
||||
rewritten — `docs.decision-lifecycle`); PID-scoping the MCP recovery is tracked as **#596**, which
|
||||
also has to answer whether a session's own MCP server is externally identifiable at all. Until that
|
||||
lands, prefer `pgrep -f ms-playwright-mcp` to identify the specific stalled PID and kill only that one.
|
||||
|
||||
**`scripts/e2e-local.sh` deliberately does NOT adopt the trap.** Its documented contract is the
|
||||
inverse of `e2e-ui.sh`'s: it boots the server, prints `PID`/`PORT`/`CONFIG_DIR`/`LOG`, and **exits
|
||||
leaving the server running** so a caller can drive the live instance. An EXIT trap there would kill the
|
||||
server the instant the launcher returned, breaking every caller including `e2e-ui.sh` and the CI
|
||||
`functional-e2e` step. The split is correct as it stands: `e2e-local.sh` is the launcher and hands
|
||||
ownership of a single PID to its caller; `e2e-ui.sh` is a lifecycle owner and traps. What #586 changed
|
||||
in `e2e-local.sh` instead is the two things that actually prevent the incident — an `lsof` pre-flight
|
||||
that names a foreign listener's PID (so an agent meeting a busy port gets a diagnosis instead of
|
||||
inventing one) and an explicit statement of who owns the printed PID.
|
||||
|
||||
## Records formerly in this file
|
||||
|
||||
Each record below moved to its own file under `records/` (ersatztv#610); the rationale is
|
||||
unchanged. Resolve by **key** — that is the stable identity. A date-based pointer from
|
||||
another doc or an old issue comment should land here and then follow the link.
|
||||
|
||||
- 2026-07-21 — "Make X consistent with Y" review findings are new code and get new-code scrutiny (#542) — [`process.consistency-fix-new-code-scrutiny`](records/process/consistency-fix-new-code-scrutiny.md)
|
||||
- 2026-07-21 — A dropped subagent connection is transient; resume, don't restart (#542) — [`process.subagent-drop-resume`](records/process/subagent-drop-resume.md)
|
||||
- 2026-07-21 — A killed CI job reports `conclusion: failure`; read the log tail before diagnosing the diff (#542) — [`ci.killed-job-triage`](records/ci/killed-job-triage.md)
|
||||
- 2026-07-21 — A lock/channel "no cross-release" verdict must enumerate every producer via grep (#542) — [`process.lock-ownership-enumerate-producers`](records/process/lock-ownership-enumerate-producers.md)
|
||||
- 2026-07-21 — A lone `decisions lifecycle` red is a known infra flake: do nothing (#542) — [`ci.decisions-lifecycle-flake`](records/ci/decisions-lifecycle-flake.md)
|
||||
- 2026-07-21 — An infra-shaped red under host load is not a code failure (#542) — [`ci.infra-shaped-red-under-load`](records/ci/infra-shaped-red-under-load.md)
|
||||
- 2026-07-21 — Arm the CI monitor at PR-open, via the commit-status endpoint (#542) — [`ci.monitor-armed-at-pr-open`](records/ci/monitor-armed-at-pr-open.md)
|
||||
- 2026-07-21 — BOM/format pre-push detection: use the `xxd` loop, and run `dotnet format --include` under `bash -c` (#542) — [`process.bom-format-detection-recipe`](records/process/bom-format-detection-recipe.md)
|
||||
- 2026-07-21 — Batch your pushes: there is no agent-side cancel route on Gitea 1.25.4 (#542) — [`ci.batch-pushes-no-cancel-route`](records/ci/batch-pushes-no-cancel-route.md)
|
||||
- 2026-07-21 — Before deleting a "workaround", enumerate every behavior it provided (#542) — [`process.enumerate-workaround-behaviors-before-deleting`](records/process/enumerate-workaround-behaviors-before-deleting.md)
|
||||
- 2026-07-21 — Bound parallel builds by FREE RAM, never regenerate goldens (#542) — [`process.build-concurrency-limits`](records/process/build-concurrency-limits.md)
|
||||
- 2026-07-21 — Build and verify locally, then trust it; CI confirms (#542) — [`ci.verify-locally-ci-confirms`](records/ci/verify-locally-ci-confirms.md)
|
||||
- 2026-07-21 — Channel branding is NOT testable through the troubleshooting-playback API (#542) — [`testing.troubleshoot-path-cannot-test-branding`](records/testing/troubleshoot-path-cannot-test-branding.md)
|
||||
- 2026-07-21 — Claim with `in-progress` before working; claiming is not collision safety (#542) — [`process.parallel-session-claim`](records/process/parallel-session-claim.md)
|
||||
- 2026-07-21 — Do not gate or throttle pushes on host health (#542) — [`ci.no-host-health-gating`](records/ci/no-host-health-gating.md)
|
||||
- 2026-07-21 — Every open issue carries a `priority:` label — run the H12 audit at session end (#542) — [`process.issue-qualification-audit`](records/process/issue-qualification-audit.md)
|
||||
- 2026-07-21 — Fix onto an unmerged feature branch by branching off it and FF-pushing (#542) — [`process.branch-off-feature-branch`](records/process/branch-off-feature-branch.md)
|
||||
- 2026-07-21 — Gitea's `?milestones=` issue filter silently no-ops on names containing `:` or `+` (#542) — [`ci.gitea-milestone-filter-noop`](records/ci/gitea-milestone-filter-noop.md)
|
||||
- 2026-07-21 — Harden a soon-to-grow feature with a runtime posture, not a hardcoded clamp (#542) — [`process.harden-with-runtime-posture-not-clamp`](records/process/harden-with-runtime-posture-not-clamp.md)
|
||||
- 2026-07-21 — Heavy-render web tests need explicit per-test vitest timeouts on the CI VM (#542) — [`ci.web-test-per-test-timeouts`](records/ci/web-test-per-test-timeouts.md)
|
||||
- 2026-07-21 — Independent cross-model review is mandatory on risky diffs; a skip is a stated, auditable exemption (#542) — [`process.independent-review-rubric`](records/process/independent-review-rubric.md)
|
||||
- 2026-07-21 — Launch Codex cheap workers via `codex exec`, not `spawn_agent` (#542) — [`process.codex-cheap-worker-launch`](records/process/codex-cheap-worker-launch.md)
|
||||
- 2026-07-21 — Live-E2E runs BEFORE the push, and downloads are curled, not browsed (#542) — [`testing.live-e2e-prepush-timing`](records/testing/live-e2e-prepush-timing.md)
|
||||
- 2026-07-21 — Local build/test gate + cold-context review run BEFORE the push, not after (#542) — [`process.local-gate-before-push`](records/process/local-gate-before-push.md)
|
||||
- 2026-07-21 — Never commit or merge inside a worktree you did not create (#542) — [`process.foreign-worktree-plumbing-merge`](records/process/foreign-worktree-plumbing-merge.md)
|
||||
- 2026-07-21 — One worktree, one committing agent (#542) — [`process.one-worktree-one-committing-agent`](records/process/one-worktree-one-committing-agent.md)
|
||||
- 2026-07-21 — Playwright-MCP: curl download endpoints, never open a tab or `window.open` (#542) — [`testing.playwright-mcp-download-and-recovery`](records/testing/playwright-mcp-download-and-recovery.md)
|
||||
- 2026-07-21 — Review disagreement on a gate PR escalates to the frontier judge — and the proposed FIX escalates with it (#542) — [`process.review-disagreement-frontier-judge`](records/process/review-disagreement-frontier-judge.md)
|
||||
- 2026-07-21 — Run `scripts/e2e-local.sh` against a FRESH config dir; a reused one hangs the readiness probe (#542) — [`testing.e2e-local-fresh-config-dir`](records/testing/e2e-local-fresh-config-dir.md)
|
||||
- 2026-07-21 — The PR routine is a fixed sequence: validate locally, then push, then only follow-up commits (#542) — [`process.pr-routine-sequence`](records/process/pr-routine-sequence.md)
|
||||
- 2026-07-21 — The shared tree at /Users/timothy/ersatztv is read-only and tells you nothing about `main` (#542) — [`process.shared-tree-readonly`](records/process/shared-tree-readonly.md)
|
||||
- 2026-07-21 — `cancelled` is not `failure`; a cancelled run is no verdict (#542) — [`ci.cancelled-is-not-a-verdict`](records/ci/cancelled-is-not-a-verdict.md)
|
||||
- 2026-07-25 — E2E cleanup kills only the PIDs it started; never a `pkill -f` pattern (#586) — [`testing.e2e-cleanup-scope-by-pid`](records/testing/e2e-cleanup-scope-by-pid.md)
|
||||
- 2026-07-25 — Name the model tier for every dispatched agent; a PreToolUse gate makes the silent default visible (#583) — [`process.per-agent-model-routing`](records/process/per-agent-model-routing.md)
|
||||
|
||||
Reference in New Issue
Block a user