docs(542): record the workflow lore, then prune the kickoff doc to instructions #544
@@ -62,6 +62,7 @@ in-file entries.
|
||||
- [`decisions/api-auth-security.md`](decisions/api-auth-security.md) — API/SPA auth & security posture (#197 bundles + #279 headers, #206, #283, #292, #295, #301, #319, #330).
|
||||
- [`decisions/release-ci-governance.md`](decisions/release-ci-governance.md) — release / CI / merge governance (#303 H3/H4/H5/H6/H9/H10, #311, #314, #315, #335).
|
||||
- [`decisions/spa-modularization.md`](decisions/spa-modularization.md) — App.tsx screen/shell extraction epic #243 (#244, #245, #247).
|
||||
- [`decisions/workflow-process.md`](decisions/workflow-process.md) — session workflow, CI-run triage, review routing, worktree/parallel-session hygiene and tooling gotchas (#542). Extracted from the kickoff handoff doc, which was their only copy.
|
||||
|
||||
**In this file:**
|
||||
|
||||
|
||||
@@ -29,15 +29,25 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `api.versioning-v1` | The entire `/api` surface is versioned to `/api/v1` uniformly (no unversioned corner); legacy unversioned callers are rewritten in-pipeline (not redirected) with Deprecation/Link/Sunset headers, and post-freeze `/api/v1` is additive-only — a breaking change requires `/api/v2`. | 2026-07-13 | [link](../decisions.md#2026-07-13--api-versioning-the-whole-api-surface-is-mounted-at-apiv1-additive-only-after-freeze-286) |
|
||||
| `blazor.rollback-tag` | The commit immediately preceding the Blazor-removal merge is tagged `blazor-final` (not a `v*` tag, so it doesn't trigger a prod release build) as the documented rollback/restore path. | 2026-07-11 | [link](../decisions.md#2026-07-11--pre-removal-blazor-rollback-tag-blazor-final-205) |
|
||||
| `blazor.ui-removed` | The legacy Blazor Server UI (`Pages/`, `Shared/`, `ViewModels/`, `Validators/`, MudBlazor + 8 other packages, Blazor Startup wiring) is fully deleted now that the SPA has parity; the legacy `MapWhen` branch is kept only for controllers/docs/OpenAPI/`LegacyUiRedirects`, and the catch-all fallback 302s any unmatched non-api/artwork/docs/openapi path to `/app`. | 2026-07-11 | [link](../decisions.md#2026-07-11--blazor-server-ui-removed-91-phase-b) |
|
||||
| `ci.batch-pushes-no-cancel-route` | Hold review fixes, doc corrections and format fixes locally and push **once** — a superseded run cannot be cancelled from the agent side and holds a runner slot until it finishes. | 2026-07-21 | [link](workflow-process.md#2026-07-21--batch-your-pushes-there-is-no-agent-side-cancel-route-on-gitea-1254-542) |
|
||||
| `ci.build-once-rejected` | CI build-once (a shared compile artifact across jobs) was implemented, measured, and rejected for a 40-85% wall-clock regression; keep the #420 cross-run tree-identity skip instead. | 2026-07-18 | [link](../decisions.md#2026-07-18--ci-build-once-was-measured-and-rejected-keep-the-420-tree-skip) |
|
||||
| `ci.cancelled-is-not-a-verdict` | Treat a `cancelled` conclusion as "no verdict" — never as pass or fail — and report FAILED and CANCELLED counts separately in any CI monitor. | 2026-07-21 | [link](workflow-process.md#2026-07-21--cancelled-is-not-failure-a-cancelled-run-is-no-verdict-542) |
|
||||
| `ci.decisions-lifecycle-flake` | When `decisions lifecycle` is the **only** red job, do not investigate and do not create a new run to clear it — no rebase, no `--amend`, no no-op push; the operator reruns that single job from the Gitea UI. | 2026-07-21 | [link](workflow-process.md#2026-07-21--a-lone-decisions-lifecycle-red-is-a-known-infra-flake-do-nothing-542) |
|
||||
| `ci.docs-only-detect-shallow-safe` | The docs-only detect script must diff against `FETCH_HEAD` (always resolves after `git fetch`, even shallow) using a two-dot tree diff — not `origin/<base>` with three-dot — because a `fetch-depth: 1` shallow clone has no remote-tracking ref and no merge-base, which silently fails the original detect into `docs_only=false` (full matrix, no functional error). A CI-behavior change must be verified by measuring the effect (job durations), not just a green check. | 2026-07-17 | [link](../decisions.md#2026-07-17--docs-only-detect-must-be-shallow-checkout-safe-fetch_head--two-dot-not-originmain--three-dot-416-follow-up) |
|
||||
| `ci.docs-only-skip-steps` | A docs-only change must still run every required job (`test`, `migrations`) so their commit-status contexts always report; each heavy job runs `scripts/ci-detect-docs-only.sh` first and gates its real STEPS on `if: steps.detect.outputs.docs_only != 'true'`, never `if:`-skips the whole job (an `if:`-skipped job reports `skipped`, not `success`, which branch protection may never unblock on). Detection biases toward running more on any doubt. | 2026-07-17 | [link](../decisions.md#2026-07-17--docs-only-ci-skip-gates-steps-in-always-running-required-jobs-never-if-skips-them-416) |
|
||||
| `ci.format-gate-folder-mode` | The blocking `format` CI job (and matching pre-commit hook) runs `dotnet format whitespace . --folder --include <files>` instead of loading the full MSBuild/Roslyn solution, cutting the gate from ~480s to ~0.5s with unchanged whitespace/charset coverage. | 2026-07-19 | [link](../decisions.md#2026-07-19--the-format-gate-runs-dotnet-format-whitespace----folder-not-the-full-solution-format-469) |
|
||||
| `ci.functional-e2e-harness` | The `functional-e2e` CI job boots the PR's own code from source via `dotnet run` (`scripts/e2e-local.sh`) and runs curl-only, deterministic assertions (`scripts/e2e-functional.sh`) as an advisory (non-blocking) job, not a `build` dependency or required check. | 2026-07-16 | [link](../decisions.md#2026-07-16--functional-e2e-ci-harness-advisory-curl-contract-job-over-an-app-booted-from-source-299) |
|
||||
| `ci.gitea-milestone-filter-noop` | Never filter issues with the server-side `?milestones=<name>` parameter — fetch all open issues once and filter LOCALLY on each issue's `.milestone.title`. | 2026-07-21 | [link](workflow-process.md#2026-07-21--giteas-milestones-issue-filter-silently-no-ops-on-names-containing--or--542) |
|
||||
| `ci.infra-shaped-red-under-load` | When a job dies inside a setup/cache step before your code compiles, check the runner host's load before diagnosing the diff, and never file a CI bug off one sample under pressure. | 2026-07-21 | [link](workflow-process.md#2026-07-21--an-infra-shaped-red-under-host-load-is-not-a-code-failure-542) |
|
||||
| `ci.killed-job-triage` | Never trust a job's `conclusion` field alone — read the log tail and require an `❌ Failure - Main …` marker before treating a red as a real failure. | 2026-07-21 | [link](workflow-process.md#2026-07-21--a-killed-ci-job-reports-conclusion-failure-read-the-log-tail-before-diagnosing-the-diff-542) |
|
||||
| `ci.monitor-armed-at-pr-open` | Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work. | 2026-07-21 | [link](workflow-process.md#2026-07-21--arm-the-ci-monitor-at-pr-open-via-the-commit-status-endpoint-542) |
|
||||
| `ci.no-host-health-gating` | Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs. | 2026-07-21 | [link](workflow-process.md#2026-07-21--do-not-gate-or-throttle-pushes-on-host-health-542) |
|
||||
| `ci.peak-anon-measurement` | The `test` job's headline memory figure is a sampled high-water mark of cgroup `anon`, produced by `scripts/ci-peak-anon.sh`; `memory.peak` and the end-of-job `anon`/`file` split are kept only as a cache-inflated reference. | 2026-07-19 | [link](../decisions.md#2026-07-19--ci-test-job-reports-a-sampled-true-peak-anon-not-cache-inflated-memorypeak-412) |
|
||||
| `ci.root-screenshot-guard` | The Husky `pre-commit` hook refuses a staged root-level `*.png` (belt-and-suspenders with the `.gitignore` rule); nested `*.png` real assets are unaffected. | 2026-07-12 | [link](release-ci-governance.md#2026-07-12--root-screenshot-guard-pre-commit-refuses-root-level-png-303-h3) |
|
||||
| `ci.runner-placement` | No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's). | 2026-07-17 | [link](../decisions.md#2026-07-17--no-persistent-compiler-servers-in-ci-every-services-container-gets-an-explicit-cap-390s-small-lane-move-reversed-406) |
|
||||
| `ci.small-lane-git-only` | `runs-on: small` is defined by what a job does (git-only), not its usual runtime; the two `docker build` jobs (docker-build.yml, ci-image.yml) move to `ubuntu-latest` because their worst-case memory, not median runtime, was pinning the small lane's per-slot cap. | 2026-07-20 | [link](../decisions.md#2026-07-20--runs-on-small-means-git-only-the-two-docker-build-jobs-move-to-ubuntu-latest-server-management639) |
|
||||
| `ci.verify-locally-ci-confirms` | Treat the local build/verify/review pass as the decision point and CI as confirmation — don't idle waiting on a run you have no reason to doubt. | 2026-07-21 | [link](workflow-process.md#2026-07-21--build-and-verify-locally-then-trust-it-ci-confirms-542) |
|
||||
| `ci.web-test-per-test-timeouts` | Give heavy-render web tests an explicit per-test vitest timeout (e.g. 15s); never raise the global default to fix one slow test. | 2026-07-21 | [link](workflow-process.md#2026-07-21--heavy-render-web-tests-need-explicit-per-test-vitest-timeouts-on-the-ci-vm-542) |
|
||||
| `concurrency.diff-scalar-fanout` | The frozen Block optimistic-concurrency recipe (api-conventions §7a) fans out to Collection/Playout×2/MultiCollection/RerunCollection, keeping a guard-returned `PreconditionFailedError` out of any handler's generic `catch(Exception)`→422 mapping, and preserving each aggregate's existing `SaveChangesAsync() > 0` gate semantics under the new unconditional `Version++`. | 2026-07-11 | [link](optimistic-concurrency.md#2026-07-11--253-pr3-diff--scalar-concurrency-fan-out-collection--playout2--multicollection--reruncollection) |
|
||||
| `concurrency.etag-rotation-completion` | Every handler that mutates a versioned root's editor-visible config state must bump `Version` (rotating the ETag) with no per-aggregate carve-outs, short-circuiting on a genuine no-op before the bump so idempotent re-submits don't fire spurious rebuild fan-out; `SaveChangesForcingVersion` rebases the retry (stored + pending delta), never adopts the stored token verbatim. | 2026-07-12 | [link](optimistic-concurrency.md#2026-07-12--cross-editor-etag-rotation-completed-for-collectionplayout-config-siblings-269) |
|
||||
| `concurrency.force-write-non-ifmatch` | Any handler that leaves a versioned root `Modified` or `Deleted` but takes no `If-Match` (deletes, item add/remove bumpers, scalar-config writers) must save through `ConcurrencyExtensions.SaveChangesForcingVersion` — force-write past a concurrent `Version` bump rather than throw an unhandled `DbUpdateConcurrencyException` (500). | 2026-07-12 | [link](optimistic-concurrency.md#2026-07-12-269--non-if-match-root-writers-force-write-past-a-concurrent-version-bump) |
|
||||
@@ -63,6 +73,24 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `media.remote-stream-probe` | `ValidatePlayoutItemPath` probes the Plex/Jellyfin/Emby remote-stream URL via `IRemoteStreamProber` before returning it; only a redirected 404 fails closed (`PlayoutItemNotAvailableFromMediaServer`), everything else fails open, and there is no toggle. | 2026-07-19 | [link](../decisions.md#2026-07-19--media-server-remote-stream-urls-are-probed-before-use-a-redirected-404-fails-closed-everything-else-fails-open-no-toggle-473) |
|
||||
| `media.remote-stream-probe-externaljson` | External-JSON playout channels' `StreamRemotely` now probes the remote-stream URL through the same `IRemoteStreamProber` seam as the generated-playout path, closing the #473 scope gap for a channel kind with no DB `PlayoutItem` rows. | 2026-07-20 | [link](../decisions.md#2026-07-20--external-json-playout-channels-now-probe-the-remote-stream-url-too-closing-the-473-scope-gap-480) |
|
||||
| `media.source-mgmt-write-api` | Media-source management (local/Plex/Jellyfin/Emby) is a REST write API + SPA under `/app/libraries/*`, wrapping existing MediatR commands 1:1 with no new commands or DB migration; connection GETs never leak a stored `apiKey`, and each PUT-replace family's identity contract is documented per-family (not assumed uniform). | 2026-07-11 | [link](../decisions.md#2026-07-11--media-source-management-rest-write-api--spa-202) |
|
||||
| `process.bom-format-detection-recipe` | Before any push touching `.cs`, detect BOMs with the `xxd` byte check and verify the format gate with `dotnet format --include` run under `bash -c`, never bare zsh. | 2026-07-21 | [link](workflow-process.md#2026-07-21--bomformat-pre-push-detection-use-the-xxd-loop-and-run-dotnet-format---include-under-bash--c-542) |
|
||||
| `process.branch-off-feature-branch` | To fix work on an unmerged feature branch, branch off that branch and land by fast-forward push — and after creating a worktree, drive the first Edit/Read from ITS absolute paths and `git status` it before building. | 2026-07-21 | [link](workflow-process.md#2026-07-21--fix-onto-an-unmerged-feature-branch-by-branching-off-it-and-ff-pushing-542) |
|
||||
| `process.build-concurrency-limits` | Run at most 3–4 concurrent dotnet/npm builds on this Mac, gate launches on FREE RAM rather than CPU load, and never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS`. | 2026-07-21 | [link](workflow-process.md#2026-07-21--bound-parallel-builds-by-free-ram-never-regenerate-goldens-542) |
|
||||
| `process.codex-cheap-worker-launch` | For bounded tool-bearing selector/recon work, launch a Codex worker with `codex exec -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only`; `spawn_agent` buys parallelism but no cost savings. | 2026-07-21 | [link](workflow-process.md#2026-07-21--launch-codex-cheap-workers-via-codex-exec-not-spawn_agent-542) |
|
||||
| `process.consistency-fix-new-code-scrutiny` | Review a "make X consistent with Y" change as new code, not as a mechanical copy — and for any timer or effect involved, ask explicitly "when does this fire?", including on mount. | 2026-07-21 | [link](workflow-process.md#2026-07-21--make-x-consistent-with-y-review-findings-are-new-code-and-get-new-code-scrutiny-542) |
|
||||
| `process.enumerate-workaround-behaviors-before-deleting` | When an issue says "delete X", enumerate every behavior X provided before removing it — a workaround often serves a second purpose that outlives the first. | 2026-07-21 | [link](workflow-process.md#2026-07-21--before-deleting-a-workaround-enumerate-every-behavior-it-provided-542) |
|
||||
| `process.foreign-worktree-plumbing-merge` | Never commit or merge inside a worktree another session created; land the merge with git plumbing against the branch ref instead. | 2026-07-21 | [link](workflow-process.md#2026-07-21--never-commit-or-merge-inside-a-worktree-you-did-not-create-542) |
|
||||
| `process.harden-with-runtime-posture-not-clamp` | When a security fix constrains a capability the roadmap will later want, make the safe state the DEFAULT OF A SWITCH rather than a wall — and read the feature's own issue for its end-state first. | 2026-07-21 | [link](workflow-process.md#2026-07-21--harden-a-soon-to-grow-feature-with-a-runtime-posture-not-a-hardcoded-clamp-542) |
|
||||
| `process.independent-review-rubric` | Run an independent review pass — preferably a different model family, otherwise a cold-context review-only agent — on any diff touching locks/concurrency, auth/security, API write-path handlers, or DB migrations, or larger than ~150 changed C# lines; skip only for a pure-SPA/docs leaf with no server-state effect, and state the skip and its reason in the PR or close comment. | 2026-07-21 | [link](workflow-process.md#2026-07-21--independent-cross-model-review-is-mandatory-on-risky-diffs-a-skip-is-a-stated-auditable-exemption-542) |
|
||||
| `process.issue-qualification-audit` | Run `scripts/issue-qualification-audit.sh` at session end and label everything it flags, including issues you filed that session. | 2026-07-21 | [link](workflow-process.md#2026-07-21--every-open-issue-carries-a-priority-label--run-the-h12-audit-at-session-end-542) |
|
||||
| `process.local-gate-before-push` | Run the local build/test gate and a cold-context, scoped "review only" adversarial review over the diff, fold the fixes, and only then push or open the PR. | 2026-07-21 | [link](workflow-process.md#2026-07-21--local-buildtest-gate--cold-context-review-run-before-the-push-not-after-542) |
|
||||
| `process.lock-ownership-enumerate-producers` | Before trusting any "single owner / no double release / no cross-release" claim, grep the whole host project for every writer of that channel message (or acquirer of that lock) — the background scheduler/worker is the usual missing producer. | 2026-07-21 | [link](workflow-process.md#2026-07-21--a-lockchannel-no-cross-release-verdict-must-enumerate-every-producer-via-grep-542) |
|
||||
| `process.one-worktree-one-committing-agent` | Never run two committing agents concurrently on one worktree — give each parallel slice its own worktree branched off the feature branch and merge back. | 2026-07-21 | [link](workflow-process.md#2026-07-21--one-worktree-one-committing-agent-542) |
|
||||
| `process.parallel-session-claim` | Apply the `in-progress` label before starting an issue, and still read its dependency notes before touching shared surfaces — a claim prevents duplicate pickup, not overlapping code changes. | 2026-07-21 | [link](workflow-process.md#2026-07-21--claim-with-in-progress-before-working-claiming-is-not-collision-safety-542) |
|
||||
| `process.pr-routine-sequence` | Worktree off origin/main → implement → regenerate API artifacts → full local tests + cold review + live-E2E ALL before the push → push, open PR, arm the CI monitor at open → fixes after the push are follow-up commits, never amend/force-push. | 2026-07-21 | [link](workflow-process.md#2026-07-21--the-pr-routine-is-a-fixed-sequence-validate-locally-then-push-then-only-follow-up-commits-542) |
|
||||
| `process.review-disagreement-frontier-judge` | When independent reviews disagree on a gate PR, escalate to the frontier judge, and put the proposed fix approach in front of it — not just the disputed finding. | 2026-07-21 | [link](workflow-process.md#2026-07-21--review-disagreement-on-a-gate-pr-escalates-to-the-frontier-judge--and-the-proposed-fix-escalates-with-it-542) |
|
||||
| `process.shared-tree-readonly` | Never commit in `/Users/timothy/ersatztv` and never read its `git log`/`git status`/HEAD to infer anything about `main` — work in a worktree off `origin/main`, which is the only source of truth. | 2026-07-21 | [link](workflow-process.md#2026-07-21--the-shared-tree-at-userstimothyersatztv-is-read-only-and-tells-you-nothing-about-main-542) |
|
||||
| `process.subagent-drop-resume` | Treat a subagent connection drop as laptop sleep or transient network and re-resume via SendMessage — the work survives. | 2026-07-21 | [link](workflow-process.md#2026-07-21--a-dropped-subagent-connection-is-transient-resume-dont-restart-542) |
|
||||
| `release.api-contract-ci-gate` | A PR touching `ErsatzTV/Controllers/Api/**` or `ErsatzTV.Core/Api/**` must ship regenerated OpenAPI artifacts (`v1.json`, `v1.d.ts`, `endpoint-index.md`) in the same diff, enforced by a blocking `api-docs` CI job that regenerates-and-diffs against a fresh build. | 2026-07-12 | [link](release-ci-governance.md#2026-07-12--blocking-ci-gate-for-api-contract-artifacts-303-h4h5) |
|
||||
| `release.done-when-merge-consent` | A PR may merge only when its linked issue's `## Done-when` checklist is fully ticked and the PR's CI is green, enforced by a PreToolUse hook on the Gitea merge tool (deny/allow/ask) plus a pre-push backstop for direct pushes to main. | 2026-07-12 | [link](release-ci-governance.md#2026-07-12--merge-consent-derived-from-state-via-a--done-when-issue-checklist-303-h6) |
|
||||
| `release.format-as-you-touch-rebase` | A blocking `format` CI job runs `dotnet format --verify-no-changes` scoped only to the PR's changed `.cs` files (never the legacy BOM backlog), and a PR branch must be kept current by rebasing on `origin/main` (never merging main in), enforced by `.husky/pre-push` → `prepush-rebase-check.sh`. | 2026-07-12 | [link](release-ci-governance.md#2026-07-12--formatting-as-you-touch-enforced-rebase-not-merge-for-pr-branches-311-h11--format-ci) |
|
||||
@@ -122,3 +150,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `spa.topbar-primary-action` | The TopBar's primary-action "+" button renders only when the active route declares a non-empty `primaryAction`, is wired (via a shared `usePrimaryAction` hook) only on single-unambiguous-create-flow list screens, and is dropped everywhere else rather than left as a dead/no-op button. | 2026-07-12 | [link](../decisions.md#2026-07-12--topbar-primary-action-button-wire-creates-drop-the-rest-238) |
|
||||
| `spa.yaml-validator-textarea` | The YAML playout validator takes pasted YAML via a `<textarea>`, not a server-side file path, since the SPA has no filesystem access. | 2026-07-09 | [link](../decisions.md#2026-07-09--yaml-playout-validator-paste-textarea-instead-of-a-server-file-path) |
|
||||
| `startup.parallel-orientation` | A fresh session runs two concurrent tracks at startup — Orientation (`AGENTS.md`/`CLAUDE.md` → `docs/README.md` task-signal map → the active decisions catalog `docs/decisions/README.md`) and, only when no issue is named, Selection (`scripts/select-queue.sh N`, deterministic live-Gitea ranking). A named issue skips Selection entirely. ersatztv#237, the closed pickup tracker this replaces, is reduced to a single archival breadcrumb and MUST NOT be read for live state. | 2026-07-21 | [link](../decisions.md#2026-07-21--parallel-orientation--selection-is-the-startup-protocol-237-retired-520) |
|
||||
| `testing.e2e-local-fresh-config-dir` | Always point `scripts/e2e-local.sh` at a fresh config dir — a reused one never emits the probe's ready line and the script kills a healthy server. | 2026-07-21 | [link](workflow-process.md#2026-07-21--run-scriptse2e-localsh-against-a-fresh-config-dir-a-reused-one-hangs-the-readiness-probe-542) |
|
||||
| `testing.live-e2e-prepush-timing` | Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab. | 2026-07-21 | [link](workflow-process.md#2026-07-21--live-e2e-runs-before-the-push-and-downloads-are-curled-not-browsed-542) |
|
||||
| `testing.playwright-mcp-download-and-recovery` | In Playwright-MCP E2E, fetch file-download endpoints with curl — never a browser tab or `window.open` — and if browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session. | 2026-07-21 | [link](workflow-process.md#2026-07-21--playwright-mcp-curl-download-endpoints-never-open-a-tab-or-windowopen-542) |
|
||||
| `testing.troubleshoot-path-cannot-test-branding` | Verify logo/watermark/bug changes through a real channel playout — a green troubleshoot run proves nothing about branding. | 2026-07-21 | [link](workflow-process.md#2026-07-21--channel-branding-is-not-testable-through-the-troubleshooting-playback-api-542) |
|
||||
|
||||
@@ -0,0 +1,526 @@
|
||||
# Session-workflow, CI-triage & review-process decisions (#542)
|
||||
|
||||
How we *work* — orchestration, CI-run triage, review routing, worktree and parallel-session hygiene,
|
||||
and the tooling gotchas that have cost real sessions. Distinct from the engineering "why the code
|
||||
does X" records: nothing here decides product behaviour.
|
||||
|
||||
These records were extracted from `docs/handoffs/chicorytv-issue-queue.md` in #542. That doc had
|
||||
become the **only** copy of most of this knowledge — an inventory found ~30 of ~38 lore bullets had
|
||||
no decision record at all — so it was neither retrievable via the catalog nor safe from a careless
|
||||
prune. The rules are now records; the handoff doc keeps the instruction and points here for the why.
|
||||
|
||||
Related: `release-ci-governance.md` (merge/release gates and hooks), `startup.parallel-orientation`
|
||||
(session-start tracks), `session.shared-checkout-refresh` (session-end shared-tree refresh).
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-21 — A killed CI job reports `conclusion: failure`; read the log tail before diagnosing the diff (#542)
|
||||
|
||||
`key: ci.killed-job-triage` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never trust a job's `conclusion` field alone — read the log tail and require an `❌ Failure - Main …` marker before treating a red as a real failure.
|
||||
**Signals:** killed job · runner restart · conclusion failure · log truncated mid-step · `❌ Failure - Main` marker · `Unable to pull refs/heads/v4` · act cache noise · UTC vs UTC+2 · semantically-null diff · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions job logs + runner container uptime on the runner host.
|
||||
|
||||
When the operator restarts the runners (a retune, a reboot), in-flight jobs die and Gitea marks them
|
||||
**`failure`**, not `cancelled`. The tell is that the log **stops mid-step with no error and no
|
||||
`❌ Failure - Main …` marker** — a real failure always leaves that marker. On run 1006, `EF migration
|
||||
integrity` and `Functional E2E` both "failed" on a **BOM-removal-only** diff (6 files, one line each,
|
||||
zero content change) that could not possibly break them; the logs simply truncated mid-`dotnet build`
|
||||
at 12:08 UTC and both runners showed `Up About an hour` — the retune had killed them.
|
||||
|
||||
**Log timestamps are UTC; the host is UTC+2.** Convert before correlating, or the restart looks two
|
||||
hours off and a correct theory gets wrongly discarded.
|
||||
|
||||
Corollary: a diff that *cannot* cause a failure is evidence the failure isn't yours — when a job that
|
||||
passed on the previous head fails on a semantically-null delta, suspect the environment and go read
|
||||
the log rather than re-litigating the diff. Beware warnings that look fatal: `Unable to pull
|
||||
refs/heads/v4: …` is act refreshing its `/root/.cache/act` action cache and is followed by `Cloned …`
|
||||
— it is noise, not a cause. Grep for the failure marker, not for the word "error".
|
||||
|
||||
## 2026-07-21 — `cancelled` is not `failure`; a cancelled run is no verdict (#542)
|
||||
|
||||
`key: ci.cancelled-is-not-a-verdict` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat a `cancelled` conclusion as "no verdict" — never as pass or fail — and report FAILED and CANCELLED counts separately in any CI monitor.
|
||||
**Signals:** conclusion cancelled · run-level vs job-level conclusion · pre-cancel genuine failure · CI monitor state != pending · phantom failure · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions run/job API; monitor logic, e.g. `fail=[j for j in jobs if j['conclusion']=='failure']; canc=[j for j in jobs if j['conclusion']=='cancelled']`.
|
||||
|
||||
The operator cancels runs by hand — they are the only party who can (see
|
||||
`ci.batch-pushes-no-cancel-route`) — and a run-level `conclusion: cancelled` means the run produced
|
||||
**no verdict** about your diff. Two traps follow. First, a run whose *overall* state is `failure` may
|
||||
hold a **genuine job failure that happened before the cancel**: check job-level `conclusion` plus
|
||||
timestamps rather than attributing the red to the cancel. Second, a cancelled run tells you nothing,
|
||||
so never claim green on one.
|
||||
|
||||
A monitor that only asks "is state != pending" will report a cancelled run as a failure and send the
|
||||
next session debugging a phantom. Split the two counts explicitly.
|
||||
|
||||
## 2026-07-21 — An infra-shaped red under host load is not a code failure (#542)
|
||||
|
||||
`key: ci.infra-shaped-red-under-load` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When a job dies inside a setup/cache step before your code compiles, check the runner host's load before diagnosing the diff, and never file a CI bug off one sample under pressure.
|
||||
**Signals:** load average 243 · `Setup .NET` 11ms failure · `remove /root/.cache/act/…/.gitignore: no such file or directory` · buildx `DeadlineExceeded: failed to compute cache key` · `Build & push image` · `EF migration integrity` · `FactoryServer-L` game server · phantom CI bug · paths: n/a · issues: #542
|
||||
**Mechanics:** `uptime` on the runner host (bumblebee); Gitea Actions job step logs.
|
||||
|
||||
On 2026-07-17, with bumblebee at load average **243**, two unrelated jobs died in ways that look like
|
||||
code bugs but weren't: `EF migration integrity` failed in ~11ms inside **`Setup .NET`** with
|
||||
`remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and main's
|
||||
`Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after ~210s of
|
||||
retries. Both re-ran green at normal load, and the migration job had no model change to test in the
|
||||
first place.
|
||||
|
||||
The diagnostic tell is the **location** of the failure: a setup or cache step, before your code
|
||||
compiles. One sample under load is not evidence of a systemic problem — this nearly got filed as #390
|
||||
lane-rebalance fallout, which the evidence did not support, and would have sent the next session
|
||||
chasing a phantom. Note also that a game server (`FactoryServer-L`) shares that host with the
|
||||
runners, so high load is not always CI's own doing.
|
||||
|
||||
## 2026-07-21 — Batch your pushes: there is no agent-side cancel route on Gitea 1.25.4 (#542)
|
||||
|
||||
`key: ci.batch-pushes-no-cancel-route` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Hold review fixes, doc corrections and format fixes locally and push **once** — a superseded run cannot be cancelled from the agent side and holds a runner slot until it finishes.
|
||||
**Signals:** cancel run 404 · Gitea 1.25.4 · `POST /api/v1/.../actions/runs/{id}/cancel` · MCP `actions_run_write` `cancel_run` · session+CSRF · `dispatch_workflow` · orphaned run · 4-slot runner · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea Actions REST + MCP `actions_run_write`; operator-only cancel in the browser UI.
|
||||
|
||||
Cancellation is impossible from the agent side on this Gitea (**1.25.4**):
|
||||
`POST /api/v1/.../actions/runs/{id}/cancel` returns **404**, MCP `actions_run_write`'s `cancel_run`
|
||||
returns **404**, and the web-UI route needs a session + CSRF that does not script (login 303s with no
|
||||
session cookie). **Only the operator can cancel, in the browser** — so if you must supersede a live
|
||||
run, say so explicitly instead of leaving it burning. (`dispatch_workflow` is a different route and
|
||||
still works for re-triggering a **main** run.)
|
||||
|
||||
This corrects the older "superseded runs drain on their own" framing: they do finish, but they hold
|
||||
one of the 4 runner slots while doing it, which is a real cost with several parallel sessions.
|
||||
|
||||
## 2026-07-21 — Do not gate or throttle pushes on host health (#542)
|
||||
|
||||
`key: ci.no-host-health-gating` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs.
|
||||
**Signals:** trust the build queue · runner retune · don't sample load before pushing · queueing is the queue's job · paths: n/a · issues: #542
|
||||
**Mechanics:** Gitea build queue (two runners, 4 slots).
|
||||
|
||||
The runners were retuned for stability (operator, 2026-07-17); queueing is the queue's job, not the
|
||||
agent's. Sampling host load before a push is both unreliable and unnecessary, and hand-scheduling
|
||||
around other sessions produces coordination that no one can verify.
|
||||
|
||||
Batch your pushes (see `ci.batch-pushes-no-cancel-route`) because orphaned runs cannot be cancelled —
|
||||
**not** because the host needs protecting. The two rules have the same action and different reasons;
|
||||
conflating them leads to load-watching behaviour that this record forbids.
|
||||
|
||||
## 2026-07-21 — A lone `decisions lifecycle` red is a known infra flake: do nothing (#542)
|
||||
|
||||
`key: ci.decisions-lifecycle-flake` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When `decisions lifecycle` is the **only** red job, do not investigate and do not create a new run to clear it — no rebase, no `--amend`, no no-op push; the operator reruns that single job from the Gitea UI.
|
||||
**Signals:** decisions lifecycle red · append-only gate · known flake · operator selective rerun · no-op push · convincing local explanation · paths: `docs/decisions.md` · issues: #542, #473, PR #479
|
||||
**Mechanics:** Gitea Actions `decisions lifecycle` job; operator-driven single-job rerun in the web UI.
|
||||
|
||||
Operator-stated, 2026-07-19. Report it as a known flake and carry on; only if *other* jobs are red too
|
||||
does the run deserve diagnosis. Same family as the killed-job rule — a spurious single-job red is
|
||||
cleared by the operator's selective rerun, never by pushing, and pushes cannot be cancelled anyway.
|
||||
|
||||
**The trap is that a convincing local explanation is always available.** On #473/PR #479 the job went
|
||||
red just after `main` landed its own `decisions.md` entry, so "mine is no longer at EOF, I must
|
||||
rebase" looked airtight. The rebase happened — and the job went red **again** on a head whose diff was
|
||||
a verified pure EOF append with zero deleted lines. A rebase that provably satisfied the gate's stated
|
||||
rule did not turn it green, which is the proof that the red was never about the diff. Check this rule
|
||||
*before* theorising about a cause.
|
||||
|
||||
## 2026-07-21 — Arm the CI monitor at PR-open, via the commit-status endpoint (#542)
|
||||
|
||||
`key: ci.monitor-armed-at-pr-open` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work.
|
||||
**Signals:** arm monitor at PR open · commit status endpoint · head sha · red run sat unnoticed · MySQL-apply flake · paths: n/a · issues: #542, #216
|
||||
**Mechanics:** `GET /api/v1/repos/timothy/ersatztv/commits/{sha}/status`.
|
||||
|
||||
CI runs concurrently with the review and E2E work that follows a push, so the cost of a late-armed
|
||||
monitor is entirely wasted wall-clock. In the #216 session three PR runs sat red for roughly an hour
|
||||
on a CI-only flake while review and E2E ran to completion — the reds were only discovered afterwards.
|
||||
|
||||
Related context for interpreting an early red: the old MySQL host-port 3306 collision is fixed on main
|
||||
(`ef8915f1`), so a lone MySQL-apply red now indicates the known infra flake and warrants a rerun
|
||||
rather than diagnosis.
|
||||
|
||||
## 2026-07-21 — Heavy-render web tests need explicit per-test vitest timeouts on the CI VM (#542)
|
||||
|
||||
`key: ci.web-test-per-test-timeouts` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Give heavy-render web tests an explicit per-test vitest timeout (e.g. 15s); never raise the global default to fix one slow test.
|
||||
**Signals:** vitest timeout · 5s default · 100+ item grid · CI VM slower than local · run 686 · paths: `web/` tests · issues: #542
|
||||
**Mechanics:** per-test timeout argument in the vitest test declaration.
|
||||
|
||||
The CI VM is materially slower than a dev Mac for render-heavy work. A web test rendering a 100+ item
|
||||
grid runs in about 1s locally but hit the **5s** vitest default on the CI VM (run 686) and went red
|
||||
for no code reason.
|
||||
|
||||
Bump the timeout on the specific test. Raising the global default hides genuine hangs across the whole
|
||||
suite in exchange for fixing one known-slow case.
|
||||
|
||||
## 2026-07-21 — Build and verify locally, then trust it; CI confirms (#542)
|
||||
|
||||
`key: ci.verify-locally-ci-confirms` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat the local build/verify/review pass as the decision point and CI as confirmation — don't idle waiting on a run you have no reason to doubt.
|
||||
**Signals:** ~9 min warm CI · docs-only PR seconds · CI VM 127 · commit status watch · local gate first · paths: n/a · issues: #542, #416
|
||||
**Mechanics:** `GET /api/v1/repos/timothy/ersatztv/commits/{sha}/status`; CI on VM 127.
|
||||
|
||||
A warm full CI run takes roughly **9 minutes**; a docs-only PR completes in seconds since ersatztv#416's
|
||||
docs-only skip. Neither duration justifies blocking on the run when the same gates already passed
|
||||
locally.
|
||||
|
||||
Watch the run by commit status rather than by polling the UI, and continue working. CI *confirms* the
|
||||
local verdict; it is not the first line of defence.
|
||||
|
||||
## 2026-07-21 — Independent cross-model review is mandatory on risky diffs; a skip is a stated, auditable exemption (#542)
|
||||
|
||||
`key: process.independent-review-rubric` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run an independent review pass — preferably a different model family, otherwise a cold-context review-only agent — on any diff touching locks/concurrency, auth/security, API write-path handlers, or DB migrations, or larger than ~150 changed C# lines; skip only for a pure-SPA/docs leaf with no server-state effect, and state the skip and its reason in the PR or close comment.
|
||||
**Signals:** cross-model review rotation · correlated blindspots · skip rubric · self-exemption · review-only agent · quota exhaustion · reviewer does not implement · paths: n/a · issues: #542, #235, #250
|
||||
**Mechanics:** rotation — Codex reviews Claude-authored changes, Claude reviews Codex-authored changes; use whatever native integration is installed (client helpers like `codex:rescue` are optional accelerators, not protocol dependencies). On quota/tool exhaustion, fall back to a cold-context review-only agent in the current client and retry cross-model next window. Independent reviewers do review and stuck-diagnosis work only — never implementation on the same change they review.
|
||||
|
||||
The justification is correlated blindspots: independent model families miss *different* classes of
|
||||
defect, so a single reviewer on a risky diff is the gap, not a saving. #235 (a reviewer clearing a lock
|
||||
as sound because it never grepped for every producer) and #250 (a vacuous negative control that
|
||||
silently ran a stale dll) are both cases the second, differently-blind pass caught.
|
||||
|
||||
The skip clause is deliberately narrow and deliberately loud. A self-exemption is an auditable claim,
|
||||
not a vibe — silence is indistinguishable from forgetting, so the exemption must be written down where
|
||||
the merge record can be re-read later.
|
||||
|
||||
## 2026-07-21 — Local build/test gate + cold-context review run BEFORE the push, not after (#542)
|
||||
|
||||
`key: process.local-gate-before-push` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run the local build/test gate and a cold-context, scoped "review only" adversarial review over the diff, fold the fixes, and only then push or open the PR.
|
||||
**Signals:** local gate before push · cold review before CI · CI is not the first line of defence · uncancellable run · queue pressure · paths: n/a · issues: #542
|
||||
**Mechanics:** local build/test gate + a scoped review-only agent over `origin/main...HEAD`; CI (armed on the PR head sha at PR-open) *confirms* the result.
|
||||
|
||||
Gitea runners should only ever see already-validated work. CI here is a scarce shared slow resource and
|
||||
a run cannot be cancelled on this Gitea version, so spending a run to surface a lint error or a
|
||||
review-catchable bug is pure waste plus queue pressure on every other in-flight session.
|
||||
|
||||
CI confirms; it is not the first line of defence. Treating it as the first line inverts the cost: the
|
||||
cheap fast private check is skipped in favour of the expensive slow shared one.
|
||||
|
||||
## 2026-07-21 — Review disagreement on a gate PR escalates to the frontier judge — and the proposed FIX escalates with it (#542)
|
||||
|
||||
`key: process.review-disagreement-frontier-judge` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When independent reviews disagree on a gate PR, escalate to the frontier judge, and put the proposed fix approach in front of it — not just the disputed finding.
|
||||
**Signals:** review disagreement · frontier judge · escalate the fix not just the verdict · gate PR reconciliation · paths: n/a · issues: #542, #235
|
||||
**Mechanics:** n/a — escalation is a routing decision by the orchestrator.
|
||||
|
||||
The reconciliation is not only "who is right about the finding". On #235 the frontier model ratified
|
||||
the disputed High *and* caught that the orchestrator's own proposed fix (a per-source lock-or-skip over
|
||||
a fixed loop order) would deterministically starve the 2nd and later media source — prescribing instead
|
||||
the pattern that shipped.
|
||||
|
||||
That is the whole argument: a judge given only the verdict question would have returned "the High is
|
||||
real" and let a starving fix ship. Send the fix approach up with the finding.
|
||||
|
||||
## 2026-07-21 — "Make X consistent with Y" review findings are new code and get new-code scrutiny (#542)
|
||||
|
||||
`key: process.consistency-fix-new-code-scrutiny` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Review a "make X consistent with Y" change as new code, not as a mechanical copy — and for any timer or effect involved, ask explicitly "when does this fire?", including on mount.
|
||||
**Signals:** consistency fix · propagating the pattern propagates the bug · mount-debounce · timer/effect fire timing · paths: n/a · issues: #542, #216
|
||||
**Mechanics:** n/a — a review-attention rule.
|
||||
|
||||
Propagating a pattern propagates its bugs. A #216 review fix copied MediaBrowseScreen's mount-debounce
|
||||
selection-wipe race into SearchScreen instead of catching it — the finding said "be consistent", and
|
||||
consistency was achieved, including with the defect.
|
||||
|
||||
The mount question is the cheap general form of the check: most of this class hides in an effect or
|
||||
timer that fires once more than the author pictured.
|
||||
|
||||
## 2026-07-21 — A lock/channel "no cross-release" verdict must enumerate every producer via grep (#542)
|
||||
|
||||
`key: process.lock-ownership-enumerate-producers` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Before trusting any "single owner / no double release / no cross-release" claim, grep the whole host project for every writer of that channel message (or acquirer of that lock) — the background scheduler/worker is the usual missing producer.
|
||||
**Signals:** cross-release · lock ownership · channel message producers · scheduler enqueues too · grep the host project · paths: n/a · issues: #542, #235, #267, #250
|
||||
**Mechanics:** grep for every sender of the message type across the host project (e.g. all `Synchronize{X}Collections(` call sites), not just the controllers present in the PR diff. General form of "re-derive from a grep rather than enumerating from a list": see `sched.weighted-shuffle`.
|
||||
|
||||
On #235/#267 a cold fork cleared a new `scan-collections` per-provider lock as sound — "only the
|
||||
acquirer enqueues, so only its message triggers the scanner unlock" — having checked the three new
|
||||
controllers in the diff and stopped there. Codex grepped: `SchedulerService` also enqueues those
|
||||
messages periodically and unlocked, and `ScannerService`'s `finally` released the lock whenever held,
|
||||
so a scheduled scan cross-releases an API scan's lock (the #250 class).
|
||||
|
||||
Corollary: the independent pass earns its keep even when the first reviewer returns MERGEABLE. The
|
||||
first verdict was not careless — it was complete over the evidence it looked at, which is exactly the
|
||||
failure mode a second, differently-scoped reviewer catches.
|
||||
|
||||
## 2026-07-21 — Before deleting a "workaround", enumerate every behavior it provided (#542)
|
||||
|
||||
`key: process.enumerate-workaround-behaviors-before-deleting` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When an issue says "delete X", enumerate every behavior X provided before removing it — a workaround often serves a second purpose that outlives the first.
|
||||
**Signals:** remove the workaround · double duty · second purpose · bounded version re-scoped · PENDING_GRACE_TICKS · paths: n/a · issues: #542, #232
|
||||
**Mechanics:** n/a — a scoping rule applied when reading a removal issue.
|
||||
|
||||
#232 said "remove the SPA `PENDING_GRACE_TICKS` grace-tick workaround now the API is honest". The grace
|
||||
ticks did double duty: compensating the lying-200 (genuinely obsolete) *and* absorbing the inherent
|
||||
queue→observed-active lag and fast-completion race (still needed). Removing them wholesale wedged the
|
||||
scan button disabled-until-reload for short scans; both the fork and Codex caught it.
|
||||
|
||||
The landing move is usually not "delete" or "keep" but a bounded version re-scoped to the purpose that
|
||||
still holds.
|
||||
|
||||
## 2026-07-21 — The shared tree at /Users/timothy/ersatztv is read-only and tells you nothing about `main` (#542)
|
||||
|
||||
`key: process.shared-tree-readonly` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never commit in `/Users/timothy/ersatztv` and never read its `git log`/`git status`/HEAD to infer anything about `main` — work in a worktree off `origin/main`, which is the only source of truth.
|
||||
**Signals:** shared checkout · "main checkout" · worktree discipline · stale HEAD · origin/main is truth · node_modules copy · paths: `/Users/timothy/ersatztv`, `.claude/worktrees/`, `.worktrees/` · issues: #542
|
||||
**Mechanics:** `git worktree add -b <branch> <path> origin/main`; `git fetch origin main` and reference `origin/main` explicitly — the local `main` ref lags and nobody is obliged to update it. The one sanctioned use of the shared tree is read-only: copying `web/node_modules` out of it (keep that fresh — a stale copy broke typecheck in a #198 worktree).
|
||||
|
||||
There is no "main checkout". `/Users/timothy/ersatztv` is a shared mutable working tree whose HEAD is
|
||||
whatever the last session left there; its name lies. The rule holds even for a one-line docs fix — make
|
||||
a worktree.
|
||||
|
||||
This is a design flaw, not a discipline failure: do not "fix" it by telling sessions to check harder,
|
||||
because a check does not stay true. It broke for two different sessions on the same day, 2026-07-17,
|
||||
both doing the obvious thing. Session A assumed it was on `main` — correct when it pulled at 12:46,
|
||||
stale by 14:17 — and committed; the commit landed on the CI session's `ci/604-lane-rebalance`, 24
|
||||
seconds after that session's own commit at 14:17:28. Two writers, one tree; nothing tangled, by luck.
|
||||
Session B ran `git log` there, read that same branch's HEAD, and concluded `main` was "4 behind origin"
|
||||
— a phantom. A session-start `git status` *appears to confirm* the false assumption, which makes it
|
||||
worse than a plainly wrong claim.
|
||||
|
||||
Cross-reference `session.shared-checkout-refresh`, which covers the complementary failure mode: any
|
||||
FILE (including a pasted kickoff prompt) read out of that tree may be stale, and the session-end
|
||||
refresh script that mitigates it. This record covers the git-state failure mode only.
|
||||
|
||||
## 2026-07-21 — One worktree, one committing agent (#542)
|
||||
|
||||
`key: process.one-worktree-one-committing-agent` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never run two committing agents concurrently on one worktree — give each parallel slice its own worktree branched off the feature branch and merge back.
|
||||
**Signals:** parallel agents · concurrent commits · worktree per slice · read-only review agent · paths: n/a · issues: #542
|
||||
**Mechanics:** `git worktree add` a fresh tree per slice off the feature branch; merge the slices back into the feature branch when they land.
|
||||
|
||||
Read-only review agents sharing a tree are fine — the constraint is on *writers*. Sequence slices only
|
||||
where one genuinely depends on another's output; "backend-first narrow, SPA-wide after" worked well for
|
||||
#216.
|
||||
|
||||
## 2026-07-21 — Never commit or merge inside a worktree you did not create (#542)
|
||||
|
||||
`key: process.foreign-worktree-plumbing-merge` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never commit or merge inside a worktree another session created; land the merge with git plumbing against the branch ref instead.
|
||||
**Signals:** sibling session worktree · branch already checked out · plumbing merge · read-tree · commit-tree · GIT_INDEX_FILE · paths: n/a · issues: #542
|
||||
**Mechanics:** `git read-tree -m <base> <ours> <theirs>` into a temporary `GIT_INDEX_FILE` → `git write-tree` → `git commit-tree -p <ours> -p <theirs>` → push the resulting commit to the branch ref. The owning session then `git pull`s it.
|
||||
|
||||
With several sessions on one machine, any given branch may already be checked out in a sibling
|
||||
session's worktree. Touching that checkout writes into another agent's working state; the plumbing
|
||||
route mutates only the ref, so the merge lands without disturbing the owner's tree.
|
||||
|
||||
## 2026-07-21 — Fix onto an unmerged feature branch by branching off it and FF-pushing (#542)
|
||||
|
||||
`key: process.branch-off-feature-branch` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** To fix work on an unmerged feature branch, branch off that branch and land by fast-forward push — and after creating a worktree, drive the first Edit/Read from ITS absolute paths and `git status` it before building.
|
||||
**Signals:** unmerged feature branch · FF-push · path leak · sibling worktree recon · wrong tree edits · "0 new tests" · paths: n/a · issues: #542
|
||||
**Mechanics:** `git worktree add -b <fix-branch> <path> <feature-branch>`; land with a fast-forward push to the feature branch. Never touch a sibling worktree that has the branch checked out (see `process.foreign-worktree-plumbing-merge`).
|
||||
|
||||
The path-leak trap, hit on 2026-07-12 (#289 → PR#76): reconnaissance by `Read`ing a *sibling* worktree
|
||||
leaked that tree's absolute paths into subsequent `Edit`s, so the edits landed there rather than in the
|
||||
fresh worktree. `dotnet` then built the unchanged tree, and the tell was a build reporting "0 new
|
||||
tests". A `git status` in the new worktree before the first build catches this in one command.
|
||||
|
||||
## 2026-07-21 — Bound parallel builds by FREE RAM, never regenerate goldens (#542)
|
||||
|
||||
`key: process.build-concurrency-limits` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run at most 3–4 concurrent dotnet/npm builds on this Mac, gate launches on FREE RAM rather than CPU load, and never set `ETV_UPDATE_GOLDENS` / `ETV_UPDATE_PLAYOUT_GOLDENS`.
|
||||
**Signals:** build concurrency · memory_pressure · RAM starvation · fan-out cap · golden regeneration · paths: n/a · issues: #542
|
||||
**Mechanics:** `memory_pressure -Q` — under 20% free, do not launch more build agents; under 10%, pause or stagger. Host is an M4, 10 cores, 16 GB.
|
||||
|
||||
Parallelize by default when slices are genuinely disjoint: use up to the client's concurrency limit,
|
||||
normally 2–3 delegated agents plus the orchestrator. Fan out bounded recon early, but do not invent
|
||||
artificial slices or exceed a runtime-imposed agent cap. CPU load spikes during builds are benign and
|
||||
are not a stop signal; the resource that actually runs out is memory — the historic crash was RAM
|
||||
starvation from an 8–9-way fan-out, which is why 5+ concurrent builds is never acceptable.
|
||||
|
||||
Separately: never set the golden-regeneration env vars. Regenerating goldens to make a diff disappear
|
||||
hides exactly the change the golden net exists to surface.
|
||||
|
||||
## 2026-07-21 — A dropped subagent connection is transient; resume, don't restart (#542)
|
||||
|
||||
`key: process.subagent-drop-resume` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Treat a subagent connection drop as laptop sleep or transient network and re-resume via SendMessage — the work survives.
|
||||
**Signals:** subagent disconnect · connection drop · laptop sleep · SendMessage resume · paths: n/a · issues: #542
|
||||
**Mechanics:** n/a — client-side resume.
|
||||
|
||||
Restarting the agent from scratch discards context the agent still holds; the drop is at the transport,
|
||||
not the agent.
|
||||
|
||||
## 2026-07-21 — Every open issue carries a `priority:` label — run the H12 audit at session end (#542)
|
||||
|
||||
`key: process.issue-qualification-audit` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run `scripts/issue-qualification-audit.sh` at session end and label everything it flags, including issues you filed that session.
|
||||
**Signals:** issue qualification · priority label · queue tiering · unqualified issues · H12 · session-end check · paths: `scripts/issue-qualification-audit.sh` · issues: #542, #312
|
||||
**Mechanics:** `scripts/issue-qualification-audit.sh` (H12, ersatztv#312) lists OPEN issues missing a `priority:` label. It is fail-open without Gitea creds and advisory — exit 1 when any issue is unqualified. Sibling to H11 (both #311/#312).
|
||||
|
||||
Queue tiering keys off `priority:` and gate labels, so an issue without one is simply invisible to the
|
||||
selector — it is filed but will never be picked. "Fully qualified" means the issue has a
|
||||
`priority: {high,medium,low}` label, which signals triage ran; gate-vs-backlog is then derivable from
|
||||
the `review` label and milestone. A milestone is **not** required — backlog is unmilestoned.
|
||||
|
||||
## 2026-07-21 — Claim with `in-progress` before working; claiming is not collision safety (#542)
|
||||
|
||||
`key: process.parallel-session-claim` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Apply the `in-progress` label before starting an issue, and still read its dependency notes before touching shared surfaces — a claim prevents duplicate pickup, not overlapping code changes.
|
||||
**Signals:** parallel sessions · in-progress label · claim race · dependency notes · shared surfaces · lore pruning · paths: `docs/handoffs/chicorytv-issue-queue.md` · issues: #542
|
||||
**Mechanics:** `in-progress` label on the Gitea issue. The tiny read→claim race window is accepted; the later claimant backs off. Runner topology: two runners (ci-runner VM 127 + bumblebee-runner), 4 slots total.
|
||||
|
||||
The critical nuance is what a claim does *not* buy. Two sessions on non-overlapping issues can still
|
||||
collide in the same files. Before touching a shared surface, read the issue's dependency notes ("#234
|
||||
after #231", "coordinate with #215").
|
||||
|
||||
When editing the standing lore/handoff doc, prune covered and stale bullets rather than appending — it
|
||||
is not append-only, and git keeps the history. `git pull --rebase` before committing it, since it is
|
||||
the single most contended file across parallel sessions.
|
||||
|
||||
## 2026-07-21 — BOM/format pre-push detection: use the `xxd` loop, and run `dotnet format --include` under `bash -c` (#542)
|
||||
|
||||
`key: process.bom-format-detection-recipe` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Before any push touching `.cs`, detect BOMs with the `xxd` byte check and verify the format gate with `dotnet format --include` run under `bash -c`, never bare zsh.
|
||||
**Signals:** UTF-8 BOM · `efbbbf` · `dotnet format --verify-no-changes` · `--include` · `mapfile` · zsh vs bash · #311 format gate · detector verification · paths: `ErsatzTV.sln`, `.editorconfig` · issues: #542, #311, #70, PR #402, PR #405
|
||||
**Mechanics:** Local pre-push shell; CI's format job recipe (`ci.format-gate-folder-mode`). The BOM *policy* is `release.format-as-you-touch-rebase`; this record is the mechanics only.
|
||||
|
||||
Detection loop, verbatim:
|
||||
|
||||
```bash
|
||||
for f in $(git diff --name-only origin/main...HEAD -- '*.cs'); do head -c3 "$f" | xxd -p | grep -q "^efbbbf" && echo "BOM: $f"; done
|
||||
```
|
||||
|
||||
**Verify your detector.** An `od -An -c | grep '357 273 277'` check reported all-clean while 19 files
|
||||
were actually dirty. A detector that can only say "ok" is worse than none — use the `xxd` form or read
|
||||
the bytes directly, and distrust a clean result you didn't prove can go dirty.
|
||||
|
||||
**`dotnet format --include` DOES work here** — an earlier note claiming it silently no-ops was WRONG.
|
||||
The apparent no-op was **the shell**: CI's recipe uses `mapfile`, which is bash-only, and the default
|
||||
shell here is zsh → empty array → zero files → exit 0. Working form:
|
||||
|
||||
```bash
|
||||
bash -c 'mapfile -t files < <(git diff --name-only --diff-filter=ACM origin/main...HEAD -- "*.cs")
|
||||
dotnet format ErsatzTV.sln --no-restore --verify-no-changes --include "${files[@]}"' # exit 0 = gate passes
|
||||
```
|
||||
|
||||
## 2026-07-21 — Run `scripts/e2e-local.sh` against a FRESH config dir; a reused one hangs the readiness probe (#542)
|
||||
|
||||
`key: testing.e2e-local-fresh-config-dir` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Always point `scripts/e2e-local.sh` at a fresh config dir — a reused one never emits the probe's ready line and the script kills a healthy server.
|
||||
**Signals:** `READY_LINE` · `Done migrating search index` · `Search index is already version 50` · readiness probe timeout · e2e-local hang · paths: `scripts/e2e-local.sh`, `docs/e2e-local.md` · issues: #542, #533, #502, #524
|
||||
**Mechanics:** The script greps one fixed `READY_LINE` — there is no "wait for either line" option today.
|
||||
|
||||
The probe waits for `Done migrating search index`, which only a **fresh** config emits. A reused config
|
||||
logs `Search index is already version 50` instead, so the line never arrives, the probe times out, and
|
||||
the script kills a perfectly healthy server — a failure that reads as a broken build.
|
||||
|
||||
Surfaced in the #502 session (two failed launches before the cause was obvious) and recovered from the
|
||||
old tracker's comment log by the #524 triage. `docs/e2e-local.md` documents the ready line and
|
||||
recommends a fresh dir, but not this failure mode. Widening the probe to
|
||||
`Done migrating search index|Search index is already version` is tracked as **#533**.
|
||||
|
||||
## 2026-07-21 — Channel branding is NOT testable through the troubleshooting-playback API (#542)
|
||||
|
||||
`key: testing.troubleshoot-path-cannot-test-branding` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Verify logo/watermark/bug changes through a real channel playout — a green troubleshoot run proves nothing about branding.
|
||||
**Signals:** `PrepareTroubleshootingPlaybackHandler` · synthetic channel · `Artwork = []` · `ChannelId > 0` branch · watermark · channel logo · bug overlay · paths: `ErsatzTV.Application` troubleshooting handlers · issues: #542
|
||||
**Mechanics:** n/a — a property of the handler, not a toggle.
|
||||
|
||||
`PrepareTroubleshootingPlaybackHandler` builds a **synthetic** channel with `Artwork = []`, and its
|
||||
`ChannelId > 0` branch ignores the media item. Both together mean no branding state ever reaches the
|
||||
pipeline through that path.
|
||||
|
||||
So the troubleshoot endpoint is a false witness for this class: it exercises the transcode pipeline but
|
||||
not the overlay inputs. Route branding verification through a real channel playout instead.
|
||||
|
||||
## 2026-07-21 — Playwright-MCP: curl download endpoints, never open a tab or `window.open` (#542)
|
||||
|
||||
`key: testing.playwright-mcp-download-and-recovery` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** In Playwright-MCP E2E, fetch file-download endpoints with curl — never a browser tab or `window.open` — and if browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session.
|
||||
**Signals:** Playwright MCP · file download endpoint · window.open · browser tab stall · `ms-playwright-mcp` · paths: n/a · issues: #542
|
||||
**Mechanics:** `pkill -f ms-playwright-mcp` then re-drive from a fresh MCP session.
|
||||
|
||||
Download endpoints hand the browser a file rather than a page; the tab has nothing to snapshot and the
|
||||
tool waits on navigation that never completes. curl gives you the bytes and the status code directly.
|
||||
|
||||
Repeated stalls are a wedged MCP browser process, not a slow app — kill it and restart rather than
|
||||
retrying the same call.
|
||||
|
||||
## 2026-07-21 — Launch Codex cheap workers via `codex exec`, not `spawn_agent` (#542)
|
||||
|
||||
`key: process.codex-cheap-worker-launch` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** For bounded tool-bearing selector/recon work, launch a Codex worker with `codex exec -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only`; `spawn_agent` buys parallelism but no cost savings.
|
||||
**Signals:** codex exec · `--ephemeral` · `spawn_agent` · `gpt-5.4-mini` · `gpt-5.1-codex-mini` HTTP 400 · `model_reasoning_effort` · `~/.codex/models_cache.json` · paths: `~/.codex/models_cache.json` · issues: #542
|
||||
**Mechanics:** Local `codex` CLI on this ChatGPT-authenticated host. Tested 2026-07-14.
|
||||
|
||||
The native `spawn_agent` interface currently exposes no model or effort selector, so it parallelises at
|
||||
full cost. Launch a separate worker instead:
|
||||
|
||||
```
|
||||
codex exec --ephemeral --color never -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only -C <repo> "<hard-capped scope contract>"
|
||||
```
|
||||
|
||||
GPT-5.4-Mini is the current supported small/cost-efficient profile on this host; the older
|
||||
`gpt-5.1-codex-mini` guess fails with **HTTP 400**. Keep `low` whenever any shell/MCP/web tool is
|
||||
exposed; reserve `minimal` for explicitly tool-free synthesis over already-supplied bounded evidence.
|
||||
On model rollover, inspect the visible entries in `~/.codex/models_cache.json` rather than guessing
|
||||
names, then update this record once verified.
|
||||
|
||||
## 2026-07-21 — Live-E2E runs BEFORE the push, and downloads are curled, not browsed (#542)
|
||||
|
||||
`key: testing.live-e2e-prepush-timing` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Run live-E2E via `scripts/e2e-local.sh` before pushing a write-path or UI change, and exercise download endpoints with curl, never a browser tab.
|
||||
**Signals:** live E2E timing · pre-push validation · `scripts/e2e-local.sh` · download endpoint · browser tab · paths: `scripts/e2e-local.sh`, `docs/e2e-local.md` · issues: #542
|
||||
**Mechanics:** Seeding recipe: `docs/e2e-local.md` → "Seeding a local TV library for E2E" (tiny ffmpeg testsrc MKVs + `LibraryPath` SQL rows + scan). The *requirement* itself is `release.live-e2e-required`; this record adds only the timing and the download rule.
|
||||
|
||||
Live-E2E after the push wastes a CI run to learn something the local harness would have said first —
|
||||
and on this Gitea a run cannot be cancelled. Fold E2E findings into the branch while nothing is pushed.
|
||||
|
||||
Download endpoints must be curled for the same reason as the Playwright rule: a tab has no page to
|
||||
settle on and the browser tool hangs.
|
||||
|
||||
## 2026-07-21 — The PR routine is a fixed sequence: validate locally, then push, then only follow-up commits (#542)
|
||||
|
||||
`key: process.pr-routine-sequence` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Worktree off origin/main → implement → regenerate API artifacts → full local tests + cold review + live-E2E ALL before the push → push, open PR, arm the CI monitor at open → fixes after the push are follow-up commits, never amend/force-push.
|
||||
**Signals:** PR routine · worktree off origin/main · `./scripts/update-openapi.sh` · `npm run generate:api` · cold-context review · CI monitor at PR open · no force-push on a pushed branch · derived-consent merge gate · paths: `scripts/update-openapi.sh`, `scripts/e2e-local.sh` · issues: #542
|
||||
**Mechanics:** CI watch by commit status: `/api/v1/repos/timothy/ersatztv/commits/{sha}/status`. Merge via the derived-consent gate (`pretooluse-merge-consent.sh`).
|
||||
|
||||
The sequence: create a worktree off `origin/main`; implement; for **API changes build the app project
|
||||
FIRST**, then `./scripts/update-openapi.sh`, then `npm run generate:api` (order matters — the generator
|
||||
reads the built app's spec). Then run the full local test pass, a cold-context adversarial review
|
||||
scoped to "review only", and live-E2E for UI/write-path changes — **all before the push**. Fold
|
||||
pre-push fixes into the branch; amend/reword is fine while nothing is pushed.
|
||||
|
||||
Then push, open the PR, and arm the CI monitor **at open**, not at the end. Once the branch is pushed,
|
||||
every further fix — a post-push cross-model pass, or a CI-surfaced issue — lands as a **follow-up
|
||||
commit**; never amend or force-push a pushed branch. On green, merge through the derived-consent gate
|
||||
and write the structured close comments the applicable `AGENTS.md` / `CLAUDE.md` protocol requires.
|
||||
|
||||
## 2026-07-21 — Harden a soon-to-grow feature with a runtime posture, not a hardcoded clamp (#542)
|
||||
|
||||
`key: process.harden-with-runtime-posture-not-clamp` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** When a security fix constrains a capability the roadmap will later want, make the safe state the DEFAULT OF A SWITCH rather than a wall — and read the feature's own issue for its end-state first.
|
||||
**Signals:** runtime posture · feature toggle · `ERSATZTV_ALLOW_WRITES` · read-only default · hardcoded clamp · roadmap end-state · paths: n/a · issues: #542, #289, #58
|
||||
**Mechanics:** `ERSATZTV_ALLOW_WRITES` env var, default `false`. See `mcp.server-foundation`.
|
||||
|
||||
The MCP hardening in #289 could have shipped a GET-only clamp. But the MCP's final design (#58) is read
|
||||
**and** write, so that clamp would have been ripped out within a release — churn plus a window where
|
||||
the removal, not the original fix, is the risky change.
|
||||
|
||||
A runtime read-only posture (`ERSATZTV_ALLOW_WRITES`, default false) gives the same safety today and
|
||||
grows into the intended design without a revert. The generalisable step is the cheap one: check the
|
||||
feature's issue for its end-state *before* choosing between a switch and a wall.
|
||||
|
||||
## 2026-07-21 — Gitea's `?milestones=` issue filter silently no-ops on names containing `:` or `+` (#542)
|
||||
|
||||
`key: ci.gitea-milestone-filter-noop` · `status: active` · `since: 2026-07-21` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Never filter issues with the server-side `?milestones=<name>` parameter — fetch all open issues once and filter LOCALLY on each issue's `.milestone.title`.
|
||||
**Signals:** `GET /issues?milestones=` · milestone name filter · returns the whole open-issue list · `:` or `+` in a milestone title · mis-tiered issues · count looks like "all open issues" · paths: `scripts/select-queue.sh` · issues: #542, #77, #72
|
||||
**Mechanics:** `?state=open&type=issues&limit=50`, then filter in the client, e.g. `[i for i in issues if (i['milestone'] or {}).get('title') == NAME]`. `scripts/select-queue.sh` already does this; the trap is for anyone writing a *new* query.
|
||||
|
||||
The filter does not error on a name it cannot round-trip — it returns the **entire open-issue list** as
|
||||
though every issue matched. Milestone titles here routinely contain the offending characters (e.g.
|
||||
`Scheduling: refactor + distribution`), so the failure is the normal case, not an edge case.
|
||||
|
||||
Anything trusting that response mis-tiers issues: during #77 selection it made unmilestoned #72 look
|
||||
like a member of the milestone and hid #77's true sibling set. **The tell is a result count suspiciously
|
||||
equal to the total number of open issues** — if a filtered query returns everything, the filter silently
|
||||
failed; re-derive membership locally rather than reasoning about the result.
|
||||
|
||||
The dependencies API is unaffected: `POST /issues/{n}/dependencies` with `{"owner","repo","index"}` sets
|
||||
blocked-by correctly, though the bare `{index}` form returns 201 without reliably attaching — verify
|
||||
with the GET. (That last sentence is carried forward from an earlier revision of the handoff doc,
|
||||
commit `f93458c7`, where it was dropped by a later prune rather than disproved.)
|
||||
@@ -15,7 +15,7 @@ history in `docs/decisions/archive/startup.md`). Do not read it for current queu
|
||||
The ChicoryTV rewrite + go-live arc is **COMPLETE** and closed out. The project is now in
|
||||
**maintenance / backlog mode**. This is a *terminal* fact (the arc will not re-open), so it belongs
|
||||
in this standing file — unlike any "current frontier issue", which must never be written here (read
|
||||
it live; see the queue-drift lesson in the archived section below).
|
||||
it live — status embedded in prose always drifts).
|
||||
|
||||
So a fresh session stops re-discovering this every time:
|
||||
- The **arc / gate / open-`review` tiers are normally EMPTY now.** An empty arc frontier is the
|
||||
@@ -115,8 +115,8 @@ never orchestrator-tier work:
|
||||
`GET /issues/{n}/dependencies` on every candidate (dropping any with an OPEN blocker), tiers by
|
||||
LOCAL `.milestone.state`/`review`/`priority:` filters (never the server `?milestones=` name filter,
|
||||
which no-ops on `:`/`+`), and orders by (tier, priority, issue#). **Its DEPS/tiering/ordering are
|
||||
deterministic — trust them, do not re-derive them by hand** (see the archived section below for why
|
||||
that used to be necessary and isn't anymore). It also raises two JUDGMENT flags it deliberately does
|
||||
deterministic — trust them, do not re-derive them by hand** (the hand-derivation failure modes the
|
||||
script encodes are recorded at `ci.gitea-milestone-filter-noop`). It also raises two JUDGMENT flags it deliberately does
|
||||
not decide — `CLAIM?` (a claim can precede the label) and `UMBRELLA?` (an epic whose children are the
|
||||
real pickups) — resolve ONLY those by reading the flagged issue's comments/body. The script does not
|
||||
rank an arc tier (there is no active arc right now — maintenance/backlog mode, see above); if a new
|
||||
@@ -255,382 +255,100 @@ is both the human-readable summary and the per-issue unit MemPalace mines for re
|
||||
```
|
||||
|
||||
HARD CONSTRAINTS:
|
||||
- Work in worktrees off `origin/main` — ALWAYS, including for a one-line docs commit. `/Users/timothy/ersatztv`
|
||||
is a **shared mutable tree, not a `main` mirror**: never commit in it, and never read its HEAD/`git log`
|
||||
to learn anything about `main` (you'll get another session's branch). Its only sanctioned use is
|
||||
**read-only**: copying `web/node_modules` out of it. `origin/main` is the source of truth — `git fetch
|
||||
origin main` and reference it explicitly. See the lore bullet for the two sessions this bit on 2026-07-17.
|
||||
- PARALLELIZE BY DEFAULT when there are genuinely disjoint slices: use up to the client's concurrency
|
||||
limit, normally 2–3 delegated agents plus the orchestrator. Fan out bounded recon early, but do not
|
||||
invent artificial slices or exceed a runtime-imposed agent cap. At most 3–4 concurrent
|
||||
dotnet/npm builds are fine on this Mac (M4, 10 cores, 16 GB); the go/no-go signal is FREE
|
||||
RAM, not CPU load (`memory_pressure -Q`: <20% free → don't launch more build agents; <10% →
|
||||
pause/stagger). CPU load spikes during builds are benign. Never 5+ builds (the historic
|
||||
crash was RAM starvation from an 8–9-way fan-out). NEVER set ETV_UPDATE_GOLDENS.
|
||||
- Never two committing agents on ONE worktree — give each parallel slice its own worktree
|
||||
branched off the feature branch and merge back. Sequence only where a slice genuinely depends
|
||||
on another's output (backend-first narrow, SPA-wide after worked well for #216).
|
||||
- **BATCH YOUR PUSHES — you cannot cancel a CI run** (operator-requested, 2026-07-17). Hold review
|
||||
fixes, doc corrections and format fixes locally, verify, and push **once**; each push spawns a run
|
||||
that orphans the previous one on a 4-slot runner. Cancellation is impossible from the agent side on
|
||||
this Gitea (**1.25.4**): `POST /api/v1/.../actions/runs/{id}/cancel` → **404**, MCP
|
||||
`actions_run_write` `cancel_run` → **404**, and the web-UI route needs a session+CSRF that does not
|
||||
script (login 303s, no session cookie). **Only the operator can cancel, in the browser** — so if you
|
||||
must supersede a live run, SAY SO explicitly instead of leaving it burning. (`dispatch_workflow` is
|
||||
a different route and still works for re-triggering a **main** run.) Corrects the older "superseded
|
||||
runs drain on their own" note below: they do finish, but they hold a slot while doing it.
|
||||
- **A lone `decisions lifecycle` red is a KNOWN INFRA FLAKE — do NOTHING** (operator-stated,
|
||||
2026-07-19). When it is the **only** red: do not investigate it, and do **not** create a new run or
|
||||
commit to clear it — no rebase, no `--amend`, no no-op push. **The operator reruns that single job
|
||||
from the Gitea UI.** Report it as a known flake and carry on; only if *other* jobs are red too does
|
||||
the run deserve diagnosis. Same family as the killed-job rule below (a spurious single-job red is
|
||||
cleared by the operator's selective rerun, never by pushing — and pushes can't be cancelled, see the
|
||||
batching constraint above). **The trap is that a convincing local explanation is always available.**
|
||||
On #473/PR #479 the job went red just after `main` landed its own `decisions.md` entry, so "mine is
|
||||
no longer at EOF, I must rebase" looked airtight — I rebased, and it went red **again** on a head
|
||||
whose diff was a verified pure EOF append with zero deleted lines. A rebase that provably satisfied
|
||||
the gate's stated rule did not turn it green, which is the proof the red was never about the diff.
|
||||
**Check this rule before theorising about the cause.**
|
||||
- **Do NOT gate or throttle your push on host health — trust the Gitea build queue.** The runners were
|
||||
retuned for stability (operator, 2026-07-17); queueing is the queue's job, not yours. Don't SSH to
|
||||
bumblebee to sample load/RAM before pushing, and don't hand-schedule around other sessions' runs. Batch
|
||||
your pushes (above) because orphaned runs can't be cancelled — not because the host needs protecting.
|
||||
- **An infra-shaped red under host pressure is not a code failure — check the host before diagnosing the
|
||||
diff** (2026-07-17). While bumblebee was saturated, two unrelated jobs died in ways that look like code
|
||||
bugs: `EF migration integrity` failed in ~11ms inside **`Setup .NET`** with
|
||||
`remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and main's
|
||||
`Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after ~210s of
|
||||
retries. Both re-ran green once the host recovered; the migration job had no model change to test in the
|
||||
first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file a CI bug
|
||||
off one sample under pressure — you'll send the next session chasing a phantom (this nearly got filed as
|
||||
#390 lane-rebalance fallout, which the evidence did not support).
|
||||
- **Before any push touching `.cs`, check the touched set for a UTF-8 BOM.** The #311 gate is
|
||||
fix-as-you-touch: a legacy file's inherited BOM becomes *your* problem the moment you touch it, and
|
||||
`charset=utf-8` means no BOM.
|
||||
`for f in $(git diff --name-only origin/main...HEAD -- '*.cs'); do head -c3 "$f" | xxd -p | grep -q "^efbbbf" && echo "BOM: $f"; done`
|
||||
Strip any hit (1-line diff, zero content change). Bit **two sessions on 2026-07-17** — PR #405 (6
|
||||
files) and #70/PR #402 (19 files, where Python `io.open(..., encoding='utf-8-sig')` wrote BOMs back
|
||||
while "preserving existing style"). **Verify your detector**: an `od -An -c | grep '357 273 277'`
|
||||
check reported all-clean while 19 files were dirty — a detector that can only say "ok" is worse than
|
||||
none. Use the `xxd` form or read bytes directly.
|
||||
**`dotnet format --include` DOES work here** (corrected 2026-07-17, #70/PR #402 — an earlier note
|
||||
claiming it silently no-ops was WRONG). The apparent no-op was **the shell**: CI's recipe uses
|
||||
`mapfile`, which is **bash-only**, and the default shell here is **zsh** → empty array → zero files
|
||||
→ exit 0. Run it under `bash -c`:
|
||||
```bash
|
||||
bash -c 'mapfile -t files < <(git diff --name-only --diff-filter=ACM origin/main...HEAD -- "*.cs")
|
||||
dotnet format ErsatzTV.sln --no-restore --verify-no-changes --include "${files[@]}"' # exit 0 = gate passes
|
||||
```
|
||||
- Merge consent is DERIVED, not asked: the H6/H10 gate (`pretooluse-merge-consent.sh`) auto-grants a
|
||||
merge when the linked issue's `## Done-when` boxes are all ticked, a fresh positive `Review-verdict`
|
||||
references the current head, and CI is green — no conversational confirmation on that satisfied path
|
||||
(ersatztv#314). Confirm in-conversation only when the gate ASKS (state not derivable: no creds, no
|
||||
linked issue, no Done-when/verdict, Gitea down). Prior pre-approvals still do NOT carry over.
|
||||
- **Local gate + cold review run BEFORE the push to Gitea CI, not after** (Timothy, 2026-07-17):
|
||||
run the local build/test gate AND a cold-context adversarial review agent (SCOPED "review only")
|
||||
over the diff, fold any fixes, and only THEN push / open the PR. Gitea runners should only ever
|
||||
see already-validated work — CI is a scarce shared slow resource and a run cannot be cancelled on
|
||||
this Gitea, so spending a run to surface a lint error or a review-caught bug is pure waste + queue
|
||||
pressure. CI (armed at PR-open, below) *confirms*; it is not the first line of defence.
|
||||
- Arm a CI monitor on the PR head sha AT PR-OPEN (commit-status endpoint), not at the end.
|
||||
- Live-E2E via scripts/e2e-local.sh (also before the push for write-path / UI changes); NEVER
|
||||
exercise download endpoints via browser tabs (curl them).
|
||||
- **Work in a worktree off `origin/main` — always**, including a one-line docs commit. Never commit in
|
||||
`/Users/timothy/ersatztv` and never read its git state to learn about `main`. → `process.shared-tree-readonly`
|
||||
- **One committing agent per worktree.** Give each parallel slice its own; read-only review agents may
|
||||
share. → `process.one-worktree-one-committing-agent`
|
||||
- **Never commit or merge inside a worktree you did not create** — land on a sibling-owned branch with a
|
||||
plumbing merge. → `process.foreign-worktree-plumbing-merge`
|
||||
- **Parallelize by default** on genuinely disjoint slices (normally 2–3 delegated agents). Cap builds at
|
||||
3–4 and gate on FREE RAM, not CPU. **Never set `ETV_UPDATE_GOLDENS`.** → `process.build-concurrency-limits`
|
||||
- **Local gate + cold-context review BEFORE the push**, never after. → `process.local-gate-before-push`
|
||||
- **Independent review is mandatory** for locks/concurrency, auth/security, API write-path handlers, DB
|
||||
migrations, or >~150 changed C# lines; a skip must be stated with its reason. → `process.independent-review-rubric`
|
||||
- **Batch your pushes — you cannot cancel a CI run.** Only the operator can cancel, in the browser.
|
||||
→ `ci.batch-pushes-no-cancel-route`
|
||||
- **Do not gate your push on host health** — trust the build queue. → `ci.no-host-health-gating`
|
||||
- **Arm the CI monitor at PR-open**, on the head sha. → `ci.monitor-armed-at-pr-open`
|
||||
- **Before any push touching `.cs`, BOM-check the touched set** and run the format gate under `bash -c`.
|
||||
→ `process.bom-format-detection-recipe`
|
||||
- **Keep a PR branch current by REBASING on `origin/main`** — never merge main in; regenerate (never
|
||||
hand-resolve) generated artifacts after a conflicted rebase. → `release.format-as-you-touch-rebase`
|
||||
- **Live-E2E via `scripts/e2e-local.sh` before the push** for write-path/UI changes; curl download
|
||||
endpoints, never browser tabs; use a fresh config dir.
|
||||
→ `testing.live-e2e-prepush-timing`, `testing.e2e-local-fresh-config-dir`
|
||||
- **Merge consent is DERIVED, not asked.** The H6/H10 gate auto-grants when the `## Done-when` boxes are
|
||||
ticked, a fresh `Review-verdict` references the current head, and CI is green; confirm in conversation
|
||||
only when the gate ASKS. Prior pre-approvals never carry over. → `release.merge-consent-autogrant`
|
||||
- **After the push, fixes are follow-up commits** — never amend or force-push a pushed branch.
|
||||
→ `process.pr-routine-sequence`
|
||||
|
||||
---
|
||||
|
||||
# Lessons / workflow lore (STANDING workflow/orchestration rules only)
|
||||
|
||||
> **Scope**: this section is *how we work* (orchestration, CI, review, cross-session hygiene) —
|
||||
> not engineering decisions. **Every "why the code does X" decision lives in `docs/decisions.md`**
|
||||
> (+ `api-conventions.md` §7a/b/c for the concurrency/side-effect mechanics, `spa-conventions.md`,
|
||||
> `e2e-local.md`). Do NOT re-record a code/API/SPA decision here — it will duplicate and drift.
|
||||
> Pruned 2026-07-12: issue-specific engineering bullets now covered in those docs, and dead
|
||||
> Blazor-parity process notes, were removed (git history has them). Pruned 2026-07-21: pre-script
|
||||
> selector lore and the #237-specific queue-drift narrative moved to the archived section below —
|
||||
> the mechanism they describe no longer exists.
|
||||
> **Scope**: *how we work* — orchestration, CI triage, review routing, cross-session hygiene. Each rule
|
||||
> below is one or two lines plus its decision `key:`; the evidence, the incident that produced it and the
|
||||
> full rationale live in that record (`docs/decisions/workflow-process.md` for most of them). **Keep it
|
||||
> that way** — this file is pasted into every session's kickoff, so a paragraph of narrative here is a tax
|
||||
> paid by every future session, while a record is retrieved only when it is needed (#542).
|
||||
>
|
||||
> Editing: prune covered and stale bullets rather than appending — this is not append-only and git keeps
|
||||
> the history. If you add a rule, write the record first and cite it here.
|
||||
|
||||
- READ docs/README.md → the convention docs FIRST; point recon/implementer agents at specific
|
||||
doc sections. Only recon the task-specific delta.
|
||||
- **The mechanical selection checks live in `scripts/select-queue.sh` — run it instead of
|
||||
re-deriving them by hand.** It does the deterministic work in code (dependency exclusion via
|
||||
`GET /dependencies`, LOCAL `.milestone.state`/`review`/`priority:` tiering, (tier, priority, issue#)
|
||||
ordering, `in-progress`/`parked`/PR exclusion) and only *flags* `CLAIM?`/`UMBRELLA?` for judgment.
|
||||
**Trust its DEPS/tiering/ordering; do the focused recheck only on the flagged rows.** If you extend
|
||||
the tiers/filters, update the script first, then these bullets. (See the archived section below for
|
||||
the pre-script failure modes that motivated writing it, if you need the history.)
|
||||
- **Codex cheap-worker launch (tested 2026-07-14)**: the native `spawn_agent` interface currently
|
||||
has no model/effort selector, so it provides parallelism but not cost savings. For bounded,
|
||||
tool-bearing selector/recon work, launch a separate worker with
|
||||
`codex exec --ephemeral --color never -m gpt-5.4-mini -c model_reasoning_effort=low -s read-only
|
||||
-C <repo> "<hard-capped scope contract>"`. GPT-5.4-Mini is the current supported small/cost-efficient
|
||||
profile on this ChatGPT-authenticated host; the older `gpt-5.1-codex-mini` guess fails with HTTP 400.
|
||||
Keep `low` whenever any shell/MCP/web tool is exposed; reserve `minimal` for explicitly tool-free
|
||||
synthesis over already-supplied bounded evidence. If the model rolls over, inspect the visible entries
|
||||
in `~/.codex/models_cache.json` instead of guessing names, then update this dated note once verified.
|
||||
- Keep the MAIN checkout's `web/node_modules` fresh (`npm install` after pulling a PR that
|
||||
adds a dep) — worktrees copy it, and a stale copy broke typecheck in a #198 worktree.
|
||||
- **Playwright-MCP E2E: never open tabs/window.open for file-download endpoints** — curl them.
|
||||
If browser tools stall repeatedly, `pkill -f ms-playwright-mcp` and drive a fresh session.
|
||||
- **THERE IS NO "main checkout". `/Users/timothy/ersatztv` is a shared mutable working tree whose HEAD
|
||||
is whatever the last session left there — it is NOT a `main` mirror, and its name lies.** Treat it as
|
||||
**read-only, and never as a source of truth**:
|
||||
- **Never commit in it** — not even a one-line docs fix. Need a commit on `main`? Make a worktree off
|
||||
`origin/main` (`git worktree add -b <branch> <path> origin/main`) and commit there.
|
||||
- **Never read `git log` / `git status` / `HEAD` there and infer anything about `main`** — you will be
|
||||
reading some other session's branch.
|
||||
- **`origin/main` is the only source of truth.** `git fetch origin main` and reference `origin/main`
|
||||
explicitly. Do not trust the local `main` ref either (it lags; nobody is obliged to update it).
|
||||
- Its **one sanctioned use is read-only**: copying `web/node_modules` into your worktree (keep it
|
||||
fresh — a stale copy broke typecheck in a #198 worktree).
|
||||
**Orientation**
|
||||
- Read `docs/README.md` → the convention docs first; point agents at named sections, recon only the
|
||||
task-specific delta. → `docs.convention-docs-session-start`
|
||||
- Run `scripts/select-queue.sh` for queue selection; trust its deps/tiering/ordering and resolve only its
|
||||
`CLAIM?`/`UMBRELLA?` flags. → `startup.parallel-orientation`
|
||||
- Claim with `in-progress` before working — but a claim prevents duplicate *pickup*, not overlapping code
|
||||
changes. → `process.parallel-session-claim`
|
||||
|
||||
**This is a design flaw, not a discipline failure — do not "fix" it by telling sessions to check
|
||||
harder.** A check does not stay true. Worktree discipline is healthy (10 feature worktrees on
|
||||
2026-07-17: Claude under `.claude/worktrees/`, Codex under `.worktrees/`, some in `/private/tmp/`);
|
||||
the shared tree is the single place the model breaks, and it broke for **two different sessions on the
|
||||
same day, both doing the obvious thing**:
|
||||
- Session A assumed it was on `main` (true at 12:46 when it pulled — **correct when made**, stale by
|
||||
14:17) and committed → the commit landed on the #604/CI session's `ci/604-lane-rebalance`, **24
|
||||
seconds after that session's own commit at 14:17:28**. Two writers, one tree. Nothing tangled, by luck.
|
||||
- Session B ran `git log` there, read `ci/604-lane-rebalance`'s HEAD, and concluded `main` was "4
|
||||
behind origin" — a phantom.
|
||||
**Building and reviewing**
|
||||
- The PR routine is a fixed sequence; for API changes build the app project FIRST, then
|
||||
`./scripts/update-openapi.sh`, then `npm run generate:api`. → `process.pr-routine-sequence`
|
||||
- Re-review the FIX COMMIT, not just the initial diff; loop until a clean verdict or an explicit
|
||||
acceptable-defer, and file a tracked issue for anything deferred. → `release.review-verdict-gate`
|
||||
- When independent reviews disagree, escalate to the frontier judge — and escalate the proposed FIX with
|
||||
the finding. → `process.review-disagreement-frontier-judge`
|
||||
- "Make X consistent with Y" findings are new code; for any timer/effect ask when it fires, including on
|
||||
mount. → `process.consistency-fix-new-code-scrutiny`
|
||||
- Before trusting a "single owner / no cross-release" claim, grep the whole host project for every
|
||||
producer — the background scheduler is the usual miss. → `process.lock-ownership-enumerate-producers`
|
||||
- Before deleting a "workaround", enumerate every behavior it provided.
|
||||
→ `process.enumerate-workaround-behaviors-before-deleting`
|
||||
- Concurrency tests need a stress loop and a negative control that inverts the real condition — never
|
||||
`if (true)`. → `ffmpeg.work-ahead-slot-atomic`
|
||||
- Enumerate lazy LanguageExt returns in tests; live-E2E is the only net for that class.
|
||||
→ `release.live-e2e-required`
|
||||
- Harden a soon-to-grow feature with a runtime posture, not a hardcoded clamp.
|
||||
→ `process.harden-with-runtime-posture-not-clamp`
|
||||
|
||||
A session-start `git status` *appears to confirm* the false assumption, which makes it worse than a
|
||||
plainly wrong claim. (Supersedes the old "the user's main checkout stays ON main and pulled each
|
||||
session" note.) Same family as the sibling bullet below: never commit inside a checkout or worktree
|
||||
you didn't create.
|
||||
**Reading a red CI run** (check these before diagnosing your diff)
|
||||
- A **killed** job reports `conclusion: failure` — read the log tail for the `❌ Failure - Main` marker;
|
||||
log timestamps are UTC, the host is UTC+2. → `ci.killed-job-triage`
|
||||
- **`cancelled` is not a verdict** — report FAILED and CANCELLED separately. → `ci.cancelled-is-not-a-verdict`
|
||||
- A failure inside a **setup/cache step**, before your code compiles, is environmental.
|
||||
→ `ci.infra-shaped-red-under-load`
|
||||
- A lone **`decisions lifecycle`** red is a known flake: do **nothing**, the operator reruns it. A
|
||||
convincing local explanation is always available — check this rule before theorising.
|
||||
→ `ci.decisions-lifecycle-flake`
|
||||
- Heavy-render web tests need explicit per-test vitest timeouts. → `ci.web-test-per-test-timeouts`
|
||||
- Verify locally and trust it; CI confirms (~9 min warm, seconds for docs-only).
|
||||
→ `ci.verify-locally-ci-confirms`
|
||||
|
||||
**Also treat any FILE read out of that tree as stale** — including a kickoff prompt pasted from it.
|
||||
The guards above only fire when you run `git` there; a stale checkout serves stale docs silently.
|
||||
Session end runs `scripts/refresh-shared-checkout.sh` to keep it current (see
|
||||
`session.shared-checkout-refresh` in decisions.md).
|
||||
- Subagent connection drops = laptop sleep/transient; re-resume via SendMessage (work survives).
|
||||
- Build/verify (and review) locally, then trust it; CI (VM 127) confirms (~9 min warm; seconds for
|
||||
a docs-only PR since ersatztv#416). Watch by commit
|
||||
status: `/api/v1/repos/timothy/ersatztv/commits/{sha}/status`.
|
||||
- PR routine (works): worktree off origin/main → implement (strongest tier for judgment-heavy /
|
||||
balanced tier for mechanical work when model routing is available; doc pointers + exact facts;
|
||||
use the available C# language tooling) → API changes: build app project
|
||||
FIRST then `./scripts/update-openapi.sh` then `npm run generate:api` → **full local test pass +
|
||||
cold-context adversarial review (SCOPED "review only") + live-E2E for UI/write-path changes
|
||||
(`scripts/e2e-local.sh`), ALL BEFORE the push** — fold pre-push review fixes into the branch
|
||||
(amend/reword is fine while nothing is pushed) → push, PR (arm the CI monitor at open) → CI
|
||||
*confirms* → any further review fixes (a post-push cross-model pass, or CI-surfaced issues) as
|
||||
follow-up COMMITS (never amend/force-push a pushed branch) → CI green → merge (derived consent
|
||||
gate) → structured close comments per the applicable `AGENTS.md` / `CLAUDE.md` protocol.
|
||||
- Never run TWO committing agents on one worktree concurrently. Read-only review agents are fine.
|
||||
- **A KILLED job reports `conclusion: failure` — the conclusion field is not enough. Read the log tail.**
|
||||
(2026-07-17.) When the operator restarts the runners (a retune, a reboot), in-flight jobs die and Gitea
|
||||
marks them **`failure`**, not `cancelled`. Tell: the log **stops mid-step with no error and no
|
||||
`❌ Failure - Main …` marker** — a real failure always leaves a marker. Run 1006's `EF migration
|
||||
integrity` + `Functional E2E` "failed" on a **BOM-removal-only** diff (6 files, 1 line each, zero
|
||||
content change) that could not possibly break them; the logs simply truncated mid-`dotnet build` at
|
||||
12:08 **UTC**, and both runners showed `Up About an hour` — the retune had killed them. **Log
|
||||
timestamps are UTC; the host is UTC+2** — convert before correlating, or the restart looks 2h off and
|
||||
the theory gets wrongly discarded.
|
||||
Corollary: **a diff that cannot cause a failure is evidence the failure isn't yours** — when a job that
|
||||
passed on the previous head fails on a semantically-null delta, suspect the environment and go read the
|
||||
log, don't re-litigate the diff. Also beware log **warnings that look fatal**: `Unable to pull
|
||||
refs/heads/v4: …` is act refreshing its `/root/.cache/act` action cache and is followed by `Cloned …`
|
||||
— it is NOISE, not a cause. Grep for the failure marker, not for the word "error".
|
||||
- **`cancelled` is NOT `failure` — never read a cancel as a CI verdict** (2026-07-17, operator-surfaced).
|
||||
The operator cancels runs by hand (they're the only one who can — see the batching constraint above), and a
|
||||
run-level `conclusion: cancelled` means **no verdict**, not a pass and not a fail. Two traps: (1) a
|
||||
run whose overall state is `failure` may hold a *genuine* job failure that happened **before** the
|
||||
cancel — check job-level `conclusion` + timestamps, don't attribute it to the cancel; (2) a
|
||||
`cancelled` run tells you nothing about your diff, so don't claim green on it. Report FAILED and
|
||||
CANCELLED counts **separately** in any CI monitor:
|
||||
`fail=[j for j in jobs if j['conclusion']=='failure']; canc=[j for j in jobs if j['conclusion']=='cancelled']`
|
||||
— a monitor that only asks "is state != pending" will call a cancelled run a failure and send you
|
||||
debugging a phantom.
|
||||
- **An infra-shaped red under host load is not a code failure — check `uptime` on bumblebee BEFORE
|
||||
diagnosing the diff** (2026-07-17). At load average **243**, two unrelated jobs died in ways that
|
||||
look like code bugs but aren't: `EF migration integrity` failed in ~11ms inside **`Setup .NET`**
|
||||
with `remove /root/.cache/act/…/.gitignore: no such file or directory` (act's shared cache), and
|
||||
main's `Build & push image` died on buildx `DeadlineExceeded: failed to compute cache key` after
|
||||
~210s of retries. Both re-ran green at normal load; the migration job had no model change to test in
|
||||
the first place. Tell: the failure is in a *setup/cache* step, before your code compiles. Don't file
|
||||
a CI bug off one sample under load — you'll send the next session chasing a phantom (this nearly got
|
||||
filed as #390 lane-rebalance fallout, which the evidence did not support). A game server
|
||||
(`FactoryServer-L`) shares that host with the runners, so load isn't always CI's doing.
|
||||
- **Arm a CI monitor the moment the PR opens** (commit-status endpoint), not at the end — in the
|
||||
#216 session three PR runs sat red ~an hour on a CI-only flake while review/E2E ran. (The old
|
||||
MySQL host-port 3306 collision is fixed on main, `ef8915f1`; a lone MySQL-apply red now = the
|
||||
known infra flake, rerun — see MEMORY / `ci-migration-job-mysql-flake`.)
|
||||
- **"Make X consistent with Y" review findings deserve new-code scrutiny** — a #216 review fix
|
||||
propagated MediaBrowseScreen's mount-debounce selection-wipe race to SearchScreen instead of
|
||||
catching it. For any timer/effect, ask "when does this fire?" (incl. mount).
|
||||
- Live-E2E seeding recipe now lives in `docs/e2e-local.md` → "Seeding a local TV library for E2E"
|
||||
(tiny ffmpeg testsrc MKVs + `LibraryPath` SQL rows + scan). Live-E2E is a **stated requirement**
|
||||
for write-path handler changes — see that doc's "When live-E2E is required" + the decisions.md
|
||||
entry; it's the only net for the lazy-enumeration / reload-through-read-path 500 class (#229).
|
||||
- **`scripts/e2e-local.sh` hangs its readiness probe on a REUSED config dir** (surfaced in the #502
|
||||
session; recovered from #237's comment log by the #524 triage). The probe waits for
|
||||
`Done migrating search index` (`e2e-local.sh` `READY_LINE`), which a **fresh** config emits — a
|
||||
reused one logs `Search index is already version 50` instead and the line never comes, so the
|
||||
script times out and kills a perfectly healthy server. `docs/e2e-local.md` documents the ready line
|
||||
and recommends a fresh dir, but not this failure mode. **Use a fresh config dir** — the script
|
||||
greps one fixed `READY_LINE`, so there is no "wait for either line" option today (widening it to
|
||||
`Done migrating search index|Search index is already version` is tracked as #533). Cost that
|
||||
session two failed launches before the cause was obvious.
|
||||
- **The troubleshooting-playback API cannot exercise channel branding** (same origin).
|
||||
`PrepareTroubleshootingPlaybackHandler` builds a synthetic channel with `Artwork = []`, and its
|
||||
`ChannelId > 0` branch ignores the media item — so logo/watermark/bug behavior is **not** testable
|
||||
through the troubleshoot path. Verify branding changes through a real channel playout instead;
|
||||
a green troubleshoot run proves nothing about the bug overlay.
|
||||
- **Parallel sessions**: claim before working (`in-progress` label — the tiny read→claim race window
|
||||
is accepted; later claimant backs off). Claiming prevents duplicate pickup, NOT overlapping code
|
||||
changes — check the issue's dependency notes ("#234 after #231", "coordinate with #215") before
|
||||
touching shared surfaces. Editing THIS lore: prune covered/stale bullets too (not append-only — git
|
||||
keeps history), `git pull --rebase` before commit. Two runners (ci-runner VM 127 + bumblebee-runner,
|
||||
4 slots); **no cancel route exists on this Gitea version — verified 1.25.4: REST and MCP
|
||||
`cancel_run` both 404** (see the batching constraint above). Superseded runs do finish on their own,
|
||||
but they hold a runner slot while doing it, so with 4 parallel sessions they are a real cost — batch
|
||||
pushes rather than relying on them to drain.
|
||||
- **Two sessions touching one machine**: a branch may be checked out in ANOTHER session's
|
||||
worktree — never commit/merge inside a worktree you didn't create. To land a merge on such a
|
||||
branch without touching their checkout: plumbing merge (`git read-tree -m base ours theirs`
|
||||
into a temp GIT_INDEX_FILE → `write-tree` → `commit-tree -p ours -p theirs` → push the
|
||||
commit to the branch ref); the owning session then `git pull`s.
|
||||
- **Cross-model independent review rotation** (2026-07-11): use the other client/model family when
|
||||
it is available — Codex reviews Claude-authored changes; Claude reviews Codex-authored changes.
|
||||
Use whatever native integration is actually installed; client-specific helpers such as
|
||||
`codex:rescue` are optional accelerators, not protocol dependencies. On quota/tool exhaustion,
|
||||
use a cold-context review-only agent in the current client and retry cross-model review next window.
|
||||
Independent reviewers do review and stuck-diagnosis work, not implementation on the same change.
|
||||
**Skip rubric (self-exemption is an auditable claim, not a vibe)**: an *independent* review pass
|
||||
(preferably cross-model; otherwise a cold agent) is **mandatory** for any diff touching
|
||||
locks/concurrency, auth/security, API write-path handlers, or DB migrations, or larger than
|
||||
~150 changed C# lines. It is **skippable** only for a pure-SPA/docs leaf with no server-state
|
||||
effect — and when you skip it, **say so and why** in the PR or close comment (a stated exemption,
|
||||
not silence). This is the correlated-blindspot net: independent model families miss *different* classes
|
||||
(see the #235 "enumerate every producer" and #250 lessons), so a single reviewer on a risky diff
|
||||
is the gap.
|
||||
- **Force enumeration of lazy LanguageExt returns in tests** (2026-07-11, PR #229): a write-path 500
|
||||
survived a green fixed-point test because the handler returned a lazy `Map` the test never enumerated.
|
||||
Enumerate handler return values in tests; live E2E remains the only net for this class. (Write-path
|
||||
reload-through-read-path mechanics: api-conventions §7.)
|
||||
- **Keep a PR branch current by REBASING on origin/main — do NOT merge main in** (ersatztv#311,
|
||||
enforced by the H11 pre-push hook: a push from a branch behind origin/main is blocked → `git fetch
|
||||
origin main && git rebase origin/main`). A *merge* commit pulls in every file main changed, incl.
|
||||
files you never touched (e.g. the ~2500 legacy-BOM `.cs`), which then trip the format hook/CI on code
|
||||
that isn't yours (the #309 session). Rebasing keeps your diff to exactly what you changed. After a
|
||||
rebase that hits conflicts in **generated artifacts** (v1.json, v1.d.ts, endpoint-index), REGENERATE
|
||||
them (`./scripts/update-openapi.sh` + `npm run generate:api`) — never hand-resolve; git text-merges
|
||||
them plausibly-but-wrong and `npm run check:api` is the guard. Escape hatch for a deliberate
|
||||
non-rebased push: `ETV_SKIP_REBASE_CHECK=1 git push`.
|
||||
- **H12 issue-qualification audit** (ersatztv#312, `scripts/issue-qualification-audit.sh`): a
|
||||
session-end check that lists OPEN issues missing a `priority:` label — the queue tiering keys off
|
||||
`priority:`/gate labels, so an unlabeled issue is invisible to it. "Fully qualified" = has a
|
||||
`priority: {high,medium,low}` label (that signals triage ran; gate-vs-backlog is then derivable
|
||||
from the `review` label / milestone, and a milestone is NOT required — backlog is unmilestoned).
|
||||
Run it at session end and label anything it flags (esp. issues you filed this session). Fail-open
|
||||
without creds; advisory (exit 1 when any are unqualified). Sibling to H11 (both #311/#312).
|
||||
- **CI VM test timeouts**: heavy-render web tests (100+ item grids) need explicit vitest
|
||||
timeouts (e.g. 15s) — the CI VM hit the 5s default on a test that runs in ~1s locally
|
||||
(run 686). Bump per-test, don't raise the global default.
|
||||
- **Concurrency tests need a negative control + a stress loop, not a single round** (2026-07-11,
|
||||
#231): a one-shot `Barrier(N)` + `Task.WhenAll` "exactly one winner" test did NOT catch a
|
||||
deliberately non-atomic check-then-set flag on this Mac — the load→store window is too narrow to
|
||||
collide when threads release once. It green-lit broken code. Fix: hammer the race over many rounds
|
||||
(N worker threads × ~20k rounds, two Barriers per round: acquire → validate one-winner → reset),
|
||||
and ALWAYS prove non-vacuous by temporarily breaking the primitive and watching the test fail.
|
||||
Cheap (sub-second) and deterministic-enough. Same discipline as #12's negative control.
|
||||
- **Before deleting a "workaround", check it isn't serving a SECOND purpose** (2026-07-11, #232):
|
||||
an issue said "remove the SPA `PENDING_GRACE_TICKS` grace-tick workaround now the API is honest".
|
||||
The grace ticks did double duty — compensating the lying-200 (now genuinely obsolete) AND
|
||||
absorbing the inherent queue→observed-active lag / fast-completion race (still needed). Removing
|
||||
wholesale wedged the scan button disabled-until-reload for short scans; both fork + Codex caught
|
||||
it. Keep a bounded version, re-scoped honestly. When an issue says "delete X", enumerate every
|
||||
behavior X provided before deleting.
|
||||
- **Re-review the FIX COMMIT, not just the initial PR diff, before closing** (2026-07-11, #242,
|
||||
user-surfaced): the pre-fix review is only half the loop — the fix commit is new code no
|
||||
independent reviewer has seen. Here the first fix (gate Edit on `saving`) only PARTIALLY closed
|
||||
the in-flight-save clobber class; a Codex re-review of the fix commit caught that **Delete** was
|
||||
the one schedule-switch path (`applySwitch`) still missing the guard. Fold a scoped re-review
|
||||
(cross-model if quota, else a cold review agent) over the fix commit into the close protocol; loop until a clean
|
||||
MERGEABLE verdict or an explicit acceptable-defer (file a tracked issue for the deferral, as #248
|
||||
here). Cheap relative to shipping a half-fix that reopens the issue.
|
||||
- **Negative-controlling a lock/flag fix: invert the condition, don't stub `if (true)`** (2026-07-11,
|
||||
#250): to prove a lock-release test is non-vacuous I temporarily broke the fix. First attempt made
|
||||
the guard `if (true)` — but that left the ownership flag "assigned but never read" → **CS0219 under
|
||||
warnings-as-errors → the Application build silently FAILED → `dotnet test --no-build` ran the STALE
|
||||
(fixed) dll → false PASS** that looked like the control had failed to catch the bug. The tell: build
|
||||
output must show `Build succeeded` / `0 Error(s)`, not just a truncated tail. Break the primitive by
|
||||
**inverting** the real condition (`if (!lockAcquired)`) so the flag stays read and the code compiles;
|
||||
always grep the build for `error CS` before trusting a `--no-build` test result. (Codex-found High
|
||||
this session: an outer `catch` that unlocked unconditionally cross-released another session's lock on
|
||||
a pre-acquisition exception — the fix was a Handle-scoped ownership flag gating the catch.)
|
||||
- **A lock/channel-ownership "no cross-release" verdict must enumerate EVERY producer of that message
|
||||
type, not just the enqueuers visible in the PR diff** (2026-07-11, #235/#267). The cold fork cleared the
|
||||
new `scan-collections` per-provider lock as sound — "only the acquirer enqueues, so only its message
|
||||
triggers the scanner unlock" — because it checked the three new controllers but never grepped for ALL
|
||||
`Synchronize{X}Collections(` senders. Codex did: `SchedulerService` also enqueues them *periodically and
|
||||
unlocked*, and `ScannerService`'s `finally` released the lock whenever held → a scheduled scan
|
||||
cross-releases an API scan's lock (#250 class). Rule: before trusting any "single owner / no double
|
||||
release / no cross-release" claim, `grep` the whole host project for every writer of that channel message
|
||||
(or acquirer of that lock) — the background scheduler/worker is the usual missing producer. This is why the
|
||||
independent Codex pass earns its keep alongside the fork even when the fork returns MERGEABLE.
|
||||
- **Independent-review disagreement on a gate PR → frontier judge; judge the fix, not just the verdict**
|
||||
(2026-07-11, #235): the reconciliation isn't only "who's right" — Fable historically ratified Codex's High AND caught
|
||||
that the orchestrator's proposed *fix* (per-source lock-or-skip) would deterministically starve the 2nd+
|
||||
media source (fixed loop order), prescribing the shipped library-scan §C7 pattern instead (lock once per
|
||||
provider, batch-enqueue, release on the `Unlock: isLast` message). Escalate the FIX approach too, not just
|
||||
the finding.
|
||||
- **Harden a soon-to-grow feature with a runtime posture/toggle, not a hardcoded clamp** (2026-07-12, #289):
|
||||
the MCP's final design (#58) is read+write, so a GET-clamp would be ripped out — shipped a runtime read-only
|
||||
posture (`ERSATZTV_ALLOW_WRITES`, default false) instead. When a security fix constrains a capability the
|
||||
roadmap will later want, make the safe state the *default of a switch*, not a wall; check the feature's issue
|
||||
for its end-state first.
|
||||
- **Fixing onto an unmerged feature branch: branch off it, land by FF-push, never touch a sibling worktree that
|
||||
has it checked out** (2026-07-12, #289 → PR#76). Gotcha: recon by `Read`ing a *sibling* worktree leaked its
|
||||
paths into my later `Edit`s → they landed there, not the fresh worktree (`dotnet` built the unchanged tree →
|
||||
"0 new tests" tipped it off). After creating a worktree, do the first Edit/Read against ITS absolute paths and
|
||||
`git status` it before building.
|
||||
- **`HttpCompletionOption.ResponseHeadersRead` moves the body read OUTSIDE `HttpClient.Timeout`** (2026-07-12,
|
||||
#289): with a `None` read token a slow-drip upstream hangs forever. Fix = per-request linked
|
||||
`CancellationTokenSource.CancelAfter(timeout)` into BOTH `SendAsync` and the stream reads +
|
||||
`HttpClient.Timeout = InfiniteTimeSpan`; catch transport/timeout exceptions and turn them into a response.
|
||||
(Not yet in decisions.md — #289 landed on PR#76, not main.)
|
||||
**Tooling gotchas**
|
||||
- Playwright-MCP: curl download endpoints, never open a tab; `pkill -f ms-playwright-mcp` on repeated
|
||||
stalls. → `testing.playwright-mcp-download-and-recovery`
|
||||
- Channel branding is NOT testable through the troubleshooting-playback API.
|
||||
→ `testing.troubleshoot-path-cannot-test-branding`
|
||||
- Codex cheap workers launch via `codex exec`, not `spawn_agent`. → `process.codex-cheap-worker-launch`
|
||||
- A dropped subagent connection is transient — resume via SendMessage, don't restart.
|
||||
→ `process.subagent-drop-resume`
|
||||
- Fixing onto an unmerged feature branch: branch off it, FF-push, and beware the sibling-worktree path
|
||||
leak. → `process.branch-off-feature-branch`
|
||||
|
||||
---
|
||||
|
||||
## Archived — pre-script selector history (do not follow)
|
||||
|
||||
> This section is history, not instruction. It explains why `scripts/select-queue.sh` exists and
|
||||
> why the closed #237 tracker used to be the queue-state store; none of it is a live protocol. The
|
||||
> current protocol is "Two concurrent tracks at session start" above.
|
||||
|
||||
- **(Pre-2026-07-19) Before the selector script existed**, a cheap model was dispatched each session
|
||||
to rank the backlog by reading tracker prose directly, and the orchestrator was told to RE-DERIVE
|
||||
its mechanical claims because the model kept getting them wrong: it reported blocked issues as
|
||||
`deps:clear` (never called `/dependencies`), mis-tiered issues when the server-side `?milestones=`
|
||||
filter silently no-ops on `:`/`+` names, ranked by issue number and ignored priority-within-tier,
|
||||
and missed live `in-progress` claims. `scripts/select-queue.sh` was written to do the mechanical
|
||||
parts deterministically in code instead — the evidence bullets below are kept for context, not
|
||||
because you need to re-derive anything by hand anymore.
|
||||
- **Gitea milestone-name issue filter silently NO-OPS on names with special chars (2026-07-17,
|
||||
#77 selection)**: `GET /issues?milestones=<name>` returns the WHOLE open-issue list — not a
|
||||
filtered set — when the milestone title contains `:` or `+` (e.g. `Scheduling: refactor +
|
||||
distribution`), because the raw name doesn't round-trip through Gitea's filter. A selector that
|
||||
trusts that response will mis-tier issues (it made #72 look like a milestone-10 member when it's
|
||||
unmilestoned, and hid #77's true sibling set). Robust recipe (now encoded in the script): fetch
|
||||
all open issues once and filter LOCALLY on each issue's `.milestone.title`.
|
||||
- **Confirmed again + widened (2026-07-17, #72/#164 selection): the packet does not reliably see
|
||||
`blocked-by`, priority-within-tier, `in-progress` claim state, or umbrella-vs-child** — that run's
|
||||
dependency data was correct, yet it still ranked #383 first while missing that #384 was
|
||||
`in-progress` (claimed 40 minutes earlier) and that #383 is that bundle's umbrella. The script's
|
||||
`CLAIM?`/`UMBRELLA?` flags now surface exactly this instead of relying on a full manual re-check.
|
||||
- **The cheap selector's dependency + priority data was NOT trustworthy — the fix was to make the
|
||||
script compute them, not to re-derive the whole contested tier by hand every session** (2026-07-17,
|
||||
#73 selection): the old packet reported #385/#386 as `deps:clear` when both were blocked, and
|
||||
missed #383 entirely (the only `priority: medium` in tier 1). The script now runs
|
||||
`GET /issues/{n}/dependencies` on every surviving candidate and tiers by priority-within-tier
|
||||
deterministically, closing this class.
|
||||
- **Queue-drift root cause (2026-07-12, user-surfaced, historical)**: pickups repeatedly re-picked
|
||||
already-done work because DONE/OPEN status was read from **prose** (the old tracker's arc body
|
||||
inline markers + session comments' "Recommended next"), which lags real issue state — worst under
|
||||
parallel sessions narrating each other's merges. STANDING FIX (still the rule, just no longer tied
|
||||
to any specific tracker): **live Gitea state is the ONLY source of truth for status**; a comment's
|
||||
"Recommended next" is a candidate that MUST be re-verified OPEN before pickup, never propagated
|
||||
when stale. Do not name today's frontier issue in this standing file — that's how the drift
|
||||
started the first time.
|
||||
**Session end**
|
||||
- Run the H12 audit and label everything it flags. → `process.issue-qualification-audit`
|
||||
- Run `scripts/refresh-shared-checkout.sh` so the next session's checkout is current.
|
||||
→ `session.shared-checkout-refresh`
|
||||
|
||||
@@ -291,7 +291,12 @@ def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--base")
|
||||
ap.add_argument("--head")
|
||||
ap.add_argument("--budget", type=int, default=4800) # aggregate; re-baselined post-#521 migration
|
||||
# Aggregate active-corpus budget. Re-baselined 2026-07-21 (#542): 4800 -> 5600. The corpus grew
|
||||
# by ~530 lines because 31 workflow/CI/review rules were MOVED into it from
|
||||
# docs/handoffs/chicorytv-issue-queue.md, which was their only copy. That is the corpus doing its
|
||||
# job, not drift — the knowledge existed already, it just wasn't retrievable. Do not raise this
|
||||
# again to accommodate genuinely new records; that is what the warning is for.
|
||||
ap.add_argument("--budget", type=int, default=5600)
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
records = dl.all_active_records()
|
||||
|
||||
Reference in New Issue
Block a user