Files
ersatztv/docs/decisions/records/testing/e2e-cleanup-scope-by-pid.md
T
timothy fba5233caf
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Failing after 23s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 1m17s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m29s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m5s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 16m5s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m6s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
feat(610): split the decision corpus into one YAML-frontmatter file per record
168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and
docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key,
so one-active-record-per-key becomes a filesystem property rather than a
validator check, and supersession becomes a `git mv`.

WHY: the monolith was a concurrency problem before an aesthetic one. A
3,900-line append target made parallel sessions collide -- PR #605 and PR #614
both hit append-vs-append conflicts during routine rebases, and hand-resolving
those inside the corpus is exactly the operation the rationale-rewrite guard
exists to police.

HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness
does not rest on reading it. The parser was taught BOTH formats first, so the
body-diff guard parses the old form at the merge-base and the new form at head --
the migration validates itself, no bypass. The proof is a field-level equivalence
harness: 168 records before and after, zero lost, zero gained, zero field
mismatches, zero rationale bodies differing. Reviewers should scrutinise the
harness; it is the actual evidence.

What measuring caught that reading would not have:

- ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each
  topic file's preamble, mostly the only copy. Source files are kept and
  stripped, never deleted. They also cannot be filed per-area: topic files hold
  several areas and 4 of 23 areas span several files.
- Archive discovery was a non-recursive glob; after the split it found ZERO
  archived records, surfacing as four bogus "supersedes points to unknown key"
  errors rather than an obvious failure.
- ~32 live docs point into the corpus BY DATE, which the split dangles. Each
  stripped file now ends with a generated "Records formerly in this file" index,
  which also rescues the identical breadcrumbs in old issue comments.
- decisions.md's "In this file:" list was 97 same-file anchor bullets that the
  split makes WRONG, not merely stale. Dropped; the generated index replaces
  them with links that resolve.

The equivalence harness now runs against a checked-in FIXTURE, not the live
corpus. The earlier version migrated the real tree, which made it a one-shot:
the moment the migration landed there was nothing left to move and the tests
failed for reasons unrelated to the code. A fixture keeps them testing the
SCRIPT rather than the repo's current state.

Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain
two directories for one concept. Renaming a key is not a move -- it changes
identity, breaks the equivalence proof, and invalidates MemPalace's per-key
drawers. Taxonomy normalisation is separate work.

refs #610
2026-07-25 19:45:09 +02:00

5.6 KiB

key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
key title status since supersedes superseded-by rule signals mechanics
testing.e2e-cleanup-scope-by-pid 2026-07-25 — E2E cleanup kills only the PIDs it started; never a `pkill -f` pattern (#586) active 2026-07-25 none none An E2E harness or agent may only kill processes whose PIDs it captured at launch — capture the PID; whoever owns the lifecycle releases it from a `trap ... EXIT INT TERM`. Never `pkill -f "dotnet ErsatzTV.dll"` (or any pattern that can match a process this run did not start). A foreign listener is reported, not reaped. `pkill -f` · `pkill -f "dotnet ErsatzTV.dll"` · E2E teardown · shared-host reap · port 8409 · port 8410 · port 8420 · `lsof -ti :$PORT` · `trap cleanup EXIT INT TERM` · truncated E2E output · delegated-brief omission · paths: `scripts/e2e-local.sh`, `scripts/e2e-ui.sh`, `docs/e2e-local.md` · issues: #586, #596, #445, #533, #440, #406 `PID=$!` at launch, `trap cleanup EXIT INT TERM` in the lifecycle-owning script; to identify a foreign listener use `lsof -ti :"$PORT"` and fail with its PID, or re-run under `ETV_UI_PORT=<other>`.

A delegated agent running the #440 live-E2E found a dotnet ErsatzTV.dll already on the default port 8409 that wasn't its own (mismatched api.key). It moved its own run to 8420 — then during cleanup ran a broad pkill -f "dotnet ErsatzTV.dll", which would have reaped the parallel #445/#533 session's instance too. No damage that time: scripts/e2e-ui.sh owns its instance lifecycle and kills its server from an EXIT/INT/TERM trap, so a free 8409 was the expected post-run state, and all four determinism runs had already printed explicit 3 passed summaries.

(That 8420 run could only have succeeded because the other instance had already released 8409 — as the port section below explains, at the time ETV_UI_PORT=8420 alone still bound 8409 as well, so it would have failed outright against a live holder. The incident report's "correctly moved to 8420" reads as a clean sidestep; it was closer to a coincidence. That is itself part of why the escape hatch needed fixing rather than just documenting.)

The near-miss is the point. A mid-run kill of an E2E harness does not fail loudly — it truncates output into plausible-looking-but-wrong data, which is exactly the failure class that survives review. Same shape as the #406 peak-anon incident: reaping a shared-host resource you did not create.

Port separation is not mitigation — and there is less of it than the incident report assumed. A pkill -f "dotnet ErsatzTV.dll" matches the command line, so it hits every instance regardless of which port each one chose. Worse, the ports are not actually separated: the CI functional-e2e step exports ETV_UI_PORT=8409 (docker-build.yml), the same port local runs use — 8410 is the ersatztv-test container on jazz, a different thing that the incident report conflated with it. (The ui-E2E step added by #445 does use 8410.) Scope by PID, not by pattern and not by port.

And "just use another port" is not a complete escape either. Program.cs binds two listeners — ListenAnyIP(UiPort) plus ListenAnyIP(StreamingPort) when they differ — and SystemEnvironment.cs defaults StreamingPort to 8409 independently of ETV_UI_PORT. So ETV_UI_PORT=8420 alone still binds 8409 and dies at startup against a foreign holder. An agent that took the obvious escape hatch would land right back in the confusion that produced the pattern kill. scripts/e2e-local.sh now defaults ETV_STREAMING_PORT to the port it was given, and probes both before launching.

Root cause is a delegation gap, not agent error. The brief specified a fresh config dir (per testing.e2e-local-fresh-config-dir) but said nothing about process cleanup, so the agent invented a reasonable-looking pattern kill. An omitted rule is not an unenforced rule — it is a rule replaced by whatever plausible default the agent reaches for. Hence the constraint lives in docs/e2e-local.md, where a brief-writer reads it, and not only here.

Known tension with testing.playwright-mcp-download-and-recovery, which prescribes pkill -f ms-playwright-mcp as the stall-recovery move. That is the same pattern-kill hazard this record forbids — on a shared machine it reaps every session's MCP server, not just the stalled one — and the rule above is deliberately general ("any pattern that can match a process this run did not start"), so it covers that case too. That record is not line-edited here (records are superseded, not rewritten — docs.decision-lifecycle); PID-scoping the MCP recovery is tracked as #596, which also has to answer whether a session's own MCP server is externally identifiable at all. Until that lands, prefer pgrep -f ms-playwright-mcp to identify the specific stalled PID and kill only that one.

scripts/e2e-local.sh deliberately does NOT adopt the trap. Its documented contract is the inverse of e2e-ui.sh's: it boots the server, prints PID/PORT/CONFIG_DIR/LOG, and exits leaving the server running so a caller can drive the live instance. An EXIT trap there would kill the server the instant the launcher returned, breaking every caller including e2e-ui.sh and the CI functional-e2e step. The split is correct as it stands: e2e-local.sh is the launcher and hands ownership of a single PID to its caller; e2e-ui.sh is a lifecycle owner and traps. What #586 changed in e2e-local.sh instead is the two things that actually prevent the incident — an lsof pre-flight that names a foreign listener's PID (so an agent meeting a busy port gets a diagnosis instead of inventing one) and an explicit statement of who owns the printed PID.