E2E briefs: scope process cleanup by PID, never a pattern-wide pkill #586

Closed
opened 2026-07-25 11:20:51 +02:00 by timothy · 2 comments
Owner

Follow-up to the 2026-07-25 incident on #445 (see comment).

What happened

A delegated agent running the #440 live-E2E found a dotnet ErsatzTV.dll already listening on the default port 8409 that wasn't its own (mismatched api.key). It correctly moved to port 8420 for its own run — then during cleanup ran a broad pkill -f "dotnet ErsatzTV.dll", which would have killed the parallel #445/#533 session's instance too.

No damage this time, confirmed by the affected session: scripts/e2e-ui.sh owns its instance lifecycle and kills its server via an EXIT/INT/TERM trap, so a free 8409 was the expected post-run state, and all four determinism runs plus the guard's negative control had printed explicit 3 passed summaries before the sweep. Nothing was truncated.

Why it's still worth fixing

The near-miss is the point. A mid-run kill of an E2E harness doesn't fail loudly — it truncates output into plausible-looking-but-wrong data, which is the failure mode that survives review (cf. docs/decisions/workflow-process.md on the #406 peak-anon incident, same shape: reaping a shared-host resource you didn't create).

Port separation is not sufficient mitigation, and this is the reusable fact from the affected session: their CI step uses 8410 while local runs use 8409, so a pattern-wide pkill -f "dotnet ErsatzTV.dll" hits either regardless. Scope by PID, not by pattern or port.

Root cause is a delegation gap, not agent error: the brief specified a fresh config dir (per testing.e2e-local-fresh-config-dir) but said nothing about process cleanup, so the agent invented a reasonable-looking pattern kill.

Correction (2026-07-25, from PR #598's review): two premises above are wrong, and the corrected facts strengthen the case rather than weaken it.

  1. "their CI step uses 8410 while local runs use 8409" — on main, docker-build.yml sets ETV_UI_PORT=8409 for functional-e2e, the same port as local runs. 8410 is the ersatztv-test container on jazz, conflated here; only the unmerged #445 branch adds an 8410 step. So there is even less port separation than assumed.
  2. "It correctly moved to port 8420 for its own run" — that could only have worked because the foreign instance had already exited. Program.cs binds a second listener whose port (ETV_STREAMING_PORT) defaults to 8409 independently of ETV_UI_PORT, so at the time ETV_UI_PORT=8420 alone still bound 8409 and would have failed outright against a live holder. The sidestep was closer to a coincidence — and it means the obvious escape hatch was itself a dead end, which is now fixed rather than merely documented.

Proposed

  • Record the constraint as a decision (testing.e2e-cleanup-scope-by-pid or similar): an E2E harness/agent may only kill PIDs it started — capture the PID at launch and trap on EXIT/INT/TERM, as scripts/e2e-ui.sh already does. Never pkill -f a pattern that can match another session's process.
  • Reflect it in docs/e2e-local.md so the constraint reaches the brief-writer, and consider adding it to the delegation guidance so future E2E briefs inherit it.
  • Consider whether scripts/e2e-local.sh should adopt the same trap-based lifecycle e2e-ui.sh has, and/or bind a session-unique port by default.

Done-when

  • Decision record added for PID-scoped E2E cleanup — testing.e2e-cleanup-scope-by-pid
  • docs/e2e-local.md states the constraint where a brief-writer will see it — plus a lore bullet in docs/handoffs/chicorytv-issue-queue.md, which is what a session actually reads at startup
  • scripts/e2e-local.sh reviewed against e2e-ui.sh's trap-based lifecycle (adopt or record why not) — recorded why not: it hands a running instance back to its caller (both callers use OUT="$(e2e-local.sh …)"), so an EXIT trap would kill the server the instant the launcher returned. It instead gained an lsof pre-flight and a working port escape hatch.
  • adversarial review passed — two rounds; round 1 BLOCKED on a real Blocker, round 2 MERGEABLE @ c8e79f49

PR: #598

Follow-up to the 2026-07-25 incident on #445 (see [comment](https://gitea.tblindustries.be/timothy/ersatztv/issues/445#issuecomment-15776)). ## What happened A delegated agent running the #440 live-E2E found a `dotnet ErsatzTV.dll` already listening on the default port 8409 that wasn't its own (mismatched `api.key`). It correctly moved to port 8420 for its own run — then during cleanup ran a broad `pkill -f "dotnet ErsatzTV.dll"`, which would have killed the parallel #445/#533 session's instance too. **No damage this time**, confirmed by the affected session: `scripts/e2e-ui.sh` owns its instance lifecycle and kills its server via an EXIT/INT/TERM trap, so a free 8409 was the expected post-run state, and all four determinism runs plus the guard's negative control had printed explicit `3 passed` summaries before the sweep. Nothing was truncated. ## Why it's still worth fixing The near-miss is the point. A mid-run kill of an E2E harness doesn't fail loudly — it truncates output into plausible-looking-but-wrong data, which is the failure mode that survives review (cf. `docs/decisions/workflow-process.md` on the #406 peak-anon incident, same shape: reaping a shared-host resource you didn't create). Port separation is **not** sufficient mitigation, and this is the reusable fact from the affected session: their CI step uses 8410 while local runs use 8409, so a pattern-wide `pkill -f "dotnet ErsatzTV.dll"` hits either regardless. **Scope by PID, not by pattern or port.** Root cause is a delegation gap, not agent error: the brief specified a fresh config dir (per `testing.e2e-local-fresh-config-dir`) but said nothing about process cleanup, so the agent invented a reasonable-looking pattern kill. > **Correction (2026-07-25, from PR #598's review):** two premises above are wrong, and the corrected facts strengthen the case rather than weaken it. > 1. **"their CI step uses 8410 while local runs use 8409"** — on `main`, `docker-build.yml` sets `ETV_UI_PORT=8409` for `functional-e2e`, the *same* port as local runs. 8410 is the `ersatztv-test` container on jazz, conflated here; only the unmerged #445 branch adds an 8410 step. So there is even less port separation than assumed. > 2. **"It correctly moved to port 8420 for its own run"** — that could only have worked because the foreign instance had already exited. `Program.cs` binds a second listener whose port (`ETV_STREAMING_PORT`) defaults to 8409 *independently* of `ETV_UI_PORT`, so at the time `ETV_UI_PORT=8420` alone still bound 8409 and would have failed outright against a live holder. The sidestep was closer to a coincidence — and it means the obvious escape hatch was itself a dead end, which is now fixed rather than merely documented. ## Proposed - Record the constraint as a decision (`testing.e2e-cleanup-scope-by-pid` or similar): an E2E harness/agent may only kill PIDs it started — capture the PID at launch and trap on EXIT/INT/TERM, as `scripts/e2e-ui.sh` already does. Never `pkill -f` a pattern that can match another session's process. - Reflect it in `docs/e2e-local.md` so the constraint reaches the brief-writer, and consider adding it to the delegation guidance so future E2E briefs inherit it. - Consider whether `scripts/e2e-local.sh` should adopt the same trap-based lifecycle `e2e-ui.sh` has, and/or bind a session-unique port by default. ## Done-when - [x] Decision record added for PID-scoped E2E cleanup — `testing.e2e-cleanup-scope-by-pid` - [x] `docs/e2e-local.md` states the constraint where a brief-writer will see it — plus a lore bullet in `docs/handoffs/chicorytv-issue-queue.md`, which is what a session actually reads at startup - [x] `scripts/e2e-local.sh` reviewed against `e2e-ui.sh`'s trap-based lifecycle (adopt or record why not) — **recorded why not**: it hands a running instance back to its caller (both callers use `OUT="$(e2e-local.sh …)"`), so an EXIT trap would kill the server the instant the launcher returned. It instead gained an `lsof` pre-flight and a working port escape hatch. - [x] adversarial review passed — two rounds; round 1 BLOCKED on a real Blocker, round 2 MERGEABLE @ c8e79f49 PR: #598
timothy added the ci-cdpriority: medium labels 2026-07-25 11:21:08 +02:00
timothy added the in-progress label 2026-07-25 12:18:23 +02:00
Author
Owner

Claiming — worked as part of a CI/repo-hygiene session bundle with #594 and #485 (disjoint file sets, one PR).

Claiming — worked as part of a CI/repo-hygiene session bundle with #594 and #485 (disjoint file sets, one PR).
Author
Owner

Closing record

Outcome: Shipped in PR #598 (merged). New decision record testing.e2e-cleanup-scope-by-pid; a teardown warning in docs/e2e-local.md written at brief-writers; a lore bullet in docs/handoffs/chicorytv-issue-queue.md. scripts/e2e-local.sh gained an lsof pre-flight that names a foreign listener's PID instead of reaping it, a numeric-port guard, and an ETV_STREAMING_PORT default. Swept with #594 and #485.

Root cause: A delegation gap, not agent error. The brief specified a fresh config dir (testing.e2e-local-fresh-config-dir) but said nothing about process cleanup, so the agent invented a plausible-looking pkill -f pattern. An omitted rule isn't an unenforced rule — it's a rule replaced by whatever default the agent reaches for.

Decisions/conventions changed: Added testing.e2e-cleanup-scope-by-pid. Documented a live tension with testing.playwright-mcp-download-and-recovery, which prescribes the same shared-host reap — deferred to #596 rather than line-edited, since reversing a record is a supersede-and-relocate operation.

Reusable knowledge: Three things worth carrying forward.

  1. ETV_UI_PORT=<other> alone does not move the app off 8409. Program.cs binds a second listener and SystemEnvironment.cs defaults StreamingPort to 8409 independently of the UI port. The obvious escape hatch was a dead end that led straight back to the confusion behind the pattern kill — verified by negative control (direct DLL launch at 8420 binds *:8420 and *:8409). Through e2e-local.sh this is now fixed; launching the DLL by hand still needs both vars.
  2. "The CI step uses 8410" was false. CI's functional-e2e exports ETV_UI_PORT=8409 — the same port as local runs. 8410 is the ersatztv-test container on jazz. There was even less port separation than the incident assumed.
  3. e2e-local.sh must NOT adopt e2e-ui.sh's trap — the contracts are inverse. It hands a running instance back to its caller, and both callers use OUT="$(e2e-local.sh …)", so an EXIT trap would fire when the command substitution returns and kill the server before anyone could use it. Launcher vs lifecycle-owner: only the owner traps.

Verification: Local boot with a negative control — old remedy binds 8420+8409, fixed script binds only 8420, CI shape (8409) boots, GET / → 302, clean kill "$PID", port freed. Pre-flight tested across six scenarios (busy UI port, busy streaming port, non-numeric, empty, free-port control). CI green: Build & test, EF migration integrity, api-docs, format, all PR Gates. functional-e2e initially failed on infra (ReferenceError: __dirname inside actions/cache, before any PR code ran); reran green on identical code with passed: 45 failed: 0, log confirming the modified script actually booted the app. Two-round cold-context adversarial review — round 1 BLOCKED on a real Blocker (the port dead end), round 2 MERGEABLE @ c8e79f49.

Deferred: #596 — PID-scope the Playwright-MCP stall recovery. It flags an honest open question: if MCP servers are externally indistinguishable across sessions, a safe PID-scoped recovery may not exist and the rule becomes "restart via the client, never the shell."

Docs updated: docs/e2e-local.md, docs/decisions/workflow-process.md (+ regenerated docs/decisions/README.md), docs/handoffs/chicorytv-issue-queue.md.

## Closing record **Outcome:** Shipped in PR #598 (merged). New decision record `testing.e2e-cleanup-scope-by-pid`; a teardown warning in `docs/e2e-local.md` written *at brief-writers*; a lore bullet in `docs/handoffs/chicorytv-issue-queue.md`. `scripts/e2e-local.sh` gained an `lsof` pre-flight that names a foreign listener's PID instead of reaping it, a numeric-port guard, and an `ETV_STREAMING_PORT` default. Swept with #594 and #485. **Root cause:** A delegation gap, not agent error. The brief specified a fresh config dir (`testing.e2e-local-fresh-config-dir`) but said nothing about process cleanup, so the agent invented a plausible-looking `pkill -f` pattern. An omitted rule isn't an unenforced rule — it's a rule replaced by whatever default the agent reaches for. **Decisions/conventions changed:** Added `testing.e2e-cleanup-scope-by-pid`. Documented a live tension with `testing.playwright-mcp-download-and-recovery`, which prescribes the same shared-host reap — deferred to #596 rather than line-edited, since reversing a record is a supersede-and-relocate operation. **Reusable knowledge:** Three things worth carrying forward. 1. **`ETV_UI_PORT=<other>` alone does not move the app off 8409.** `Program.cs` binds a *second* listener and `SystemEnvironment.cs` defaults `StreamingPort` to 8409 independently of the UI port. The obvious escape hatch was a dead end that led straight back to the confusion behind the pattern kill — verified by negative control (direct DLL launch at 8420 binds `*:8420` **and** `*:8409`). Through `e2e-local.sh` this is now fixed; launching the DLL by hand still needs **both** vars. 2. **"The CI step uses 8410" was false.** CI's `functional-e2e` exports `ETV_UI_PORT=8409` — the *same* port as local runs. 8410 is the `ersatztv-test` container on jazz. There was even less port separation than the incident assumed. 3. **`e2e-local.sh` must NOT adopt `e2e-ui.sh`'s trap** — the contracts are inverse. It hands a *running* instance back to its caller, and both callers use `OUT="$(e2e-local.sh …)"`, so an `EXIT` trap would fire when the command substitution returns and kill the server before anyone could use it. Launcher vs lifecycle-owner: only the owner traps. **Verification:** Local boot with a negative control — old remedy binds 8420+8409, fixed script binds only 8420, CI shape (8409) boots, `GET /` → 302, clean `kill "$PID"`, port freed. Pre-flight tested across six scenarios (busy UI port, busy streaming port, non-numeric, empty, free-port control). CI green: `Build & test`, `EF migration integrity`, `api-docs`, `format`, all `PR Gates`. `functional-e2e` initially failed on infra (`ReferenceError: __dirname` inside `actions/cache`, before any PR code ran); reran green on identical code with `passed: 45 failed: 0`, log confirming the modified script actually booted the app. Two-round cold-context adversarial review — round 1 BLOCKED on a real Blocker (the port dead end), round 2 MERGEABLE @ c8e79f49. **Deferred:** #596 — PID-scope the Playwright-MCP stall recovery. It flags an honest open question: if MCP servers are externally indistinguishable across sessions, a safe PID-scoped recovery may not exist and the rule becomes "restart via the client, never the shell." **Docs updated:** `docs/e2e-local.md`, `docs/decisions/workflow-process.md` (+ regenerated `docs/decisions/README.md`), `docs/handoffs/chicorytv-issue-queue.md`.
timothy removed the in-progress label 2026-07-25 13:52:35 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#586