timothy
e8ad79623f
Codex round-2 verdict was BLOCKED @ 2a5f26c0 on one High finding, which was
correct and is the subtlest defect in this whole change.
## The defect
Bash DEFERS a trapped signal while it waits on a FOREGROUND command. The boot was
OUT="$(ETV_UI_PORT=... scripts/e2e-local.sh "$CONFIG_DIR")"
so a TERM arriving during the (up to 120s) readiness wait did not run `on_signal`
until boot completed — and a supervisor escalating TERM->KILL means the trap
never runs at all, orphaning the server on its port. The pidfile added in the
previous commit only helps IF the trap runs; this is the case where it doesn't.
Proved the mechanism in isolation rather than asserting it, since the real boot
is only ~2s locally and kept winning the race:
foreground command substitution : TERM at t=1s -> trap ran at t=6s (deferred)
background job + `wait` : TERM at t=1s -> trap ran at t=1s (prompt)
## The fix
Boot in the background and `wait` on it. `wait` is interruptible, so the handler
runs immediately; by then e2e-local.sh has already written $ETV_PIDFILE, so
cleanup can reap a server whose PID this script has not yet parsed. Cleanup also
now stops the backgrounded e2e-local.sh itself, so it cannot sit waiting on a
server we just killed, and removes its temp output file.
Boot diagnostics are preserved on the failure path (verified with a deliberately
bad ETV_BUILD_CONFIG: the underlying "bin/<config> does not exist" error is
surfaced, along with the exit status, which is now propagated rather than
flattened to 1).
## Verification
- mechanism: deferred-vs-prompt trap proven as above
- passing run 3/3 green; failing run (--grep ZZZ_NOPE) exits 1
- boot-failure path prints e2e-local.sh's real error and its exit status
- targeted SIGTERM: exit 143, no listener, no stray process
- curl harness unaffected: 45/45 PASS
- no orphaned processes after the full gate (an earlier count of 1 was a
graceful shutdown still in flight, not a leak — re-checked clean)
Refs #445
210 lines
10 KiB
Bash
Executable File
210 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# scripts/e2e-ui.sh — boot a FRESH ErsatzTV instance and run the UI-interactive Playwright flows
|
|
# (ersatztv#445). The curl counterpart is scripts/e2e-functional.sh; see docs/e2e-local.md for both.
|
|
#
|
|
# Usage:
|
|
# scripts/e2e-ui.sh [CONFIG_DIR] # extra args after CONFIG_DIR are passed to `playwright test`
|
|
#
|
|
# CONFIG_DIR defaults to a fresh `mktemp -d`. A FRESH dir is REQUIRED, not merely preferred: the first
|
|
# spec asserts the Setup (first-run admin claim) gate, which only appears while the server has no
|
|
# local admin. Point this at a reused config dir and that spec fails by design.
|
|
#
|
|
# Unlike e2e-local.sh, this script OWNS the instance lifecycle: it boots the server, runs the specs,
|
|
# and always kills the server on the way out (success, failure, or interrupt). Exit status is
|
|
# Playwright's, so this is safe to use directly as a CI gate.
|
|
#
|
|
# Assumes `dotnet build` + `cd web && npm ci && npm run build` have already run — same contract as
|
|
# e2e-local.sh, which this delegates the boot to.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(git rev-parse --show-toplevel)"
|
|
PORT="${ETV_UI_PORT:-8409}"
|
|
CONFIG_DIR="${1:-$(mktemp -d)}"
|
|
shift || true
|
|
|
|
# The Chromium build Playwright drives is baked into the CI toolchain image at this path
|
|
# (docker/ci/Dockerfile); locally it lands in the default per-user cache. Only export the shared
|
|
# path when it actually exists, so a local run keeps using the user cache.
|
|
if [ -z "${PLAYWRIGHT_BROWSERS_PATH:-}" ] && [ -d /ms-playwright ]; then
|
|
export PLAYWRIGHT_BROWSERS_PATH=/ms-playwright
|
|
fi
|
|
|
|
if [ ! -d "$REPO_ROOT/web/node_modules/@playwright/test" ]; then
|
|
echo "error: web/node_modules/@playwright/test is missing — run 'cd web && npm ci' first" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Version-drift guard: actually LAUNCH the browser and close it. Playwright pins a browser REVISION
|
|
# per package version, so bumping web/package.json's @playwright/test pin (e.g. by Renovate) without
|
|
# rebuilding the CI toolchain image leaves no usable browser. Failing here with an actionable message
|
|
# beats Playwright's bare "Executable doesn't exist at ..." mid-suite.
|
|
#
|
|
# Probing by LAUNCH rather than by path is deliberate: the CI image bakes only
|
|
# `chromium-headless-shell` (267M vs 656M for full chromium), and `chromium.executablePath()` reports
|
|
# the FULL chromium path, which does not exist there — a path check would fail on a perfectly good
|
|
# image. Launching exercises exactly what the specs do.
|
|
PW_VERSION="$(cd "$REPO_ROOT/web" && node -p "require('@playwright/test/package.json').version")"
|
|
# mktemp, not a predictable /tmp/...$$ path: on a shared host a pre-created symlink at a guessable
|
|
# name would redirect this write.
|
|
PW_PROBE_ERR="$(mktemp)"
|
|
if ! (cd "$REPO_ROOT/web" && node -e "require('@playwright/test').chromium.launch().then(b=>b.close())") 2>"$PW_PROBE_ERR"; then
|
|
echo "error: could not launch headless chromium for @playwright/test ${PW_VERSION}:" >&2
|
|
sed 's/^/ /' "$PW_PROBE_ERR" >&2 || true
|
|
rm -f "$PW_PROBE_ERR"
|
|
echo " In CI this means the browser baked into the toolchain image no longer matches" >&2
|
|
echo " web/package.json's @playwright/test pin (${PW_VERSION}): bump PLAYWRIGHT_VERSION in" >&2
|
|
echo " docker/ci/Dockerfile to match, let ci-image.yml publish the new :<sha>, then update" >&2
|
|
echo " all five container pins in docker-build.yml (docs/ci-cd.md -> 'CI toolchain image')." >&2
|
|
echo " Locally: cd web && npx playwright install chromium" >&2
|
|
exit 1
|
|
fi
|
|
rm -f "$PW_PROBE_ERR"
|
|
|
|
# Pre-flight: fail fast and clearly if the port is already taken. Otherwise the symptom is
|
|
# e2e-local.sh's 120s readiness timeout, which reads like a broken build rather than "something else
|
|
# is already listening" — a real hazard when several sessions share this machine.
|
|
#
|
|
# `-sTCP:LISTEN` matters: a bare `lsof -ti :PORT` also matches outbound/ESTABLISHED/TIME_WAIT
|
|
# sockets touching that port number, which would abort on a false positive.
|
|
#
|
|
# This is a LOCAL-developer guard. The CI toolchain image ships no `lsof`, so the check silently
|
|
# no-ops there — acceptable, because each CI job gets a fresh container where nothing else is bound.
|
|
if command -v lsof >/dev/null 2>&1 && [ -n "$(lsof -ti :"$PORT" -sTCP:LISTEN 2>/dev/null || true)" ]; then
|
|
echo "error: port $PORT already has a listener: PID(s) $(lsof -ti :"$PORT" -sTCP:LISTEN | tr '\n' ' ')" >&2
|
|
echo " Another instance (possibly another session's) is listening. Stop it, or re-run with" >&2
|
|
echo " a free port, e.g. ETV_UI_PORT=8419 scripts/e2e-ui.sh" >&2
|
|
echo " (8409/8410 are prod + ersatztv-test on jazz; 8411 is scripts/security-scan.sh.)" >&2
|
|
echo " Do NOT blanket-kill 'dotnet ErsatzTV.dll' — that can reap another session's server." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# --- Server lifecycle -------------------------------------------------------------------------
|
|
#
|
|
# The trap is installed BEFORE the server is booted, not after. A signal (or any early exit)
|
|
# arriving while `e2e-local.sh` is still starting the app would otherwise leave a live server bound
|
|
# to the port with no handler to reap it — precisely the "stray process" failure this script exists
|
|
# to prevent.
|
|
#
|
|
# `wait "$PID"` cannot be used to confirm the kill: e2e-local.sh launches the app inside its own
|
|
# subshell and then exits, so the process is NOT a child of THIS shell — `wait` fails instantly
|
|
# ("not a child of this shell") and returns before the port is released. Poll `kill -0` instead,
|
|
# then escalate to SIGKILL.
|
|
SERVER_PID=""
|
|
# e2e-local.sh writes the server PID here the instant it forks, BEFORE its readiness wait. That
|
|
# closes the mid-boot window: if this script is signalled while the app is still starting, we have
|
|
# never parsed stdout, but the pidfile already names the process.
|
|
#
|
|
# This replaces an earlier "kill whatever LISTENS on $PORT" fallback, which was wrong on two counts:
|
|
# it needed `lsof` (ABSENT from the CI toolchain image — verified — so it silently no-opped exactly
|
|
# where it was needed), and it inferred ownership from the pre-flight instead of proving it, so a
|
|
# process that grabbed the port after the pre-flight — or a real instance on a shared host — could be
|
|
# killed. A pidfile we asked for proves ownership outright and needs no external tool.
|
|
PIDFILE="$(mktemp)"
|
|
export ETV_PIDFILE="$PIDFILE"
|
|
|
|
# Reap $1 with a bounded graceful wait, then SIGKILL. `wait` is unusable here: the server is not a
|
|
# child of THIS shell (e2e-local.sh forked it and exited), so `wait` fails instantly and would return
|
|
# before the port is released.
|
|
reap() {
|
|
local pid="$1"
|
|
kill "$pid" 2>/dev/null || true
|
|
local i=0
|
|
while [ "$i" -lt 50 ]; do # up to ~5s
|
|
kill -0 "$pid" 2>/dev/null || return 0
|
|
sleep 0.1
|
|
i=$((i + 1))
|
|
done
|
|
echo "warning: server $pid did not exit after SIGTERM; sending SIGKILL" >&2
|
|
kill -9 "$pid" 2>/dev/null || true
|
|
}
|
|
|
|
cleanup() {
|
|
# If we were signalled mid-boot, e2e-local.sh itself is still running (backgrounded, see below).
|
|
# Stop it first so it can't keep waiting on a server we are about to kill.
|
|
if [ -n "${BOOT_PID:-}" ]; then
|
|
kill "$BOOT_PID" 2>/dev/null || true
|
|
BOOT_PID=""
|
|
fi
|
|
rm -f "${BOOT_OUT:-}" 2>/dev/null || true
|
|
|
|
local pid="$SERVER_PID"
|
|
if [ -z "$pid" ] && [ -s "$PIDFILE" ]; then
|
|
pid="$(tr -dc '0-9' <"$PIDFILE" 2>/dev/null || true)"
|
|
fi
|
|
SERVER_PID=""
|
|
rm -f "$PIDFILE"
|
|
[ -n "$pid" ] || return 0
|
|
reap "$pid"
|
|
}
|
|
|
|
# Re-raise INT/TERM after cleaning up, so the wrapper dies BY the signal rather than exiting 0.
|
|
# Without this a TERM landing next to a passing Playwright run reports success for a cancelled run.
|
|
on_signal() {
|
|
cleanup
|
|
trap - EXIT "$1"
|
|
kill -"$1" $$
|
|
}
|
|
trap cleanup EXIT
|
|
trap 'on_signal INT' INT
|
|
trap 'on_signal TERM' TERM
|
|
|
|
echo "Booting instance for UI-E2E (config: $CONFIG_DIR, port: $PORT)..."
|
|
|
|
# Boot in the BACKGROUND and `wait` on it, rather than the obvious
|
|
# `OUT="$(... e2e-local.sh ...)"`. Bash defers a trapped signal until the current FOREGROUND command
|
|
# finishes, so with a command substitution a TERM arriving during the (up to 120s) readiness wait
|
|
# would not run `on_signal` until boot completed — and a supervisor that escalates TERM->KILL would
|
|
# mean the trap never runs at all, orphaning the server on its port. `wait` is interruptible: the
|
|
# handler runs immediately, and by then e2e-local.sh has already written $ETV_PIDFILE, so cleanup
|
|
# can reap a server whose PID this script has not yet parsed.
|
|
BOOT_OUT="$(mktemp)"
|
|
ETV_UI_PORT="$PORT" "$REPO_ROOT/scripts/e2e-local.sh" "$CONFIG_DIR" >"$BOOT_OUT" 2>&1 &
|
|
BOOT_PID=$!
|
|
|
|
set +e
|
|
wait "$BOOT_PID"
|
|
boot_status=$?
|
|
set -e
|
|
|
|
OUT="$(cat "$BOOT_OUT")"
|
|
rm -f "$BOOT_OUT"
|
|
|
|
if [ "$boot_status" -ne 0 ]; then
|
|
echo "error: e2e-local.sh failed to boot the instance (exit $boot_status). Its output:" >&2
|
|
printf '%s\n' "$OUT" >&2
|
|
exit "$boot_status"
|
|
fi
|
|
printf '%s\n' "$OUT"
|
|
SERVER_PID="$(printf '%s\n' "$OUT" | awk -F= '/^PID=/{print $2}')"
|
|
SERVER_LOG="$(printf '%s\n' "$OUT" | awk -F= '/^LOG=/{print $2}')"
|
|
|
|
if [ -z "$SERVER_PID" ]; then
|
|
echo "error: could not determine the server PID from e2e-local.sh output" >&2
|
|
exit 1 # the EXIT trap's port-based fallback reaps the server we just started
|
|
fi
|
|
|
|
echo "Running Playwright UI-E2E specs against http://localhost:${PORT} ..."
|
|
cd "$REPO_ROOT/web"
|
|
|
|
# Capture the status WITHOUT `if ! cmd; then` — under `!` negation bash sets `$?` to the *logical
|
|
# negation*, so `$?` reads 0 inside the failure branch and the script would `exit 0` on a FAILING
|
|
# spec run, silently passing CI. (Verified: `if ! (exit 42); then echo $?; fi` prints 0.) `set +e`
|
|
# around the call, then read `$?` directly, is unambiguous.
|
|
set +e
|
|
ETV_BASE_URL="http://localhost:${PORT}" npx --no-install playwright test "$@"
|
|
status=$?
|
|
set -e
|
|
|
|
if [ "$status" -ne 0 ]; then
|
|
# No workflow in this repo uploads artifacts, so traces/screenshots in web/e2e/.output never leave
|
|
# the runner. The server log is the only server-side evidence a CI failure would otherwise lack.
|
|
if [ -n "${SERVER_LOG:-}" ] && [ -f "$SERVER_LOG" ]; then
|
|
echo "--- server log tail ($SERVER_LOG) ---" >&2
|
|
tail -n 40 "$SERVER_LOG" >&2 || true
|
|
echo "--- end server log ---" >&2
|
|
fi
|
|
fi
|
|
|
|
exit "$status"
|