fix(809,822): isolate the suite from the production hook-fire log by construction
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 39s
Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 37s
PR Gates / Docs update reminder (pull_request) Successful in 39s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 48s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 17s
PR Gates / decisions lifecycle (pull_request) Successful in 14s
review-verdict/h10 Review-verdict: MERGEABLE @ 1d14161 (base: main)
Review verdict / Set review-verdict status (pull_request_target) Successful in 23s
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 7m6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m9s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m16s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m4s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s

The guard meant to keep `pytest scripts/tests` out of the shared hook-fire log
had two independent defects, and both issue bodies land on the same answer: make
the shared path unreachable rather than watch it.

#822 — its oracle was global mutable state. It snapshotted `st_mtime_ns` across
`~/.cache/ersatztv/hook-fire/` and required no change, but that directory is
written by every live Claude Code session on the machine, so an unrelated
session firing a hook inside the window failed the suite with `a test run
modified the production hook-fire log`. Seen on three separate branches, green
on every immediate re-run. Reproduced on demand while writing this: with a
thread touching a file there, the withdrawn oracle goes RED and the replacement
stays GREEN.

#809 — its isolation was per test. The autouse fixture runs at test setup, after
every module has been imported, so it never reached a module holding a
`{**os.environ}` snapshot (#785, 58 records per run).

Measuring changed the diagnosis twice. #809 proposes a source-text scan for
module-level dicts, flagging in advance that such a predicate costs rounds.
Instrumenting `subprocess.Popen` across a full run and asking which launches
resolve to `$HOME/.cache/ersatztv/hook-fire` found 83 — 37 during collection and
44 inside module- or session-scoped fixtures, which are set up outside any single
test and so were never covered by the autouse fixture at all; only 2 resembled
the route the issue names. A source scan would have reported the suite clean.

Instrument `Popen` ALONE: `subprocess.run` reaches it, so a census wrapping both
counts every launch twice. The first one here did, and published 166/74/88/4
until cold review refuted it as exactly 2x — the figures above are a single
re-measurement on this branch's own base. Every count is now quoted with the
predicate that produced it, because the from-scratch launches also number 83: a
different set of the same size, which is precisely what a later reader
reconciles wrongly.

So: `pytest_configure` sets `ETV_HOOK_FIRE_LOG_DIR` before collection (ordering
verified by execution, not read), and a `Popen` wrapper fails any launch that
does not CARRY the isolated directory.

That second rule is deliberately "carries an isolated dir" and not "is not the
production one" — cold review found the first draft's blacklist modelled one of
the sink's two default branches and was blind to the other. `${HOME:-/tmp}`
means a child with no HOME lands in `/tmp/.cache/ersatztv/hook-fire`, which is
equally shared and equally persistent; measured, 81 of the 83 from-scratch
launches were exactly that shape. The whitelist covers both branches and does not depend
on what HOME is on the machine running the suite — which is also what stops it
firing where HOME is unset or is itself /tmp. A second round then found the same
blind spot surviving in the clause that rejects an explicit WRONG value, which
compared against the $HOME path alone; it now rejects both shared logs. Its
corollary is that a from-scratch environment must carry the variable, one line
each. No count of those sites is kept in the docs: it is a hand-written
population nothing derives, and this commit falsified its own by adding sites in
the tests that prove the rule. The guard enumerates them.

The resolver reimplements one shell expansion and fails OPEN, so it is
differential-tested against `etv_hook_fire_log_dir` every run — the empty-value
cases where `dict.get(k, d)` and `${k:-d}` disagree, and the normalisation cases
where shell concatenation and `Path` spell the same directory differently, which
is why it compares realpath identity rather than bytes.

Proofs, each witnessed red by disarming one clause and restoring it: reverting
to the production-path blacklist leaves the `/tmp` branch unguarded (DID NOT
RAISE); removing the pre-collection assignment makes an import-time snapshot
leak into a fake HOME (both arms nested pytest, launch guard disabled in both so
the comparison is about that clause alone); removing the wrapper install
un-guards a launch; breaking the resolver's empty-value semantics reddens the
differential; and with both layers off the cross-suite assertion fires on its
own. A fourth round, again on two fronts, found the `cwd` plumbing raising
`TypeError` on bytes — a shape `Popen` legally accepts — so a legal launch
crashed the guard instead of being judged. The first fix for that also added a
branch for an int file descriptor, and cold review then measured two things: a
descriptor is NOT a launch shape (`Popen` calls `os.fsencode(cwd)`, which rejects
an int), and the branch was dead anyway because `os.fspath` already raises on
one — deleting it left its own test green. Two consecutive rounds whose finding
was created by the previous round's fix is this corpus's stated trigger to
subtract rather than guard, so the branch and its unreachable fail-closed arm are
gone; `cwd` is normalised across the three shapes that exist, and the premise
that there are three is pinned by a test that reddens if CPython ever adds a
fourth. The same round caught that test being vacuous — it used an ABSOLUTE log
dir, which short-circuits before `cwd` is consulted at all.

A fifth round found the last of that family: `Popen` also accepts a bytes-keyed
or bytes-valued `env` on POSIX, and the guard rejected one such launch as
carrying no isolated dir and crashed on the other — both reds on a correctly
isolated child. Fixed at the boundary rather than per lookup: the environment is
normalised to str once, in one place, and a test pins that the normalisation
does not blunt the guard. That round also showed the cwd shape test proved only
"does not crash": ignoring the child's `cwd` entirely left it green, so it now
judges the same relative value against two different cwds and requires two
different verdicts.

One finding of that round is recorded as REFUTED rather than fixed: it reported
the from-scratch launch count as 391 rather than 83. Re-measured directly —
instrumenting `Popen` and running `test_ci_dropped_step_guard.py` alone gives 65
launches, matching the census artifact, against the 345 claimed for that file.
The 83 = 65 + 14 + 2 + 2 breakdown stands.

Five rounds each falsified another hand-written claim in the prose, so the last
pass subtracts prose rather than adding more: the durable docs no longer restate
refuted figures or carry round-by-round chronology, which is `docs.no-session-
narrative` — that belongs here, in the commit message. The traps and the rejected
approaches stay, per that rule's carve-out. Full suite
green (1224 passed/2 skipped on this branch's base, 1228/2 after — the four added
tests) with zero UNINTENDED guard rejections — the only ones are those the
launch-guard test provokes on purpose —
so the strictness costs nothing on a clean checkout.

A third round, this one cross-family, found the same blind spot once more in a
form neither earlier round reached: a RELATIVE `ETV_HOOK_FIRE_LOG_DIR` resolves
against the CHILD's working directory, so `.cache/ersatztv/hook-fire` with
`cwd=$HOME` lands exactly on the shared log while a guard anchoring it at its own
cwd sees an unrelated path under the repo and clears it. Reproduced, then closed
by anchoring at the launch's `cwd=`; both `env` and `cwd` positions are read from
`inspect.signature` rather than hand-counted. Witnessed red by reverting to
parent-anchored resolution. That round also caught a count left at 82 in one file
where every other copy said 83, and two docstrings still describing records as
leaking into the REAL log when they now land in the session directory.

Two defects found in the new tests while writing them. pytest's assert rewriting
printed the whole environment — API keys included — into the failure output,
because the violation was computed inside the `assert` expression. And
`os.environ["HOME"]` would KeyError where HOME is unset, which is a legal
environment the sink has a `${HOME:-/tmp}` default for.

fixes #809
fixes #822

Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RK43vYYMULNirkJ824ggud
This commit is contained in:
2026-08-29 03:58:12 +02:00
co-authored by Claude Opus 5
parent 90f96c14a5
commit 1d14161889
13 changed files with 945 additions and 74 deletions
+3 -2
View File
@@ -136,8 +136,9 @@ bounds, what's mined per issue): `docs/handoffs/chicorytv-issue-queue.md` → "K
detector, since the class has no plausible linter (`docs/defect-shapes-773.md` §4 detector D).
- **`docs/guard-inventory.md`** — every executable guard file, what it blocks, whether it is a
`GUARD` or `TOOLING`, and whether it ships a mutation proof (`MUTATION` / `BEHAVIOUR-ONLY` /
`NONE`) with a `file::function` ref. The population is derived from the filesystem and the
workflow/hook call sites and compared for set equality by `scripts/tests/test_guard_inventory.py`,
`NONE`) with a `file::function` ref. The population is derived from the GIT INDEX (not a
filesystem walk, since ersatztv#806) and the workflow/hook call sites, and compared for set
equality by `scripts/tests/test_guard_inventory.py`,
so a new guard cannot ship unclassified and a renamed test cannot leave a row claiming coverage it
has lost. Guards implemented inline in workflow YAML are deliberately outside that population —
the doc states the limit rather than implying coverage.
File diff suppressed because one or more lines are too long
@@ -0,0 +1,55 @@
---
key: testing.suite-isolated-from-production-hook-fire-log
title: '2026-08-28 — the test suite is isolated from the production hook-fire log BY CONSTRUCTION, and never by observing it (#809, #822)'
status: active
since: '2026-08-28'
supersedes: none
superseded-by: none
rule: 'THE SUITE MUST NOT WRITE TO `$HOME/.cache/ersatztv/hook-fire/`, AND THAT PROPERTY IS ESTABLISHED STRUCTURALLY RATHER THAN OBSERVED. Two layers in `scripts/tests/conftest.py`, which close different routes and must both stay. FIRST, `pytest_configure` sets `ETV_HOOK_FIRE_LOG_DIR` BEFORE COLLECTION. An autouse fixture alone runs at test setup — after every module has been imported — so it cannot reach a module that snapshots `{**os.environ}` at import time (the ersatztv#785 defect, 58 leaked records per run) and cannot reach a MODULE- or SESSION-SCOPED FIXTURE at all; measured on the pre-change tree by instrumenting `subprocess.Popen` for a full run and asking which launches resolve to `$HOME/.cache/ersatztv/hook-fire`, 83 did — 37 during collection, 44 inside module- or session-scoped fixtures, and 2 from an environment built from scratch. INSTRUMENT `Popen` ALONE: `subprocess.run`, `call` and `check_output` all reach it, so a census wrapping `run` as well counts every launch twice and every figure doubles. STATE THE PREDICATE WITH THE COUNT: this one also UNDERCOUNTS the from-scratch route, because such an environment usually carries no `HOME` either and lands in the sink''s OTHER default, which this predicate cannot see. The launches that carry no `ETV_HOOK_FIRE_LOG_DIR` and are built from scratch number 83 as well, and 81 of those carry no `HOME` — two different sets of the same size, which is precisely the shape a later reader reconciles wrongly. The per-test fixture stays, because a test that inspects its own log dir must not see another test''s records. SECOND, a `Popen` wrapper fails EVERY launch that does not CARRY the isolated directory — the route left open when an environment is built FROM SCRATCH rather than derived from `os.environ`, where no snapshot is involved and import ordering cannot help. THE RULE IS "CARRIES AN ISOLATED DIR", NOT "IS NOT THE PRODUCTION ONE", and that is the difference between a guard and a guard-shaped hole: `etv_hook_fire_log_dir` has TWO default branches — `$HOME/.cache/...` and, when `HOME` is unset or empty, `/tmp/.cache/...` — so a check comparing against one resolved production path models one branch and waves the other through, and `/tmp/.cache/ersatztv/hook-fire` is just as shared, just as persistent and just as readable with `hook-fire-log.sh report --dir`. Measured: of the 83 from-scratch launches, 81 carry no `HOME` either. Requiring the variable covers both branches and any a future sink adds, and is independent of what `HOME` happens to be on the machine running the suite — which is also what keeps it from firing on legitimate launches where `HOME` is unset or is itself `/tmp`. The corollary is that every from-scratch environment in the suite must carry the variable, one line each. NO COUNT OF THOSE SITES IS KEPT HERE: it is a hand-written population literal that nothing derives, and the commit introducing this rule already falsified its own, by adding further construction sites in the tests that prove the rule. The guard itself is the enumerator; a missing site is a red, not a stale sentence. Its population is every launch that actually happens, derived at runtime, so a new suite is covered by existing rather than by remembering. NEVER OBSERVE THE PRODUCTION DIRECTORY TO PROVE THIS. The withdrawn guard snapshotted `st_mtime_ns` across it and required no change, which makes the oracle state every concurrent session on the machine writes: an unrelated session firing a hook inside the window failed the suite with `a test run modified the production hook-fire log`, an accusation about the suite when the writer was another process. Seen on three separate branches, green on every immediate re-run. The misattribution is the expensive part — it aims the next reader at the suite, and a guard that cries wolf gets waved through. THE RESOLVER IS A REIMPLEMENTATION AND FAILS OPEN, so it is differential-tested against `etv_hook_fire_log_dir` every run: a resolver that disagrees clears a launch the sink then points at the real log. Shell''s `:-` treats an EMPTY value as absent, which `dict.get(k, default)` does not — that case is in the matrix because it is where a Python transliteration goes wrong. AND BIND A VIOLATION BEFORE ASSERTING ON IT: pytest rewrites an assert expression and prints the repr of every sub-expression, so `assert isolation_violation(os.environ) is None` dumps the environment — API keys included — into the failure output and from there into a CI log.'
signals: 'test suite wrote to the production hook-fire log · a test run modified the production hook-fire log · import-time os.environ snapshot · pytest_configure runs before collection · module- and session-scoped fixtures escape an autouse function fixture · env built from scratch with the real HOME · Popen launch guard · differential against etv_hook_fire_log_dir · shell `:-` treats empty as absent · assert rewriting leaks secrets into the log · paths: `scripts/tests/conftest.py`, `scripts/tests/hook_fire_isolation.py`, `scripts/tests/test_hook_fire_log.py`, `scripts/hook-fire-log.sh` · issues: #809, #822, #785, #776'
mechanics: 'A RELATIVE `ETV_HOOK_FIRE_LOG_DIR` resolves against the CHILD''s working directory, so the guard anchors it at the `cwd=` of the launch and not at its own; cross-family review found the parent-anchored version clearing `ETV_HOOK_FIRE_LOG_DIR=.cache/ersatztv/hook-fire` with `cwd=$HOME`, which lands exactly on the shared log. Both `env` and `cwd` positions are read from `inspect.signature`, never hand-counted, and `cwd` is normalised across the THREE shapes `Popen` accepts — str, bytes and `PathLike` — because the first anchoring called `os.fspath` on it and `os.path.join` then raised `TypeError` on mixing str with bytes. THREE, not four: `Popen` calls `os.fsencode(cwd)`, which rejects an int, so a file descriptor is not a launch shape (measured on CPython 3.9 and 3.13). A branch for the int case is deliberately ABSENT rather than defensive: it would be dead, because `os.fspath` already raises on an int, so it could not be witnessed red. The three-shape premise is pinned by a test asserting `Popen` itself rejects one, so a future CPython that accepts a descriptor reddens instead of silently reopening the gap. The `env` mapping has shapes too — bytes keys and bytes values are both legal — and is normalised at the boundary for the same reason. Ordering was verified by execution, not read: a module-level `{**os.environ}` does carry a value set in `pytest_configure`. `monkeypatch` in the per-test fixture reverts to the value in place at setup — the session directory — never to unset, so the first layer survives every teardown. The pure helpers live in `scripts/tests/hook_fire_isolation.py` rather than in `conftest.py` because importing `conftest` under its own name constructs a SECOND guard, whose `inspect.signature` call then reads the already-wrapped `Popen` and fails at import. Only `Popen` is wrapped: `subprocess.run`, `check_output` and `call` all reach the module global by name at call time. The `env` position is read from `inspect.signature` rather than hand-counted. A new module under `scripts/tests/` must be `git add`-ed before the mutation harness sees it — its sandbox is built from `git ls-files`, so an untracked helper makes every nested conftest import fail.'
---
**The two issues are one mechanism.** #822 is the oracle: a guard whose comparison reads a directory
every live session writes to. #809 is the scope: isolation applied per test, claiming to hold per
suite. Both bodies independently arrive at the same answer — make the production path unreachable
rather than watch it — which is `docs/defect-shapes-773.md` §4 detector C, dedup by construction.
Fixing them separately would have built the mechanism twice.
**What the measurement changed about the diagnosis.** #809's body names one route, the import-time
`{**os.environ}` snapshot, and proposes a source-text predicate to find modules that hold one —
flagging in advance that a string-matching predicate costs rounds. Instrumenting `Popen` across a
full run instead found 83 production-resolving launches, and only 2 came from anything resembling
that route — an environment built from scratch, not a module-level snapshot. The other 81 were code
running before any test: 37 during collection and 44 inside module- or session-scoped fixtures,
which are set up outside any single test and so were never covered by the autouse fixture at all. A source scan for module-level dicts would have
found nothing and reported the suite clean. The population that mattered was the set of launches
that actually happen, and only executing the suite produces it.
**Why the launch guard is conservative on purpose.** It rejects any child that does not carry the
variable, whether or not that child would source the sink. `bash` and `git` do not write fire-log
records, so some rejections are launches that would not actually have leaked. That is accepted: the
alternative is deciding per launch whether the command *might* source the sink, which is
unanswerable statically, and the fix at any real site is one line. It is also what makes the rule
independent of `HOME`, and therefore of the machine — the conditional version, "reject when this
resolves to the production path", is both blind to the `/tmp` branch and prone to firing on
legitimate launches wherever `HOME` is unset. Measured after the change, the whole suite produces
zero UNINTENDED rejections — the only ones are those the launch-guard test provokes on purpose —
so the strictness costs nothing on a clean checkout.
`testing.guard-derives-population-from-source` is the reason the population is not a list of
hook-driving suites instead.
**The sites it found are why the layer exists** — `test_ci_dropped_step_guard.py::_env`,
`test_ci_release_path_scan_job.py::_env` and `::_run_the_real_scan_body`, and a dict in
`test_pr_changed_files.py`, joined since by the tests proving this rule. Each builds a CI-shaped
environment from scratch, and `_run_the_real_scan_body` carries the real `HOME` so the scan step
behaves as it does on a runner before handing it to a nested `pytest`. No `os.environ` spread is
involved in any of them, so the pre-collection layer cannot reach them, and nothing about the code
reads as wrong. They were found by measuring launches — and by asking the right question of the
measurement: a predicate that looks only for the `$HOME` branch undercounts them badly.
**What was NOT done, and why.** #809 also suggests asserting at session teardown that the real
directory gained no test-shaped records. That is the same racy oracle deferred to the end of the run:
it still reads a directory other processes write, and it still cannot tell a test's record from a
concurrent session's without a field distinguishing them — which `conftest.py` rejected in #776,
because such a field moves the inference problem rather than removing it.
+57 -15
View File
@@ -412,7 +412,9 @@ lines flagged wrongly — and cold review then constructed more of both (`[[ "$x
a backslash continuation, a `>` inside a quoted string). Deleted rather than patched a fifth time,
on the same reasoning as the vocabulary-parity withdrawal above.
**`test_hook_fire_log.py` proves THREE clauses, and its row claims two of them.** Coverage (every
**`test_hook_fire_log.py` proves FOUR clauses, and its row claims two of them.** The fourth is
suite isolation, added in ersatztv#809/#822 and described in the section below; the three that
predate it are as follows. Coverage (every
hook reports that it fired) via `test_a_hook_that_LOSES_its_instrumentation_is_DETECTED`;
transparency (the wrapper changes nothing the harness can see) via
`test_instrumentation_changes_NOTHING_the_harness_can_see` with its mutation proof
@@ -422,20 +424,59 @@ column covers the first two — "stops reporting that it fired, **or whose repor
harness sees**" — and the `Proof ref` column holds one ref because the column holds one, not because
the second is unproven. Placement is proved and unclaimed.
**Its `test_the_suite_does_not_write_to_the_PRODUCTION_log` is narrower than its docstring**, which
says `conftest.py` "must isolate every test, not just this file's". What it checks is that the
fixture set `ETV_HOOK_FIRE_LOG_DIR` *for the test currently running* and that a hook it drives in
its own sandbox does not touch the real log. It cannot see another suite that snapshots
`os.environ` at IMPORT time — before the autouse fixture runs — and hands that stale mapping to its
subprocesses. That suite's hooks then write to `$HOME/.cache/ersatztv/hook-fire/` while every
assertion stays green, because the fire-log library is fail-open. #785 shipped exactly that defect,
inside the file added to prove those very hooks.
**`test_the_suite_does_not_write_to_the_PRODUCTION_log` is a structural check, and deliberately
reads nothing under `$HOME`.** It asserts that the environment a module sees at IMPORT time already
resolves away from `$HOME/.cache/ersatztv/hook-fire/`, and that a hook driven with that snapshot
lands its records in the isolated directory instead. `scripts/tests/conftest.py` supplies the
property in two layers, and they fail differently: `pytest_configure` sets `ETV_HOOK_FIRE_LOG_DIR`
before collection, which is what reaches import-time snapshots and the module- and session-scoped
fixtures no autouse function fixture can cover; and a `Popen` wrapper fails any launch that does
not CARRY the isolated directory — the route left when an environment is built from scratch rather
than derived from `os.environ`. That rule is deliberately "carries an isolated dir" and NOT "is not
the production one": the sink's default has two branches (`$HOME/.cache/...`, and `/tmp/.cache/...`
when `HOME` is unset or empty), so a single-production-path comparison models one and waves the
other through — of the 83 launches here whose environment is built from scratch, 81 carry no `HOME`
either and so land in the branch such a comparison cannot see. A from-scratch environment must
therefore carry the variable. No count of those sites is kept here — it is a hand-written population
nothing derives, and the commit introducing the rule already falsified its own by adding sites in
the tests that prove it. The guard enumerates them: a missing one is a red, not a stale sentence.
Proofs: `test_REMOVING_the_pre_collection_isolation_LETS_an_IMPORT_TIME_snapshot_LEAK` runs both
arms as nested pytest sessions under a fake `HOME`, and
`test_a_launch_NOT_CARRYING_the_isolated_log_dir_FAILS_THE_LAUNCH` drives the wrapper over both
branches, an explicitly-wrong value, and a RELATIVE value made to land on a shared log by the
child's `cwd` — that last one found by cross-family review, which is also why the guard anchors a
relative path at the launch's `cwd=` rather than at its own — each with a negative control.
`test_the_resolver_agrees_with_the_SINKS_OWN_expansion` runs the Python
resolver and `etv_hook_fire_log_dir` over the same environments every run, because a resolver that
disagrees with the sink fails OPEN — it clears a launch the sink then points at the real log.
**Two earlier shapes are recorded because both read as correct and neither is** (ersatztv#809,
ersatztv#822). The guard used to snapshot `st_mtime_ns` across the real log directory and require it
unchanged; that oracle is state every concurrent Claude Code session on the machine writes, so an
unrelated session firing a hook inside the window failed the suite with `a test run modified the
production hook-fire log` — an accusation about the suite when the writer was another process.
Observed on three separate branches, green on every immediate re-run, and corroborated by a reviewer
who found a file there written 13 seconds earlier by someone else. A flake that misattributes is
worse than one that just fails. And an autouse fixture alone isolates too late: it runs at test
setup, after every module in the session has been imported. Measured on the pre-change tree by
instrumenting `subprocess.Popen` for a full run and asking which launches resolve to
`$HOME/.cache/ersatztv/hook-fire`: 83, of which 37 during collection and 44 inside module- or
session-scoped fixtures — both outside any test, and so beyond an autouse function fixture's reach.
The other 2 came from an environment built from scratch with the real `HOME`, the route the second
layer exists for. Instrument `Popen` ALONE: `subprocess.run`, `call` and `check_output` all reach
it, so wrapping `run` as well counts every launch twice and doubles every figure — which is how the
first census here was wrong. That predicate also UNDERCOUNTS the from-scratch route, because such an environment
usually carries no `HOME` either and so resolves to the sink's other default, which it cannot see.
Both agree on the finding that matters: almost none of it was the module-level-snapshot route
ersatztv#809 names.
The reproduction is the part worth keeping: restore a module-level `{**os.environ}` snapshot — or
leave the helper correct and point a single `env=` argument at one — then run that file and count
records for its synthetic session ids. **58 per run**, measured identically on macOS/git 2.55 and
Linux/git 2.47.3. (The accumulated total observed before the fix ran to four figures across many
runs; that is an observation rather than a reproducible measurement, so check the per-run figure.)
runs; that is an observation rather than a reproducible measurement, so check the per-run figure.) Since the pre-collection isolation those 58 land in the session's own
temporary directory, so the figure now measures this file's hook traffic rather than a leak — the
reproduction still shows the mistake, not the consequence it used to have.
The pin is `test_worktree_ownership_guard.py::test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir`,
which asserts the EFFECT — records land in the fixture's dir — rather than the shape of the fix. Its
@@ -443,11 +484,12 @@ predecessor asserted the helper's return value, and cold review showed that leav
unguarded: `_env()` correct, one `env=` reverted, all tests green, records still leaking. Pinning to
a hand-written revert rather than to the property is `verify-against-the-REAL-predecessor`.
It is still narrower than the property: it guards the launch path it drives. A second launcher in
the same file that passed a stale environment would leave it green — measured, 18 records — because
the hooks it drives would still log correctly. Every hook in that file goes through one helper
today, which is what makes it sufficient there. Generalising it is #809, and the reason that is hard
is that the obvious version races against a real session's hooks firing during the run.
It is narrower than the property: it guards the launch path it drives. A second launcher in the
same file that passed a stale environment would leave it green — measured, 18 records to the wrong
directory — because the
hooks it drives would still log correctly. That is why it is a pin on one file and not the general
property; the general property is the two conftest layers above, which do not depend on any suite
remembering to route through a helper.
**That unproven set carries no number here on purpose.** It restated the count, drifted the moment
the BOM guard was regraded, and `test_the_summary_counts_match_the_table` cannot see it — the parser
+4 -3
View File
@@ -42,9 +42,10 @@
# scripts/hook-fire-log.sh report --json
# scripts/hook-fire-log.sh path # where the current session logs
#
# `scripts/tests/test_hook_fire_log.py` is the guard: it derives the hook population from the
# filesystem (never a list — `testing.guard-derives-population-from-source`) and fails if any hook
# is uninstrumented, and it proves stdin, stdout and the exit code survive the wrapper.
# `scripts/tests/test_hook_fire_log.py` is the guard: it derives the hook population from the GIT
# INDEX (never a list, and not a filesystem walk since ersatztv#806 —
# `testing.guard-derives-population-from-source`) and fails if any hook is uninstrumented, and it
# proves stdin, stdout and the exit code survive the wrapper.
# ---------------------------------------------------------------------------------------------
# Shared: where the log lives, and how a record is written
+93 -7
View File
@@ -1,8 +1,9 @@
"""Keep the test suite out of the PRODUCTION hook-fire log (ersatztv#776).
"""Keep the test suite out of the PRODUCTION hook-fire log (ersatztv#776, #809, #822).
Four suites here drive real hooks as subprocesses — `test_merge_consent_exemption.py`,
`test_merge_consent_base_change.py`, `test_prepush_rebase_check_tag_exemption.py`,
`test_pr_changed_files.py` — and they predate the instrumentation. Once every hook records its own
Several suites here drive real hooks as subprocesses. Four of them predate the instrumentation —
`test_merge_consent_exemption.py`, `test_merge_consent_base_change.py`,
`test_prepush_rebase_check_tag_exemption.py` and `test_pr_changed_files.py`; `test_hook_fire_log.py`
and `test_worktree_ownership_guard.py` have joined them since. Once every hook records its own
execution, running `pytest scripts/tests` wrote its synthetic invocations into
`$HOME/.cache/ersatztv/hook-fire/`: 96 `pretooluse-merge-consent` fires including two `deny`s, and
12 `blocked` decisions from `prepush-rebase-check`, none of them a real session.
@@ -13,15 +14,100 @@ answers a different question while looking identical — inference reintroduced
record has no field distinguishing a test invocation from a real one, and adding one would only move
the problem, so the fix is that tests never write to the real log at all.
Autouse and session-independent: a new suite that drives a hook is isolated by existing, without
having to know this file is here.
TWO LAYERS, because one is not enough and the two fail differently.
`pytest_configure` puts an isolated directory into `os.environ` BEFORE any test module is imported.
That is what carries the isolation to suites other than the hook-driving four: a module that
snapshots `{**os.environ}` at import time captures the isolated value, because `pytest_configure`
runs before collection. #785 shipped exactly that snapshot, inside the file added to prove #776's
hooks, and it leaked 58 records per run while every assertion stayed green — the fire-log library is
fail-open by design, so nothing can surface from the hook side. The ordering this relies on was
verified by execution rather than read: a module-level `{**os.environ}` does carry a value set in
`pytest_configure`. The autouse fixture below still re-points the variable per test, so a test that
inspects its own log dir sees only its own records.
The route the first layer cannot see is an environment built from SCRATCH rather than derived from
`os.environ`, carrying no `ETV_HOOK_FIRE_LOG_DIR` at all — so `scripts/hook-fire-log.sh` falls back
to its own default. No snapshot is involved, so nothing about import ordering helps.
`ProductionLogGuard` closes it at the launch itself: a child that does not carry the isolated
directory fails the test that launched it. Its population is every launch that actually happens,
derived at runtime rather than from a list or from source text, so a new suite is covered by
existing — every launch, that is, that goes through `subprocess.Popen` and is the DIRECT child.
`ProductionLogGuard`'s own docstring enumerates what that leaves out (a re-exec, `os.execve`, a
script that unsets the variable itself, a `Popen` bound by name before this hook ran); none occur
here today, and the list is written down so the population is not read as total.
The rule is "CARRIES AN ISOLATED DIRECTORY", not "is not the production one", and the difference is
load-bearing: the sink's default has TWO branches — `$HOME/.cache/...` and, when `HOME` is unset or
empty, `/tmp/.cache/...` — so comparing against a single resolved production path models one and
waves the other through. On the same tree, the launches whose environment is built FROM SCRATCH —
the ones the first layer cannot reach — also number 83 across the helpers this change updates;
81 of those carry no `HOME` either and so RESOLVE TO `/tmp/.cache/ersatztv/hook-fire`, which
is just as shared, just as persistent, and which the predicate above cannot see. (Two different sets
of the same size, said out loud because a later reader would otherwise reconcile them as one.)
Resolve to, not write to: those particular children are `bash`, `git` and a nested pytest that
never source the sink, and that directory does not exist on this machine. The guard is deliberately
conservative about which launches it rejects — see `ProductionLogGuard` — because whether a given
command will reach the sink is not answerable at the launch. Requiring the variable covers both
branches and does not depend on what `HOME` is on the machine running the suite — which is also what
keeps it from firing on a legitimate launch wherever `HOME` is unset or is itself
`/tmp`. Its corollary is that a from-scratch environment must carry the variable; the guard
enumerates those sites, so a missed one is a red rather than a stale sentence here.
WHY NEITHER LAYER OBSERVES THE PRODUCTION DIRECTORY. The obvious check — snapshot it around each
test and compare — is what shipped, and its oracle is state every concurrent session on the machine
writes: that directory is shared by every live Claude Code session, so an unrelated session firing a
hook inside the window reddened the suite with `a test run modified the production hook-fire log` —
an accusation about the suite when the writer was another process (#822). Observed three times on
three different branches, each time green on an immediate re-run. Both layers here are structural
and read nothing under `$HOME`.
"""
from __future__ import annotations
import os
import shutil
import tempfile
import pytest
from scripts.tests.hook_fire_isolation import ENV_VAR, ProductionLogGuard
_GUARD = ProductionLogGuard()
_SESSION_LOG_DIR: str | None = None
def pytest_configure(config):
"""Isolate BEFORE collection, so an import-time `{**os.environ}` snapshot inherits the isolation.
This is also what covers MODULE- and SESSION-scoped fixtures, which the autouse fixture below
cannot reach: they are set up outside any single test, so `os.environ` had no isolated value
while they ran. Measured once on the pre-change tree, asking which launches resolve to
`$HOME/.cache/ersatztv/hook-fire`: 83 — 37 during collection and 44 inside module- or
session-scoped fixtures. The remaining 2 came from an environment built from scratch, which is
the group this layer cannot help with and the reason there is a second one.
INSTRUMENT `Popen` ONLY. `subprocess.run`, `call` and `check_output` all reach `Popen`, so
wrapping `run` as well counts every launch twice and doubles every figure — which is how the
first version of this measurement was wrong.
"""
global _SESSION_LOG_DIR
_SESSION_LOG_DIR = tempfile.mkdtemp(prefix="etv-hook-fire-session-")
os.environ[ENV_VAR] = _SESSION_LOG_DIR
_GUARD.install()
def pytest_unconfigure(config):
_GUARD.uninstall()
if _SESSION_LOG_DIR:
shutil.rmtree(_SESSION_LOG_DIR, ignore_errors=True)
@pytest.fixture(autouse=True)
def isolate_hook_fire_log(tmp_path_factory, monkeypatch):
monkeypatch.setenv("ETV_HOOK_FIRE_LOG_DIR", str(tmp_path_factory.mktemp("hook-fire-log")))
"""Re-point per test, so a test reading its own log dir sees only its own records.
`monkeypatch` reverts to the value in place at setup — the session directory from
`pytest_configure`, never to unset — so the layer above survives every teardown.
"""
monkeypatch.setenv(ENV_VAR, str(tmp_path_factory.mktemp("hook-fire-log")))
+255
View File
@@ -0,0 +1,255 @@
"""Resolve where a subprocess would write its hook-fire records, and require an isolated one.
Split out of `conftest.py` so it can be imported: a test that drives these functions, and a suite
that has to build a subprocess environment from scratch, both need them, and importing `conftest`
under its own name would construct a SECOND `ProductionLogGuard` — whose `inspect.signature` call
would then read the already-wrapped `Popen` and fail at import. A plain module is imported once and
cached, so there is one guard and one production path per session.
See `conftest.py`'s docstring for what the two isolation layers are and why neither observes the
production directory (ersatztv#776, #809, #822).
"""
from __future__ import annotations
import inspect
import os
import subprocess
from collections.abc import Mapping
from pathlib import Path
ENV_VAR = "ETV_HOOK_FIRE_LOG_DIR"
# `scripts/hook-fire-log.sh`'s `etv_hook_fire_log_dir()` is one parameter expansion:
# ${ETV_HOOK_FIRE_LOG_DIR:-${HOME:-/tmp}/.cache/ersatztv/hook-fire}
# Both `:-` defaults are reproduced below rather than restated as a literal path, because a guard
# that hardcodes where the sink writes stops guarding the moment the sink moves — and does so
# silently, by comparing against a directory nothing uses any more. The agreement is not left as an
# argument: `test_hook_fire_log.py::test_the_resolver_agrees_with_the_SINKS_OWN_expansion` runs both
# over the same matrix of environments.
_DEFAULT_SUFFIX = (".cache", "ersatztv", "hook-fire")
_HOME_FALLBACK = "/tmp" # noqa: S108 — mirrors the sink's own `${HOME:-/tmp}`
def resolved_log_dir(env: Mapping[str, str]) -> Path:
"""The directory `etv_hook_fire_log_dir` yields for a child launched with `env`.
Shell's `:-` treats an EMPTY value as absent, so `""` must fall through to the default exactly
as it does there; `env.get(k) or default` is that semantic and `env.get(k, default)` is not.
SCOPE, because the obvious wider claim would be false: this models the log DIRECTORY, not the
final file. `etv_hook_fire_record` writes to `${ETV_HOOK_FIRE_LOG_FILE:-...}`, so an inherited
`ETV_HOOK_FIRE_LOG_FILE` would in principle steer a record anywhere. It cannot in practice —
`etv_hook_fire_begin` recomputes that variable from this directory before any write, on both of
its paths, and nothing exports it — so it is deliberately not modelled here rather than
modelled wrongly. The one path reaching a write WITHOUT `begin` is the `record` CLI subcommand,
which does honour an inherited value; it has no call site in this repo. If a future change stops
recomputing the variable, or a caller starts using `record`, this becomes a real gap. (Cited by
SYMBOL rather than line number: the commit that added this note also shifted every line number
it first carried, by editing the header of the very file it points at.)
"""
return Path(env.get(ENV_VAR) or Path(env.get("HOME") or _HOME_FALLBACK, *_DEFAULT_SUFFIX))
# Resolved from a mapping holding only HOME — what the sink sees when the variable is unset. The
# mapping carries no `ETV_HOOK_FIRE_LOG_DIR`, so this reads the sink's DEFAULT regardless of what
# this session later exports; it does not depend on running before `pytest_configure`.
PRODUCTION_LOG_DIR = resolved_log_dir({"HOME": os.environ.get("HOME", "")})
# BOTH of the sink's default branches, because there are two shared logs and an explicit value can
# name either. `${HOME:-/tmp}` means an empty mapping yields `/tmp/.cache/ersatztv/hook-fire`, which
# is machine-wide, survives every run and is readable with `hook-fire-log.sh report --dir` exactly
# like the `$HOME` one. Requiring the variable (below) closes the OMISSION route for both branches;
# this tuple is what stops the variable being set TO one of them. Where `HOME` is unset the two
# collapse to the same path, which is harmless.
SHARED_LOG_DIRS = (PRODUCTION_LOG_DIR, resolved_log_dir({}))
def _anchored(path: Path, cwd) -> str:
"""Absolute realpath, resolving a RELATIVE value against the CHILD's working directory.
`Popen(cwd=...)` changes the directory the child resolves a relative `ETV_HOOK_FIRE_LOG_DIR`
against, so anchoring at the parent's cwd models the wrong process. Cross-family review found
that as a live bypass: `ETV_HOOK_FIRE_LOG_DIR=.cache/ersatztv/hook-fire` with `cwd=$HOME` lands
exactly on the shared log while a parent-anchored check sees an unrelated path under the repo
and clears it.
THREE cwd shapes, not four. `Popen` accepts str, bytes and `PathLike`; it calls `os.fsencode`,
which rejects an int, so a file DESCRIPTOR is not a launch shape and needs no handling here —
measured on CPython 3.9 and 3.13, and pinned by
`test_the_guard_JUDGES_every_cwd_SHAPE_that_Popen_ACCEPTS`. A previous version of this added a
branch for the int case; the branch was DEAD, because `os.fspath` already raises `TypeError` on
an int, and deleting it left that branch's own test green. `os.fsdecode(os.fspath(...))`
normalises the three shapes that do exist, which is what the FIRST version got wrong — it called
`os.fspath` alone, and `os.path.join` then raised on mixing str with bytes.
realpath, not `==`: `/tmp` is `/private/tmp` on macOS and `$HOME` may itself be a symlink.
"""
value = os.fsdecode(os.fspath(path))
if os.path.isabs(value):
return os.path.realpath(value) # `cwd` cannot change where an absolute path points
base = os.getcwd() if cwd is None else os.fsdecode(os.fspath(cwd))
return os.path.realpath(os.path.join(base, value))
def _decoded(env: Mapping) -> dict[str, str]:
"""Normalise a subprocess environment to str keys and values, ONCE, at the boundary.
`Popen` accepts a bytes-keyed and/or bytes-valued `env` on POSIX, and both reach here from a
legal launch. Unnormalised, a bytes KEY makes the `ETV_HOOK_FIRE_LOG_DIR` lookup miss and the
launch is rejected as carrying no isolated dir — a false positive on a correctly isolated
child — while a bytes VALUE makes `Path(...)` raise `TypeError`. Both are the "red on a
legitimate launch" failure ersatztv#809 explicitly forbids, and they are the same shape as the
bytes-`cwd` regression an earlier round shipped.
Done here rather than at each lookup so there is one place that knows about the two encodings,
and no later reader has to remember which call sites handle which.
"""
return {os.fsdecode(k): os.fsdecode(v) for k, v in env.items()}
def isolation_violation(env: Mapping | None, cwd=None) -> str | None:
"""The message for a child that would log outside this session's isolation, else `None`.
THE RULE IS "CARRIES AN ISOLATED DIRECTORY", NOT "IS NOT THE PRODUCTION ONE", and the difference
is the whole point. `etv_hook_fire_log_dir` has TWO fallback branches — `$HOME/.cache/...` and,
when `HOME` is unset or empty, `/tmp/.cache/...` — so a check that compares against a single
resolved production path models one of them and is blind to the other. Measured on this suite:
of the 83 launches whose environment is built from scratch, 81 carry no `HOME` either, so they
resolve to `/tmp/.cache/ersatztv/hook-fire` — a machine-wide, run-to-run persistent hook-fire
log that a production-path comparison waves straight through. Requiring the variable
covers both branches and every branch a future sink might add, and it does not depend on what
`HOME` happens to be on the machine running the suite — which is also what stops the check
firing on legitimate launches wherever `HOME` is unset or is itself `/tmp`.
`env=None` means the child inherits `os.environ`, which `pytest_configure` has already isolated
— but it is RESOLVED rather than assumed safe, so a test that deletes the variable is caught by
this clause instead of by nothing.
"""
env = _decoded(os.environ if env is None else env)
if not env.get(ENV_VAR):
return (
f"this test launched a subprocess carrying no {ENV_VAR}, so `scripts/hook-fire-log.sh` "
f"falls back to its own default — {resolved_log_dir(env)} — which is a hook-fire log "
"shared between runs and between everyone using this machine. Any hook the child drives "
"appends there, and the fire-log library is fail-open by design, so the leak is "
"invisible from the hook side and every assertion stays green. Build the child's "
f"environment from `os.environ` (conftest.py has already isolated it), or copy {ENV_VAR} "
"into it explicitly if it must be built from scratch."
)
# An explicit value still must not BE a shared log — the variable can be set wrongly as easily
# as it can be omitted, and the likeliest wrong value is the sink's own default transliterated
# into a `dict.get(k, "<default>")`, which is how BOTH branches become reachable again.
target = _anchored(resolved_log_dir(env), cwd)
for shared in SHARED_LOG_DIRS:
# Absolute by construction, so `cwd` cannot change where they point.
root = os.path.realpath(shared)
# CONTAINMENT, not equality: `etv_hook_fire_report` reads its directory with
# `find "$dir" -name '*.jsonl' -type f`, which RECURSES, so a subdirectory of a shared log
# is picked up by `hook-fire-log.sh report --dir <shared>` exactly like a file at its root.
# An equality test calls `$HOME/.cache/ersatztv/hook-fire/sub` clean while records written
# there show up in the report — the same "looks identical, answers a different question"
# failure the whole isolation exists to remove.
if target == root or target.startswith(root + os.sep):
return (
f"this test launched a subprocess whose {ENV_VAR} is inside a shared hook-fire log "
f"({shared}) — one of the two directories `etv_hook_fire_log_dir` falls back to, "
"both of which outlive the run, are shared with everyone using this machine, and "
"are read RECURSIVELY by `hook-fire-log.sh report --dir`. Point it at a directory "
"this test session owns and deletes."
)
return None
class ProductionLogGuard:
"""Resolve every subprocess launch against the sink's rule, at the launch itself.
Only `Popen` is wrapped, because `subprocess.run`, `check_output` and `call` all reach the
module-global `Popen` by name at call time — so wrapping the one covers the four, and a second
wrapper would be a second place to keep in step with the first.
Its population is every launch that actually happens. That is the point: a list of hook-driving
suites, or a scan of their source, would have to be kept in step with the suite, and
`testing.guard-derives-population-from-source` is the record of what that costs.
WHAT IT DOES NOT SEE, stated so the next reader does not have to rediscover it. A child started
outside `Popen` — `os.execve`, `os.posix_spawn`, `os.system` — is not routed through this at
all. Neither is a child that is launched correctly and then re-execs, nor a shell script the
suite writes that unsets `ETV_HOOK_FIRE_LOG_DIR` itself before sourcing the sink: the guard
inspects the environment handed OVER, not what the child does with it afterwards. Nor is a
caller that bound `Popen` BY NAME (`from subprocess import Popen`) before `pytest_configure`
installed this — only a pytest plugin imports that early, and none here does. None of these
occur in this suite today, and each would be a deliberate act rather than the accident this
exists to catch; the accident is an environment assembled without the variable, and that is
visible at the launch.
A test that DELETES the variable from `os.environ` and then launches with `env=None` IS caught,
because `isolation_violation` resolves `os.environ` rather than assuming it safe.
Nor does it check that the directory is a GOOD one, only that it is not a shared hook-fire log:
`ETV_HOOK_FIRE_LOG_DIR=/tmp` or `=$HOME/.cache/ersatztv` passes while writing a session file into
a machine-wide, run-to-run persistent directory. The rule's name — "carries an isolated dir" —
is wider than what is implemented, which is "carries a dir that is not one of the two shared
logs". Nothing does that today. A bytes-keyed or bytes-valued `env` IS handled, at the boundary.
One more residual, measured rather than assumed: containment is a STRING comparison, and macOS
volumes are case-insensitive, so `$HOME/.cache/ErsatzTV/hook-fire` is the same directory as
`$HOME/.cache/ersatztv/hook-fire` and is cleared. Left as a note rather than fixed with an
inode comparison, deliberately: nothing spells it that way, the likeliest wrong value — the
sink's own default transliterated — matches case exactly, and this change has already had two
consecutive review rounds whose finding was created by the previous round's fix, which is this
corpus's stated trigger for subtracting a layer rather than adding one.
"""
def __init__(self) -> None:
self._real_popen = subprocess.Popen
# Derived from the real signature, never a literal index: `env` is a positional parameter of
# `Popen.__init__`, and a hand-counted position would silently start reading `cwd` — or
# nothing — if CPython ever reordered it, leaving the guard passing on every launch.
params = list(inspect.signature(self._real_popen.__init__).parameters)
self._env_pos = params.index("env") - 1 # `self` is not passed at the call site
# `cwd` too, because a RELATIVE log dir resolves against the CHILD's directory, not ours.
self._cwd_pos = params.index("cwd") - 1
self.checked = 0
self.installed = False
def install(self) -> None:
guard = self
real = self._real_popen
class GuardedPopen(real): # type: ignore[misc,valid-type]
# Declared in the class body, not attached afterwards, so it is a real attribute a type
# checker can see — and so `test_a_launch_NOT_CARRYING_...` can assert the guard is
# INSTALLED before concluding anything from its own launch going through.
etv_production_log_guard = guard
def __init__(self, *args, **kwargs):
guard.check(args, kwargs)
super().__init__(*args, **kwargs)
GuardedPopen.__name__ = real.__name__
GuardedPopen.__qualname__ = real.__qualname__
subprocess.Popen = GuardedPopen
self.installed = True
def uninstall(self) -> None:
subprocess.Popen = self._real_popen
self.installed = False
def _argument(self, name: str, pos: int, args: tuple, kwargs: dict):
"""What a `Popen` call passed for `name`, whether by keyword or in its positional slot."""
if name in kwargs:
return kwargs[name]
return args[pos] if len(args) > pos else None
def env_of(self, args: tuple, kwargs: dict):
return self._argument("env", self._env_pos, args, kwargs)
def cwd_of(self, args: tuple, kwargs: dict):
return self._argument("cwd", self._cwd_pos, args, kwargs)
def check(self, args: tuple, kwargs: dict) -> None:
self.checked += 1
violation = isolation_violation(self.env_of(args, kwargs), self.cwd_of(args, kwargs))
if violation is not None:
raise AssertionError(violation)
@@ -30,6 +30,8 @@ from pathlib import Path
import pytest
import yaml
from scripts.tests import hook_fire_isolation
REPO_ROOT = Path(__file__).resolve().parents[2]
# ASSESSED FOR ersatztv#806: this file has NO filesystem-derived population. Its members come from
# the PARSED workflow (`_marked(job)` reads the marked steps out of `_DOC`), which is already an
@@ -602,6 +604,11 @@ GATE_VALUES_IN_THE_WILD = ("false", "", None)
def _env(tmp_path, **extra):
env = {
"PATH": os.environ["PATH"],
# Built from scratch, so no `os.environ` spread carries the isolated hook-fire log dir and
# `scripts/hook-fire-log.sh` would fall back to `${HOME:-/tmp}/.cache/ersatztv/hook-fire` —
# a log shared between runs and between everyone using the machine. Carried explicitly; the
# launch guard in `scripts/tests/conftest.py` rejects a child without it (ersatztv#809).
hook_fire_isolation.ENV_VAR: os.environ[hook_fire_isolation.ENV_VAR],
"GITHUB_WORKSPACE": str(REPO_ROOT),
"RUNNER_TEMP": str(tmp_path),
"GITHUB_JOB": "test",
+10 -1
View File
@@ -81,7 +81,7 @@ import yaml
# population — see `_repo_copy`. Recorded as such in POPULATION_EXEMPT in
# `test_guard_populations_derive_from_git.py`; the exemption lives there, not here, because a marker
# a file grants itself is a kill switch any prose mention can trip.
from scripts.tests import tracked_files
from scripts.tests import hook_fire_isolation, tracked_files
# Both spellings, matching what the converted guards consider the workflow set.
WORKFLOWS = (".gitea/workflows", ("*.yml", "*.yaml"))
@@ -374,6 +374,12 @@ def _run_the_real_scan_body(repo: Path, tmp_path: Path):
env = {
"PATH": os.environ["PATH"],
"HOME": os.environ.get("HOME", str(tmp_path)),
# Built from scratch WITH the real HOME, so `scripts/hook-fire-log.sh` would otherwise
# resolve straight back to `$HOME/.cache/ersatztv/hook-fire` for everything this body runs
# — including the nested pytest. Carried explicitly because no `os.environ` spread does it
# here; this is the from-scratch route that `conftest.py`'s pre-collection isolation cannot
# reach, and the launch guard rejects it (ersatztv#809).
hook_fire_isolation.ENV_VAR: os.environ[hook_fire_isolation.ENV_VAR],
"GITHUB_WORKSPACE": str(repo),
"RUNNER_TEMP": str(tmp_path / "runner"),
"GITHUB_JOB": JOB,
@@ -648,6 +654,9 @@ def _mark_line(step) -> str:
def _env(tmp_path, **extra):
env = {
"PATH": os.environ["PATH"],
# See `_run_the_real_scan_body` — built from scratch, so the isolated hook-fire log dir has
# to be carried explicitly or the sink falls back to a shared log (ersatztv#809).
hook_fire_isolation.ENV_VAR: os.environ[hook_fire_isolation.ENV_VAR],
"GITHUB_WORKSPACE": str(REPO_ROOT),
"RUNNER_TEMP": str(tmp_path),
"GITHUB_JOB": JOB,
@@ -255,8 +255,11 @@ def test_no_derivation_admits_an_untracked_file(label, derive):
Its own `monkeypatch` context, never the shared fixture instance: the function-scoped fixture is
the same object `conftest.py`'s autouse `isolate_hook_fire_log` patched, so calling `undo()` on
it here also unsets `ETV_HOOK_FIRE_LOG_DIR` and silently re-points a later hook-driving test at
the PRODUCTION log — the #776 isolation disarmed from inside the file that argues for proofs.
it here reverts `ETV_HOOK_FIRE_LOG_DIR` too. Since ersatztv#809 that reverts to the SESSION
directory `pytest_configure` set rather than to unset, so it no longer re-points anything at the
PRODUCTION log — but it still collapses THIS test's remaining per-test isolation into the shared
session dir, and reaching into another fixture's patch object to undo it is the habit worth not
having either way.
WHAT REMOVAL CANNOT SEE, so it is not read as more than it is: a source contributing ONLY
untracked members has nothing here to remove, and #778's defect was exactly that shape (an
+406 -18
View File
@@ -32,11 +32,20 @@ from __future__ import annotations
import os
import pty
import re
import shlex
import shutil
import subprocess
import sys
from pathlib import Path
import pytest
from scripts.tests.hook_fire_isolation import (
ENV_VAR,
SHARED_LOG_DIRS,
isolation_violation,
resolved_log_dir,
)
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
@@ -47,6 +56,18 @@ HUSKY = (".husky", ("*",))
HOOKS_DIR = REPO_ROOT / ".claude" / "hooks"
SINK = REPO_ROOT / "scripts" / "hook-fire-log.sh"
# CAPTURED AT IMPORT TIME ON PURPOSE — this is a specimen, not an oversight.
#
# `{**os.environ}` at module level is the shape #785 shipped: a suite that snapshots the environment
# once, at collection, and hands that stale mapping to every subprocess it launches. Because the
# autouse isolation fixture runs at test SETUP, such a snapshot used to predate it and every hook
# driven from it wrote to `$HOME/.cache/ersatztv/hook-fire/` — 58 records per run, with every
# assertion green, since the fire-log library is fail-open by design.
#
# `test_the_suite_does_not_write_to_the_PRODUCTION_log` asserts this specimen is now SAFE, which is
# the cross-suite claim that test has always made in its docstring and never checked in its body.
_IMPORT_TIME_ENV = {**os.environ}
# The three lines a hook must carry. Matched structurally rather than as one frozen blob so that a
# comment reflow does not redden the suite, but the two things that matter — sourcing the single
# shared sink, and calling begin with this hook's own name — are both pinned.
@@ -1102,22 +1123,49 @@ def test_the_session_id_cannot_ESCAPE_the_log_directory(sandbox, tmp_path):
def test_the_suite_does_not_write_to_the_PRODUCTION_log(sandbox):
"""`conftest.py` must isolate every test, not just this file's.
"""`conftest.py` isolates every test, and every MODULE — not just this file, and not just tests.
Before it, running `pytest scripts/tests` put 115 synthetic fires into
`$HOME/.cache/ersatztv/hook-fire/` — 96 merge-consent including two `deny`s — so
`hook-fire-log.sh report` described the test suite while looking exactly like a record of real
sessions. That is the inference problem this change exists to abolish, one layer up.
WHAT THIS ASSERTED BEFORE, and why that had to go. It snapshotted the `st_mtime_ns` of every
`~/.cache/ersatztv/hook-fire/*.jsonl`, drove one hook, and required the two lists equal. That
directory is the production log EVERY live Claude Code session on the machine writes to, so the
oracle was global mutable state: an unrelated session firing a hook inside the window failed
this test with `a test run modified the production hook-fire log` — an accusation about the
suite when the writer was another process. Observed on three separate branches, green on an
immediate re-run each time, and corroborated by an independent reviewer who found a file in that
directory written 13 seconds earlier by someone else (ersatztv#822). The misattribution was the
expensive part: it points the next reader at the suite, and the honest diagnosis costs a re-run
plus a directory listing.
WHAT IT ASSERTS NOW. The same invariant, structurally, reading nothing under `$HOME`.
`_IMPORT_TIME_ENV` is the #785 defect's own shape — the environment as a module sees it at
COLLECTION, before any fixture has run — and a real hook driven with it must land its records in
the isolated directory that snapshot carries. That is the cross-suite claim: isolation has to be
in place before a module is IMPORTED, not merely before a test runs, and the same ordering is
what covers module- and session-scoped fixtures, which no autouse function fixture can reach.
It asserts the EFFECT — records in the isolated dir — rather than the shape of the fix. Pinning
the shape is what `test_worktree_ownership_guard.py` had to correct once already: `_env()` can be
perfectly correct while a call site passes something else.
"""
assert os.environ.get("ETV_HOOK_FIRE_LOG_DIR"), (
"ETV_HOOK_FIRE_LOG_DIR is not set for this test, so any hook it drives writes to the real "
"log. scripts/tests/conftest.py should be setting it for every test."
)
default = Path(os.path.expanduser("~/.cache/ersatztv/hook-fire"))
assert Path(os.environ["ETV_HOOK_FIRE_LOG_DIR"]).resolve() != default.resolve()
# Both violations are BOUND before being asserted on, never called inside the `assert`
# expression: pytest rewrites the expression and prints the repr of every sub-expression, so
# `assert isolation_violation(os.environ) is None` dumps the whole environment — API keys and
# tokens included — into the failure output and from there into a CI log. Measured, not feared.
here = isolation_violation(os.environ)
assert here is None, here
before = sorted(p.stat().st_mtime_ns for p in default.glob("*.jsonl")) if default.exists() else []
root, env = sandbox
# THE CROSS-SUITE CLAUSE. An autouse fixture alone cannot satisfy this: it runs at test setup,
# which is after every module in the session has been imported.
snapshot_violation = isolation_violation(_IMPORT_TIME_ENV)
assert snapshot_violation is None, (
"a module-level `{**os.environ}` snapshot resolves to the production hook-fire log. "
f"conftest.py's `pytest_configure` must set {ENV_VAR} BEFORE collection.\n{snapshot_violation}"
)
snapshot_dir = resolved_log_dir(_IMPORT_TIME_ENV)
before = {p.name for p in snapshot_dir.glob("*.jsonl")} if snapshot_dir.exists() else set()
root, _env = sandbox
hook = root / "prod-check.sh"
hook.write_text(
"#!/usr/bin/env bash\nset -uo pipefail\n"
@@ -1127,14 +1175,26 @@ def test_the_suite_does_not_write_to_the_PRODUCTION_log(sandbox):
)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
input=b'{"session_id":"import-time-snapshot","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env=env,
env=dict(_IMPORT_TIME_ENV),
timeout=60,
)
after = sorted(p.stat().st_mtime_ns for p in default.glob("*.jsonl")) if default.exists() else []
assert before == after, "a test run modified the production hook-fire log"
# ANTI-VACUITY. Without this the test passes when the hook logged NOTHING AT ALL — which is also
# what a broken sink looks like, and it would then be asserting the absence of a leak by
# asserting the absence of any record.
after = {p.name for p in snapshot_dir.glob("*.jsonl")} if snapshot_dir.exists() else set()
# The EXACT record, not merely "a new file": the log file is named for the payload's session id,
# so `after > before` would also be satisfied by an `unknown-session.jsonl` written because
# payload parsing broke — a hook that logged the wrong thing, passing a test about isolation.
assert "import-time-snapshot.jsonl" in after - before, (
f"driving a hook with the import-time snapshot did not add `import-time-snapshot.jsonl` to "
f"{snapshot_dir} (new files: {sorted(after - before)}). Either the instrumentation stopped "
"firing — in which case this test proves nothing — or those records went somewhere else, "
"and the only somewhere else is a shared log this suite must never write."
)
def test_DELETING_the_replay_makes_the_differential_go_RED(sandbox, positives, tmp_path):
@@ -1494,7 +1554,12 @@ def test_an_INVALID_UTF8_byte_in_a_decision_is_still_classified(tmp_path, locale
'printf \'{"hookSpecificOutput":{"permissionDecision":"deny",'
'"permissionDecisionReason":"caf\\xe9"}}\'\n'
)
env = {"PATH": os.environ["PATH"], "HOME": os.environ["HOME"], "ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
# HOME copied only if present, like the launch-guard test below: an environment without it is
# legal (the sink has a `${HOME:-/tmp}` default) and indexing would redden this on a valid
# checkout for a reason unrelated to what it tests. This is the twin of the site fixed there.
env = {"PATH": os.environ["PATH"], "ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
if os.environ.get("HOME"):
env["HOME"] = os.environ["HOME"]
p = subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
@@ -1653,3 +1718,326 @@ def test_the_field_helper_does_not_LEAK_into_the_hooks_namespace(sandbox):
)
assert b"LEAKED" not in p.stdout, f"a helper leaked into the hook's namespace: {p.stdout!r}"
assert b"CLEAN" in p.stdout
def test_the_resolver_agrees_with_the_SINKS_OWN_expansion(tmp_path):
"""`resolved_log_dir` reimplements one shell expansion — so run both and compare, every run.
It is the guard's only model of where a child will write, so a disagreement fails OPEN: the
guard clears a launch, and the sink then points it at the real log. The two `:-` defaults are
exactly where a Python transliteration goes subtly wrong, because `dict.get(k, default)` returns
`""` for an empty value and shell's `${k:-default}` does not — which is why the empty cases are
here rather than only the obvious ones.
A differential against the real `scripts/hook-fire-log.sh`, not against a restatement of it: a
test that asserted the resolver equals a hardcoded path would agree with itself forever after
the sink moved.
"""
cases = [
{"HOME": str(tmp_path), ENV_VAR: str(tmp_path / "explicit")},
{"HOME": str(tmp_path)},
{"HOME": str(tmp_path), ENV_VAR: ""}, # empty is ABSENT to `:-`
{"HOME": ""}, # ... and so `${HOME:-/tmp}` takes /tmp
{},
# Normalisation: shell concatenates strings, `Path` collapses them. These agree on the
# DIRECTORY and differ in spelling — `/tmp/x/` vs `/tmp/x`, `$HOME//.cache` vs `$HOME/.cache`
# — which is exactly why the comparison below is the one the guard makes and not `==`.
{"HOME": str(tmp_path), ENV_VAR: str(tmp_path / "explicit") + "/"},
{"HOME": str(tmp_path) + "/"},
{"HOME": str(tmp_path), ENV_VAR: str(tmp_path) + "//double"},
]
for case in cases:
# The case is applied INSIDE the child, not through its launch environment.
# `etv_hook_fire_log_dir` reads the shell's environment at call time either way, so the
# differential is unchanged — but the launch itself then carries the isolated log dir like
# every other launch in the suite, instead of this test needing the guard to make an
# exception for it. (Several cases deliberately have no `ETV_HOOK_FIRE_LOG_DIR` at all,
# which is precisely what the guard exists to reject.)
setup = "\n".join(
f"export {var}={shlex.quote(case[var])}" if var in case else f"unset {var}" for var in ("HOME", ENV_VAR)
)
res = subprocess.run(
["bash", "-c", f'{setup}\n. "{SINK}"\netv_hook_fire_log_dir'],
capture_output=True,
text=True,
env={"PATH": os.environ["PATH"], ENV_VAR: os.environ[ENV_VAR]},
timeout=60,
)
assert res.returncode == 0, f"could not source the sink for {case}: {res.stderr}"
# SAME DIRECTORY, not same string. `os.path.realpath` is the comparison `isolation_violation`
# itself makes, so this asserts the contract the guard depends on; requiring byte equality
# would redden on a trailing slash that changes nothing about where the records land, and a
# guard nobody believes is the failure this file exists to avoid. A genuine divergence — a
# DIFFERENT directory — still fails, because realpath does not collapse those.
assert os.path.realpath(res.stdout) == os.path.realpath(resolved_log_dir(case)), (
f"the Python resolver and `etv_hook_fire_log_dir` resolve to DIFFERENT directories for "
f"{case}: the sink says {res.stdout!r}, resolved_log_dir says "
f"{str(resolved_log_dir(case))!r}. The guard would clear a launch the sink then points "
"somewhere else."
)
def test_a_launch_NOT_CARRYING_the_isolated_log_dir_FAILS_THE_LAUNCH():
"""The launch guard, driven end to end — the route pre-collection isolation cannot reach.
An environment assembled from scratch rather than derived from `os.environ` carries no
`ETV_HOOK_FIRE_LOG_DIR`, so the sink falls back to its own default. No snapshot is involved, so
nothing about import ordering helps. This is not hypothetical: three suites build exactly that
environment for the CI-shaped children they run, and they were found by MEASURING every launch
the suite makes, not by reading it.
BOTH FALLBACK BRANCHES ARE PINNED, because a guard that compared against one resolved production
path would pass the other. `${HOME:-/tmp}` means a child with no `HOME` lands in
`/tmp/.cache/ersatztv/hook-fire` — just as shared, just as persistent, and on this suite that was
the majority case: of the 83 from-scratch launches, 81 also carry no `HOME`.
The negative control is the last case — a guard that rejected every launch would satisfy all
three assertions above it and be useless.
"""
guard = getattr(subprocess.Popen, "etv_production_log_guard", None)
assert guard is not None, (
"the launch guard is not installed, so this test would pass by never being consulted. "
"conftest.py's `pytest_configure` should have installed it."
)
# INSTALLED IS NOT CONSULTED — but state precisely what this proves, because the obvious
# reading is wider than the fact. `checked` counts launches the wrapper has already seen, and
# this module's own `@pytest.mark.parametrize(... hook_scripts())` decorators shell out to
# `git ls-files` at IMPORT, so it is >= 2 even when this test is selected alone. What that
# establishes is that installation happened BEFORE this module was collected — i.e. the wrapper
# is in the path of launches made outside this test. Run alone, those are this module's own
# collection-time `git ls-files` calls, so it does order installation before collection; in a
# whole-suite run the counted launches may all belong to other modules, and it establishes only
# that the wrapper is consulted at all. It does NOT establish whole-suite coverage — nothing
# here could, since a launch that never reaches `Popen` is invisible by construction.
assert guard.checked > 0, (
"the launch guard has been consulted for no launch at all before this test, so it was "
"installed after this module was collected and cannot have seen collection-time launches"
)
def launch(env, cwd=None):
return subprocess.run(["bash", "-c", "true"], env=env, cwd=cwd, capture_output=True, timeout=60)
# HOME is COPIED IF PRESENT rather than read with `os.environ["HOME"]`: an environment without
# it is legal — the sink has a `${HOME:-/tmp}` default precisely for that — and indexing would
# raise KeyError, reddening this test on a valid checkout for a reason unrelated to the guard.
scratch = {"PATH": os.environ["PATH"]}
if os.environ.get("HOME"):
scratch["HOME"] = os.environ["HOME"]
# (1) the `$HOME` branch
with pytest.raises(AssertionError, match="carrying no ETV_HOOK_FIRE_LOG_DIR"):
launch(scratch)
# (2) the `/tmp` branch — no HOME either. This is the one a production-path comparison misses.
with pytest.raises(AssertionError, match="carrying no ETV_HOOK_FIRE_LOG_DIR"):
launch({"PATH": os.environ["PATH"]})
# (3) the variable set, and set WRONG. Omitting it is not the only way to reach a shared log,
# and BOTH shared logs are rejected — the likeliest wrong value is the sink's own default
# transliterated into a `dict.get(k, "<default>")`, which reaches whichever branch that default
# was copied from. Pinning only the `$HOME` one would leave the `/tmp` one clear.
for shared in SHARED_LOG_DIRS:
with pytest.raises(AssertionError, match="inside a shared hook-fire log"):
launch({**scratch, ENV_VAR: str(shared)})
# (4) a SUBDIRECTORY of a shared log. `etv_hook_fire_report` reads its directory with a
# recursive `find`, so records here are picked up by `hook-fire-log.sh report --dir <shared>`
# exactly like ones at its root — verified by driving a real hook into a nested directory and
# seeing it counted. An equality test would call this clean.
with pytest.raises(AssertionError, match="inside a shared hook-fire log"):
launch({**scratch, ENV_VAR: str(SHARED_LOG_DIRS[0] / "sub")})
# (5) the variable set RELATIVE, with a `cwd` that makes it land on a shared log. Found by
# cross-family review as a live bypass: a relative value resolves against the CHILD's directory,
# so a guard anchoring it at the parent's sees an unrelated path under the repo and clears it.
# The guard raises before `Popen` runs anything, so this launch never enters that directory.
shared = SHARED_LOG_DIRS[0]
with pytest.raises(AssertionError, match="inside a shared hook-fire log"):
launch({**scratch, ENV_VAR: ".cache/ersatztv/hook-fire"}, cwd=shared.parent.parent.parent)
# (6) NEGATIVE CONTROL: carrying the isolated dir, the same launch must go through.
ok = launch({**scratch, ENV_VAR: os.environ[ENV_VAR]})
assert ok.returncode == 0, "the guard rejected a correctly-isolated launch"
# The specimen the mutation proof below runs in a nested pytest: #785's shape, reduced to the one
# thing that matters. The hook it drives is written beside the sandbox rather than inline here, so
# this source carries no nested escaping.
_SPECIMEN_MODULE = """\
import os
import subprocess
from pathlib import Path
_ENV = {**os.environ} # snapshotted at IMPORT, before any fixture has run — this is the defect
HOOK = Path(__file__).resolve().parents[2] / "specimen-hook.sh"
def test_specimen_drives_a_hook():
subprocess.run(
["bash", str(HOOK)],
input=b'{"session_id":"specimen"}',
capture_output=True,
env=_ENV,
timeout=60,
)
"""
def test_the_guard_JUDGES_every_env_and_cwd_SHAPE_that_Popen_ACCEPTS(tmp_path):
"""`env` and `cwd` each have several legal shapes; the guard must judge, not crash, on all.
`Popen` accepts `cwd` as str, bytes or `PathLike`. The first version of the relative-path
anchoring called `os.fspath(cwd)` and then `os.path.join`, which raises `TypeError` on mixing
str with bytes — a guard that raises `TypeError` on a legitimate launch is not a stricter guard,
it is a broken one, and the failure names the wrong thing entirely.
THE PREMISE IS PINNED, not assumed. A file descriptor is NOT a fourth shape: `Popen` calls
`os.fsencode(cwd)`, which rejects an int. A previous version of the guard added a branch for
that case; the branch was dead — `os.fspath` already raises on an int — and deleting it left its
own test green, so it was removed rather than proved. The assertion below is what makes that
removal safe: if a future CPython starts accepting a descriptor, this goes red and says so.
"""
# A RELATIVE value, so the anchoring is actually REACHED. An absolute one short-circuits before
# `cwd` is consulted at all — an earlier draft of this test used one and passed identically for
# every shape, including shapes that crashed the guard the moment anything relative arrived.
relative = {ENV_VAR: "isolated/log"}
for cwd in (str(tmp_path), tmp_path, os.fsencode(str(tmp_path))):
assert isolation_violation(relative, cwd) is None, (
f"a relative log dir under {tmp_path} was rejected for cwd={cwd!r} — it resolves inside "
"the test's own directory, so this is the guard failing rather than judging"
)
isolated = {ENV_VAR: str(tmp_path / "log")}
assert isolation_violation(isolated, None) is None, "an absolute isolated dir was rejected"
# AND THE ANCHOR MUST BIND TO THE LAUNCH'S cwd, for every shape — not merely fail to crash.
# The same relative value judged against two different `cwd`s must give two different verdicts;
# otherwise `cwd` is being ignored, which is the round-3 defect, and the shape loop above would
# pass while it was.
shared_parent = SHARED_LOG_DIRS[0].parent.parent.parent
for shape in (str, Path, os.fsencode):
onto_shared = shape(str(shared_parent))
assert isolation_violation({ENV_VAR: ".cache/ersatztv/hook-fire"}, onto_shared) is not None, (
f"a relative log dir was cleared for cwd={onto_shared!r} even though it resolves onto a "
"shared log — the anchor is not binding to the launch's cwd for this shape"
)
assert isolation_violation({ENV_VAR: ".cache/ersatztv/hook-fire"}, shape(str(tmp_path))) is None, (
f"the same relative log dir was rejected for cwd={tmp_path} — the anchor is not binding "
"to the launch's cwd, so both verdicts come from somewhere else"
)
# THE ENV HAS SHAPES TOO. `Popen` accepts a bytes-keyed and/or bytes-valued environment on
# POSIX (verified: both launch). Unnormalised, a bytes KEY makes the lookup miss and the launch
# is rejected as carrying no isolated dir; a bytes VALUE makes `Path(...)` raise. Both are reds
# on a correctly isolated child.
log = str(tmp_path / "log")
for env in (
{os.fsencode(ENV_VAR): os.fsencode(log), b"PATH": b"/usr/bin"},
{ENV_VAR: os.fsencode(log), "PATH": "/usr/bin"},
):
assert isolation_violation(env) is None, (
f"a correctly isolated child was rejected for a legal env encoding: {env!r}"
)
# ...and the decoding must not blunt the guard: the same encoding pointed at a shared log fails.
assert isolation_violation({os.fsencode(ENV_VAR): os.fsencode(str(SHARED_LOG_DIRS[0]))}) is not None, (
"normalising the env encoding also stopped the guard seeing a shared log through it"
)
fd = os.open(str(tmp_path), os.O_RDONLY)
try:
with pytest.raises(TypeError):
subprocess.run(
["bash", "-c", "true"],
cwd=fd,
env={"PATH": os.environ["PATH"], **isolated},
capture_output=True,
timeout=60,
)
finally:
os.close(fd)
def test_REMOVING_the_pre_collection_isolation_LETS_an_IMPORT_TIME_snapshot_LEAK(tmp_path):
"""THE MUTATION PROOF FOR THE CROSS-SUITE CLAUSE — the #785 defect, reproduced and neutralised.
Both arms are a real nested pytest over an identical sandbox with a FAKE `HOME`, so nothing here
touches the machine's own log. The specimen module is the #785 shape verbatim: `{**os.environ}`
at module level, handed to a hook subprocess.
ONE clause differs between the arms — `os.environ[ENV_VAR] = _SESSION_LOG_DIR` in
`pytest_configure`. Remove it and the specimen's records land in the fake production directory;
restore it and they do not. That is what makes the pre-collection assignment load-bearing rather
than decorative, and it is the red #809 asks to witness.
WHAT IT NO LONGER TRAVERSES, since the fake directory is handed over explicitly: the sink's
`${ETV_HOOK_FIRE_LOG_DIR:-${HOME:-/tmp}/…}` fallback. The mutant arm now leaks by INHERITANCE
rather than by fallback, so deleting that expansion outright leaves both arms unchanged —
measured. That is covered instead by
`test_the_resolver_agrees_with_the_SINKS_OWN_expansion`, which reddens on the same deletion.
The proof remains sensitive to the weakening that matters here: replacing the assignment with
`os.environ.setdefault(...)` makes the control arm leak.
THE MEASUREMENT ARRANGEMENT, stated because it is a deliberate weakening: the launch guard is
disabled in BOTH arms. Left on, it would reject the specimen's launch in the mutant arm and the
comparison would be about the guard rather than about the clause under test. Both arms are
otherwise byte-identical, so the difference measured is the clause's alone.
"""
real_conftest = (REPO_ROOT / "scripts" / "tests" / "conftest.py").read_text()
install_clause = " _GUARD.install()\n"
isolate_clause = " os.environ[ENV_VAR] = _SESSION_LOG_DIR\n"
for clause in (install_clause, isolate_clause):
assert real_conftest.count(clause) == 1, (
f"the clause {clause!r} has moved or changed in conftest.py, so this proof no longer "
"targets it. RETARGET it rather than loosening the match — a mutation that silently "
"stops mutating proves nothing, which is the failure class this file exists to catch."
)
def arm(name: str, conftest_text: str) -> set[str]:
sb = tmp_path / name
(sb / "scripts" / "tests").mkdir(parents=True)
(sb / "scripts" / "__init__.py").write_text("")
(sb / "scripts" / "tests" / "__init__.py").write_text("")
shutil.copy(SINK, sb / "scripts" / "hook-fire-log.sh")
(sb / "specimen-hook.sh").write_text(
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{sb}/scripts/hook-fire-log.sh"\n'
'etv_hook_fire_begin specimen "" capture || true\n'
"input=$(cat)\n"
)
shutil.copy(REPO_ROOT / "scripts" / "tests" / "hook_fire_isolation.py", sb / "scripts" / "tests")
(sb / "scripts" / "tests" / "conftest.py").write_text(conftest_text)
(sb / "scripts" / "tests" / "test_specimen.py").write_text(_SPECIMEN_MODULE)
home = sb / "home"
home.mkdir()
leaked = home / ".cache" / "ersatztv" / "hook-fire"
res = subprocess.run(
[sys.executable, "-m", "pytest", "-q", "scripts/tests/test_specimen.py"],
cwd=str(sb),
capture_output=True,
text=True,
# The fake production dir is handed over EXPLICITLY, and `HOME` points at the same tree
# as a backstop. Passing it explicitly is what lets this launch satisfy the launch guard
# like every other one, and it sharpens the proof rather than weakening it: the mutant
# arm inherits this value because its `pytest_configure` no longer overwrites it, the
# control arm does not because its does. The clause under test IS that overwrite.
env={"PATH": os.environ["PATH"], "HOME": str(home), ENV_VAR: str(leaked)},
timeout=300,
)
assert res.returncode == 0, f"the {name} arm's specimen run did not pass:\n{res.stdout}\n{res.stderr}"
return {f.name for f in leaked.glob("*.jsonl")} if leaked.exists() else set()
mutant = arm("mutant", real_conftest.replace(install_clause, "").replace(isolate_clause, ""))
control = arm("control", real_conftest.replace(install_clause, ""))
# The mutant arm is also this proof's anti-vacuity check: if the specimen logged nowhere, the
# control arm's emptiness would be about a dead hook rather than about the isolation.
assert "specimen.jsonl" in mutant, (
"removing the pre-collection isolation did NOT make an import-time snapshot leak, so this "
f"proof establishes nothing about that clause. Fake-home log dir held: {sorted(mutant)}"
)
assert control == set(), (
"an import-time `{**os.environ}` snapshot still reaches the production log with the "
f"pre-collection isolation in place — it leaked {sorted(control)}. That is #785's defect, "
"and the cross-suite claim in this file is false."
)
+4
View File
@@ -31,6 +31,7 @@ from pathlib import Path
import pytest
from scripts.tests import hook_fire_isolation
from scripts.tests.tracked_files import tracked_paths
REPO_ROOT = Path(__file__).resolve().parents[2]
@@ -4084,6 +4085,9 @@ def test_the_dropped_step_guard_BEHAVIOURALLY_fails_without_the_marker_and_passe
)
env = {
"PATH": os.environ["PATH"],
# Built from scratch, so the isolated hook-fire log dir is carried explicitly rather than
# letting the sink fall back to a shared log (ersatztv#809).
hook_fire_isolation.ENV_VAR: os.environ[hook_fire_isolation.ENV_VAR],
"RUNNER_TEMP": str(tmp_path),
"GITHUB_RUN_ID": "424242",
"GITHUB_RUN_ATTEMPT": "7",
+45 -26
View File
@@ -37,6 +37,8 @@ import os
import subprocess
from pathlib import Path
from scripts.tests.hook_fire_isolation import resolved_log_dir
REPO_ROOT = Path(__file__).resolve().parents[2]
GUARD = REPO_ROOT / ".claude" / "hooks" / "pretooluse-worktree-guard.sh"
MARKER_HOOK = REPO_ROOT / ".claude" / "hooks" / "posttooluse-worktree-marker.sh"
@@ -56,16 +58,22 @@ def _env() -> dict:
that fallback finds nothing. Without the pin the mutant differs from the subject in a second way
and the comparison stops being about the mutated clause.
And it is a FUNCTION rather than a module-level dict because `conftest.py`'s autouse
And it is a FUNCTION rather than a module-level dict because the autouse
`isolate_hook_fire_log` fixture monkeypatches `ETV_HOOK_FIRE_LOG_DIR` into `os.environ` at test
setup — which happens AFTER this module is imported. A `{**os.environ}` snapshot taken at import
time captures the environment as it was before the fixture ran, so every hook subprocess writes
to the REAL `$HOME/.cache/ersatztv/hook-fire/` log instead of the fixture's tmp dir. That is not
untidiness: it is #776's defect reintroduced in the file that is meant to prove #776's hooks,
and it corrupts the `hook-fire-log.sh report` surface this repo cites as the observability claim
for every guard still graded NONE. The reproduction, rather than a figure whose evidence has
since been deleted: reintroduce the snapshot and run this file, then count records for the two
synthetic session ids below — 58 per run, on macOS and Linux alike.
setup — which happens AFTER this module is imported, so a `{**os.environ}` snapshot taken at
import time carries the SESSION value and never any per-test one. `conftest.py`'s
`pytest_configure` now also sets the variable before collection, so such a snapshot no longer
reaches the REAL `$HOME/.cache/ersatztv/hook-fire/` log (ersatztv#809) — but it would still pin
every subprocess to one directory for the whole session, which is not what this file wants.
WHAT THAT USED TO COST, kept because it is why the helper is shaped this way and not as a live
warning: before the pre-collection isolation, such a snapshot predated the fixture entirely and
its hooks appended to the REAL `$HOME/.cache/ersatztv/hook-fire/` — #776's defect reintroduced
in the file meant to prove #776's hooks, corrupting the `hook-fire-log.sh report` surface this
repo cites as the observability claim for every guard still graded NONE. Reproduce by
reintroducing the snapshot and counting records for the two synthetic session ids below: 58 per
run, on macOS and Linux alike. Those 58 now land in the session's own temporary directory
instead, so the figure is a measure of this file's hook traffic and no longer of a leak.
"""
return {**os.environ, "CLAUDE_PROJECT_DIR": str(REPO_ROOT)}
@@ -159,11 +167,12 @@ def test_the_subprocess_env_CARRIES_the_isolated_hook_fire_log_dir():
"""The hooks these tests drive must log to the fixture's dir, never the production one.
`conftest.py`'s autouse `isolate_hook_fire_log` monkeypatches `ETV_HOOK_FIRE_LOG_DIR` into
`os.environ` at test setup. Anything that snapshots `os.environ` at IMPORT time captures the
value from before the fixture ran and silently defeats it — the hooks then append to
`$HOME/.cache/ersatztv/hook-fire/`, which is #776's defect reintroduced inside the file that
proves #776's hooks, corrupting the one surface this repo cites as the observability claim for
every guard still graded NONE.
`os.environ` at test setup, and its `pytest_configure` sets the same variable before collection.
Since ersatztv#809 a snapshot taken at IMPORT time therefore carries the session directory, not
a stale pre-isolation value, so it no longer reaches `$HOME/.cache/ersatztv/hook-fire/` — the
shape that was #776's defect reintroduced inside the file meant to prove #776's hooks. What it
would still do is pin every subprocess in this file to one directory for the whole session,
which is why `_env()` remains a function.
It is invisible from the outside: the tests pass either way, because the fire-log library is
fail-open by design. So it needs its own assertion.
@@ -173,12 +182,15 @@ def test_the_subprocess_env_CARRIES_the_isolated_hook_fire_log_dir():
assert isolated, "the autouse isolation fixture did not run; conftest.py is not being loaded"
assert env.get("ETV_HOOK_FIRE_LOG_DIR") == isolated, (
"the subprocess environment does not carry the isolated log dir, so every hook driven by "
"this file is writing into the production hook-fire log. Build the env per call; do not "
"this file is not logging where the fixture put it. Build the env per call; do not "
f"snapshot os.environ at import time. env has {env.get('ETV_HOOK_FIRE_LOG_DIR')!r}"
)
# The path `hook-fire-log.sh` falls back to when ETV_HOOK_FIRE_LOG_DIR is unset, derived the
# same way it derives it rather than restated as a literal.
production = Path(os.environ.get("HOME", "/tmp")) / ".cache" / "ersatztv" / "hook-fire" # noqa: S108 — mirrors hook-fire-log.sh's own ${HOME:-/tmp}
# The path `hook-fire-log.sh` falls back to when ETV_HOOK_FIRE_LOG_DIR is unset — taken from the
# SHARED resolver rather than transliterated again here. The hand-written version this replaces
# used `os.environ.get("HOME", "/tmp")`, which returns `""` for an empty HOME where the shell's
# `${HOME:-/tmp}` takes `/tmp`: the exact mistake `resolved_log_dir`'s docstring exists to warn
# about, sitting two files from the warning.
production = resolved_log_dir({"HOME": os.environ.get("HOME", "")})
assert Path(isolated).resolve() != production.resolve(), (
f"the 'isolated' log dir IS the production one ({production}), so the fixture is isolating "
"nothing and this test would pass while the leak continued"
@@ -191,8 +203,10 @@ def test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir(tmp_path):
`test_the_subprocess_env_CARRIES_...` above checks `_env()`'s return value, and that is not the
same claim: `_env()` can be perfectly correct while a call site passes something else. Cold
review demonstrated exactly that — restore the module-level snapshot and change one `env=_env()`
back to `env=_ENV`, and all thirteen tests pass while 54 records leak into the real log. The
guard was pinned to the shape of the fix instead of to the property, which is
back to `env=_ENV`, and every test in this file passes while 54 records go somewhere it did not
choose. Before ersatztv#809's pre-collection isolation that somewhere was the REAL log; it is now
the session directory, so the guard still catches the mistake but the consequence is contained.
The guard was pinned to the shape of the fix instead of to the property, which is
`verify-against-the-REAL-predecessor`: a hand-written revert is not the code a future tidy-up
produces.
@@ -202,11 +216,15 @@ def test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir(tmp_path):
ITS SCOPE, stated because the first version of this docstring claimed more than it delivers: it
guards THE LAUNCH PATH IT DRIVES, not the file. Cold review demonstrated the gap — add a second
launcher alongside `_run()` that passes a stale snapshot and point the mutation tests at it, and
this test stays green while 18 records leak, because the hooks IT drives still log correctly.
this test stays green while 18 records go to the session dir rather than the fixture's, because
the hooks IT drives still log correctly. (Before ersatztv#809 that somewhere was the REAL log.)
Every hook in this file goes through `_run()` today, which is what makes the guard sufficient
HERE and not a general property. The general form is a suite-level check, tracked in #809; the
reason it is hard is that the obvious version — diff the production log around each test — races
against a real session's hooks firing concurrently.
HERE and not a general property. The general form now lives in `conftest.py` (ersatztv#809):
`pytest_configure` isolates before collection, so an import-time snapshot cannot be stale, and a
`Popen` wrapper rejects any launch that does not CARRY an isolated log dir — the whitelist, not a
comparison against the production path, which would model only one of the sink's two defaults. The obvious
version — diff the production log around each test — was tried and removed: it races against a
real session's hooks firing concurrently, and blames the suite when it loses (ersatztv#822).
"""
isolated = Path(os.environ["ETV_HOOK_FIRE_LOG_DIR"])
before = {p.name for p in isolated.glob("*.jsonl")} if isolated.exists() else set()
@@ -221,8 +239,9 @@ def test_driving_a_hook_LANDS_its_records_in_the_ISOLATED_dir(tmp_path):
after = {p.name for p in isolated.glob("*.jsonl")} if isolated.exists() else set()
assert after > before, (
f"driving two hooks added no record to the isolated log dir {isolated}. Either the "
"instrumentation stopped firing, or these hooks are logging somewhere else — and the only "
"somewhere else is the production log this file must never touch"
"instrumentation stopped firing, or these hooks are logging somewhere else — since "
"ersatztv#809 that is the session dir rather than the production log, but it is still not "
"where this file put it"
)