feat(780): commit a ruff config and enforce it in CI (#813)
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 28s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m29s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 5m54s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 5m59s
Build ErsatzTV Image / Build & push image (amd64) (push) Failing after 15s

Python lint here was a property of the operator's laptop: the global instructions
say to run ruff, no workflow ran it, and with no committed config ruff fell back
to whichever ~/.config/ruff/ruff.toml the machine happened to have.

- ruff.toml at the root, pinned ruff==0.12.11 in the script-tests job.
- Both lint steps pass an EXPLICIT population from `git ls-files` with
  `--no-force-exclude`, never `ruff check .` — an `exclude` empties a
  discovery-based run into a GREEN one (top level empties both commands, [lint]
  empties check, [format] empties format --check), and `ruff check .` over zero
  files exits 0 with only a stderr warning. Guarded by an empty-population arm.
- Tree clean: 74 findings at 706674272, 57 fixed in code, 17 per-site noqa with
  reasons inline. S105 deliberately per-site, not a directory blanket. RUF100
  selected so a suppression that suppresses nothing is itself a finding.
- pyright stays ungated; reasoning in the record.

Both steps witnessed red on the runner against the shipped bodies: run 2173 job
9176 (ruff check) and run 2170 job 9163 (ruff format --check).

Docs: new record ci.python-lint-ruff-config-committed, ci.script-tests-job
cross-ref, docs/ci-cd.md (also correcting a stale ~190-tests/~10s figure to the
measured 773 tests / ~4.5 min), docs/defect-shapes-773.md §5.2 resolved.

fixes #780

Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
This commit was merged in pull request #813.
This commit is contained in:
2026-08-22 00:33:18 +00:00
committed by timothy
parent 6a4265d81d
commit d4c72697f2
27 changed files with 1737 additions and 996 deletions
+85 -15
View File
@@ -359,7 +359,7 @@ jobs:
[ "$failed" -eq 0 ]
script-tests:
name: Script tests (pytest)
name: Script lint and tests (ruff + pytest)
runs-on: small
if: github.event_name == 'pull_request'
steps:
@@ -369,6 +369,90 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: '3.x'
# Preflight, not an install (ersatztv#390 removed run-time `apt-get` from CI on purpose).
# Two consumers need `git`: the lint steps below derive their population from `git ls-files`,
# and test_post_review_verdict.py / test_merge_consent_exemption.py exec the REAL
# post-review-verdict.sh / pretooluse-merge-consent.sh. `curl` those tests shim on PATH; `jq`
# and `git` they do NOT. It stays AHEAD of the lint steps, not merely ahead of pytest: without
# it, a missing git reaches the lint steps as an empty population, which they report as a
# population problem. One actionable line beats a misdirected one, and beats the wall of
# unattributable assertion failures the suite produces without git.
- name: Preflight external tools
run: |
if ! command -v git >/dev/null 2>&1; then
echo "::error::script-tests needs git on PATH but it is absent. The lint steps derive" \
"their population from it and the suite execs real shell scripts that use it." \
"Bake it into the runner image rather than apt-get installing here (ersatztv#390)."
exit 1
fi
echo "Preflight OK: $(git --version)"
# ersatztv#780. Lint runs EARLY — after the git preflight it depends on, but before the test
# dependencies, the jq preflight and the ~4-minute pytest run. A style red therefore arrives in
# seconds, and, more importantly, the lint does not sit behind `Preflight jq version`: that is
# an `--expect` tripwire, so a runner jq bump would take the lint dark for as long as the jq
# contract is broken, under a red that says "jq".
#
# The version is PINNED: an unpinned ruff makes the verdict a function of whenever the job ran
# — the same environment-divergence the committed ruff.toml exists to close. Bumping it is a
# deliberate PR (new rules may fire), exactly like the jq pin below. `pytest`/`pyyaml` are
# deliberately NOT pinned: a pytest release does not add assertions to your suite, a ruff
# release adds rules to your lint.
- name: Install ruff
run: python3 -m pip install --disable-pip-version-check --quiet 'ruff==0.12.11'
# POPULATION. Both steps lint an EXPLICIT list from `git ls-files`, never `ruff check .`, and
# pass `--no-force-exclude`. Measured with ruff 0.12.11 and `exclude = ["scripts/**"]` — a
# per-FILE pattern, because `exclude` matches per file: a bare `["scripts"]` still works at the
# top level but matches nothing under `[lint]`/`[format]`. The subject is a planted tracked file
# holding an unused import, a hardcoded credential and a formatting error. GREEN means the gate
# was silently off:
#
# DISCOVERY FORM EXPLICIT FORM (what ships)
# exclude scope check . format --check . check format --check
# top-level GREEN GREEN red red
# [lint] GREEN red red red
# [format] red GREEN red red
# top + force-exclude GREEN GREEN red red <- with the flag
# GREEN GREEN <- without it
#
# Only the top-level scope empties BOTH discovery commands; `[lint]` empties `check` and
# `[format]` empties `format --check`, so in those two the job would still redden on the other
# step. `[format]` is where a line appended to ruff.toml lands, by TOML rules. `include = []`,
# `extend-exclude` and a nested `scripts/ruff.toml` behave the same way and are equally inert
# against the explicit form. The last row is the whole reason for `--no-force-exclude`:
# `force-exclude = true` re-applies excludes to explicitly-passed paths, and is the one setting
# that reaches explicitly-passed paths at all.
#
# `ruff check .` over an empty tree exits **0** with only a stderr warning, so every GREEN above
# is a gate that was switched off without a red.
#
# This also derives the population from source rather than from the filesystem
# (docs/decisions/records/testing/guard-derives-population-from-source.md) and covers
# tracked-but-gitignored files, which `ruff check .` skips. The empty-population arm is the
# anti-vacuity check: a completeness check whose population is empty reports that it proved
# everything. What it does NOT cover: an emptied RULE set. `select = []` silences every selected
# rule, so the `ruff check` step goes green over any lint violation (a syntax error still reds)
# while printing a reassuring file count.
# `ruff format --check` is unaffected, because formatting is not rule-selected. So half the
# gate is killable by a config edit, and only a human reading that edit catches it.
- name: Lint scripts (ruff check)
run: |
mapfile -d '' -t PYFILES < <(git ls-files -z '*.py' '*.pyi' '*.ipynb')
if [ "${#PYFILES[@]}" -eq 0 ]; then
echo "::error::the lint population is EMPTY — git tracks no Python files. Either the" \
"checkout is wrong or the glob is. A lint over nothing passes; see ersatztv#780."
exit 1
fi
echo "Linting ${#PYFILES[@]} tracked Python files"
python3 -m ruff check --no-force-exclude -- "${PYFILES[@]}"
- name: Lint scripts (ruff format --check)
run: |
mapfile -d '' -t PYFILES < <(git ls-files -z '*.py' '*.pyi' '*.ipynb')
if [ "${#PYFILES[@]}" -eq 0 ]; then
echo "::error::the format population is EMPTY — git tracks no Python files. See ersatztv#780."
exit 1
fi
echo "Format-checking ${#PYFILES[@]} tracked Python files"
python3 -m ruff format --check --no-force-exclude -- "${PYFILES[@]}"
# pytest + PyYAML. PyYAML is NOT a contradiction of the dependency-free decisions READ path:
# `decisions_lib._read_frontmatter` is hand-written precisely so validation runs where nothing
# is installed, but the one-shot WRITE path `migrate_decisions_split.py` uses PyYAML by
@@ -379,20 +463,6 @@ jobs:
# went red in CI on a collection error.
- name: Install test dependencies
run: python3 -m pip install --disable-pip-version-check --quiet pytest pyyaml
# Preflight, not an install (ersatztv#390 removed run-time `apt-get` from CI on purpose).
# test_post_review_verdict.py and test_merge_consent_exemption.py exec the REAL
# post-review-verdict.sh / pretooluse-merge-consent.sh, which shell out to `jq` ~26 times.
# `curl` those tests shim on PATH; `jq` they do NOT. If it were missing, the suite would fail
# as ~20 opaque assertion errors — this turns that into one actionable line.
- name: Preflight external tools
run: |
if ! command -v git >/dev/null 2>&1; then
echo "::error::script-tests needs git on PATH but it is absent. The suite execs real" \
"shell scripts that use it. Bake it into the runner image rather than apt-get" \
"installing here (see ersatztv#390)."
exit 1
fi
echo "Preflight OK: $(git --version)"
# jq gets its OWN step because its VERSION, not merely its presence, is load-bearing
# (ersatztv#648). `--expect` makes this a TRIPWIRE: scripts/tests exercises the jq 1.6 code path
# only because this runner ships 1.6, so an upgrade would silently delete that coverage — and
+29 -3
View File
@@ -928,14 +928,14 @@ this job (`prove-fix.sh` exits 5 on a harness/git failure or a signal, and the j
error), so what the exit-code discipline buys is the other way round — a **green** here means a claim
was witnessed, never that a run was cancelled or broke.
### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`)
### `script-tests` job (`Script lint and tests (ruff + pytest)`, PR-only — in `pr-checks.yml`)
> Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the
> three required status checks on `main` (`Build & test (.NET)`, `EF migration integrity`,
> `review-verdict/h10`). Promoting it to required is a branch-protection change, tracked separately.
Runs the repository's Python test suite: `PYTHONPATH=. python3 -m pytest scripts/tests -q`
(~190 tests at time of writing, ~10s; the suite grows, so treat the figure as indicative). It covers the decision-corpus parser/validator/catalog builder, the ersatztv#610
(773 tests at `706674272`, ~4.5 min; the suite grows fast — it was ~190 tests / ~10s when this job landed in #631 — so treat the figure as a dated snapshot, not a budget). It covers the decision-corpus parser/validator/catalog builder, the ersatztv#610
migration-equivalence harness, the merge-consent exemption logic and the ersatztv#622 review-verdict
poster.
@@ -956,7 +956,33 @@ name keeps a real failure unambiguous.
input set spans more than one directory — `test_post_review_verdict.py` and
`test_merge_consent_exemption.py` execute the real `scripts/post-review-verdict.sh` and
`.claude/hooks/pretooluse-merge-consent.sh` — so a `scripts/**` filter would silently miss a
`.claude/hooks/**` edit. At ~10s, a filter buys nothing but drift.
`.claude/hooks/**` edit. The reason is the input set, not the cost: the suite was ~10s when that was
decided and is now ~4.5 min, and it would still be wrong to filter on `scripts/**`.
**It also lints (ersatztv#780).** Early in the job it installs a **pinned** `ruff==0.12.11` and runs
`ruff check` and `ruff format --check` against the repo-root `ruff.toml`. Five things are deliberate:
- The config is **committed**. Without it ruff falls back to whatever `~/.config/ruff/ruff.toml` the
operator's machine has, so a second machine lints this repo differently or not at all.
- The version is **pinned** — an unpinned install makes the verdict a function of when the job ran,
the same divergence one layer up, and the same argument as the `jq` pin below. `pytest`/`pyyaml`
stay unpinned on purpose: a pytest release does not add assertions to your suite, a ruff release
adds rules to your lint.
- Lint runs **before the jq preflight**, and after the `git` one. `Preflight jq version` is a hard
`--expect` tripwire; a lint sitting behind it goes dark for as long as the jq contract is broken,
under a red that says "jq". `Preflight external tools` stays ahead, because the lint steps consume
`git` — without it a missing git reaches them as an empty population and they blame the glob.
- Neither step is `ruff check .`. Both pass an **explicit population** from
`git ls-files -z '*.py' '*.pyi' '*.ipynb'` with `--no-force-exclude`, and fail if that list is
empty. Discovery-based invocation is silently emptied by an `exclude` in the right config scope —
top level empties both commands, `[lint]` empties `check`, `[format]` empties `format --check`
(and `[format]` is where an appended line lands) — and
`ruff check .` over zero files exits **0** with only a stderr warning, so the failure mode is a
green gate. The measured matrix is in `ci.python-lint-ruff-config-committed`.
- `RUF100` is selected, so a `# noqa` that no longer matches anything is itself a finding.
`pyright` is not gated; the reasoning and the exemption list are in
`ci.python-lint-ruff-config-committed`.
**Dependencies: `pytest` and `pyyaml`** — the complete third-party set across `scripts/`, established
by an AST import scan rather than by reading the files that looked relevant. PyYAML does **not**
+2 -1
View File
@@ -56,10 +56,11 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
| `ci.monitor-armed-at-pr-open` | Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work. | 2026-07-21 | [link](records/ci/monitor-armed-at-pr-open.md) |
| `ci.no-host-health-gating` | Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs. | 2026-07-21 | [link](records/ci/no-host-health-gating.md) |
| `ci.peak-anon-measurement` | The `test` job's headline memory figure is a sampled high-water mark of cgroup `anon`, produced by `scripts/ci-peak-anon.sh`; `memory.peak` and the end-of-job `anon`/`file` split are kept only as a cache-inflated reference. | 2026-07-19 | [link](records/ci/peak-anon-measurement.md) |
| `ci.python-lint-ruff-config-committed` | The repo commits `ruff.toml`, and the `script-tests` job runs `ruff check` + `ruff format --check` under a PINNED ruff over an EXPLICIT population from `git ls-files`, never `ruff check .`. Never rely on `~/.config/ruff/ruff.toml`, and never add a lint rule to the config without making the tree clean against it in the same PR. | 2026-08-21 | [link](records/ci/python-lint-ruff-config-committed.md) |
| `ci.required-job-step-execution-markers` | A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`'s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job's LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step's `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships. The ban is enforced on the RELEASE PATH itself, not only in review (#767): a `scan` job runs the PyYAML-based ban test and `build` lists it in `needs:`, so a delimiter means `build` never runs and no image is published. A guard STEP inside `build` was tried first and is wrong — a step cannot protect the job it publishes from, and "my body has no opener so I cannot be dropped" is circular when only the PR-only test enforces that. The pytest in `script-tests` remains, but it is `on: pull_request` and not a required context, so it alone left the tag path unchecked. | 2026-08-10 | [link](records/ci/required-job-step-execution-markers.md) |
| `ci.root-screenshot-guard` | The Husky `pre-commit` hook refuses a staged root-level `*.png` (belt-and-suspenders with the `.gitignore` rule); nested `*.png` real assets are unaffected. | 2026-07-12 | [link](records/ci/root-screenshot-guard.md) |
| `ci.runner-placement` | No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's). | 2026-07-17 | [link](records/ci/runner-placement.md) |
| `ci.script-tests-job` | The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it. | 2026-07-26 | [link](records/ci/script-tests-job.md) |
| `ci.script-tests-job` | The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest pyyaml`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`; since #780 it also runs a pinned ruff over a `git ls-files` population first), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it. | 2026-07-26 | [link](records/ci/script-tests-job.md) |
| `ci.shared-pr-file-enumeration` | A PR's complete set of changed file paths is computed by exactly one implementation, `scripts/pr-changed-files.sh`, called by both `.claude/hooks/pretooluse-merge-consent.sh` (advisory — a failure falls through to a human prompt) and `.gitea/workflows/review-verdict.yml` (enforced — a failure must fail closed, because a match here posts the branch-protection-required `review-verdict/h10` status with nobody in the loop). The script owns exhaustiveness (pagination, rename/path validation, head-sha binding, base-ref binding — see `ci.exemption-provenance` — and base-TIP binding, #707: the ref answers "did this PR RETARGET", the tip answers "did the base ADVANCE mid-enumeration", and only the second can see `/pulls/{n}/files` recomputing each offset-paged page against a moved base and dropping a path out of an already-consumed range; both ends of the window are bound, and an advance BEFORE the window is deliberately not an error, or ordinary churn on `main` would fail every open PR) and returns exit 0 only for a verified-complete list; it does NOT classify paths — each caller keeps its own docs-only allow-list, and the two allow-lists differ on purpose and stay separate. | 2026-07-26 | [link](records/ci/shared-pr-file-enumeration.md) |
| `ci.small-lane-git-only` | `runs-on: small` is defined by what a job does (git-only), not its usual runtime; the two `docker build` jobs (docker-build.yml, ci-image.yml) move to `ubuntu-latest` because their worst-case memory, not median runtime, was pinning the small lane's per-slot cap. | 2026-07-20 | [link](records/ci/small-lane-git-only.md) |
| `ci.ui-e2e-harness` | The UI-interactive E2E flows run as headless Playwright specs (`web/e2e/*.spec.ts`, driven by `scripts/e2e-ui.sh`) in a **second step of the existing advisory `functional-e2e` job**, never their own job; the browser is `chromium-headless-shell` **baked into the CI toolchain image** (`docker/ci/Dockerfile`, `PLAYWRIGHT_VERSION` kept equal to `web/package.json`'s EXACT `@playwright/test` pin), never installed per run; specs are `serial` with `retries: 0` and assert only contracts the curl harness structurally cannot reach. | 2026-07-25 | [link](records/ci/ui-e2e-harness.md) |
@@ -0,0 +1,97 @@
---
key: ci.python-lint-ruff-config-committed
title: 2026-08-21 — Python lint is a committed ruff.toml enforced in CI, not the operator machine's global config (#780)
status: active
since: '2026-08-21'
supersedes: none
superseded-by: none
rule: The repo commits `ruff.toml`, and the `script-tests` job runs `ruff check` + `ruff format --check` under a PINNED ruff over an EXPLICIT population from `git ls-files`, never `ruff check .`. Never rely on `~/.config/ruff/ruff.toml`, and never add a lint rule to the config without making the tree clean against it in the same PR.
signals: 'ruff · pyright · python lint · `ruff format --check` · lint passes on my machine but not yours · no repo lint config · S105 on a test stub credential · paths: `ruff.toml`, `.gitea/workflows/pr-checks.yml` · issues: #780, #773, #648, #512'
mechanics: 'Config at repo root; `.gitea/workflows/pr-checks.yml` -> `script-tests` pins `ruff==0.12.11` via pip and runs both commands ahead of the jq preflight and pytest. Population is `git ls-files -z ''*.py'' ''*.pyi'' ''*.ipynb''` passed explicitly with `--no-force-exclude`, guarded by an empty-list arm; discovery-based invocation is defeated by an `exclude` in three config scopes, two of them per command. Bumping the pin is a deliberate PR because a new ruff release adds rules.'
---
The global instructions tell every session to run `ruff check`, `ruff format --check` and `pyright`
after touching Python. Before this, the repo enforced none of them and committed no config, so ruff
fell back to whichever `~/.config/ruff/ruff.toml` the operator's machine happened to have — **a second
machine lints this repo differently, or not at all.** That is the same shape as #643/#647/#648 (a
shell gate whose behaviour was a function of an untested interpreter version) and #512 (a test that
passed on a fast laptop and flaked on a starved CI VM): the verdict was a property of the environment
rather than of the repo.
The committed config is the operator's global one apart from `per-file-ignores`, which is narrowed
to `scripts/tests/**`. That is what the tree was de-facto written against, so adopting it cost a
mechanical reformat rather than a rewrite: 74 findings against `706674272`, of which 57 were fixed in
code (mostly by the format pass) and 17 carry a per-site `# noqa` with its reason inline. `RUF100` is
selected so those suppressions stay honest — a `# noqa` that suppresses nothing is otherwise
invisible, and three were live the moment the rule was switched on: one whose rule had stopped firing,
one for a rule this config never enables, and one added mid-branch on a site the same branch had
already fixed in code.
**One exemption is directory-wide, and it is the boring one.** `S101` for `scripts/tests/**`, because
a test suite asserts. **`S105` is deliberately NOT directory-wide.** All eight of its hits among
those 74 findings are stub credentials handed to the real hooks (`env["ETV_GITEA_TOKEN"] = "stub"`),
with no true positive in the tree today (a ninth `# noqa: S105` predates this and sits on a
commit-message marker in `decisions_validate.py`). A directory blanket would give up
hardcoded-credential coverage over the largest Python surface in the repo, permanently, to suppress
eight known lines — and this is the only Python lint the repo runs, so nothing else would catch a real
token pasted into a fixture next year. Per-site `# noqa: S105` costs the same and keeps the rule live.
**The population comes from `git ls-files`, not from ruff's discovery, and that is the load-bearing
part.** `ruff check .` reports on what it *discovers*, and an `exclude` defeats discovery in three
different config scopes — including `[format]`, which is where an appended line lands by TOML rules (two of the
three defeat each command). Measured with ruff 0.12.11 and `exclude = ["scripts/**"]`, against a
tracked file holding an unused import, a hardcoded credential and a formatting error. The pattern
matters: `exclude` is matched per FILE, so a bare `["scripts"]` works at the top level but matches
nothing under `[lint]`/`[format]`. GREEN means the gate was silently off:
| `exclude` in | `ruff check .` | explicit `check` | `ruff format --check .` | explicit `format` |
|---|---|---|---|---|
| top level | GREEN | red | GREEN | red |
| `[lint]` | GREEN | red | red | red |
| `[format]` | red | red | GREEN | red |
| top + `force-exclude` | GREEN | GREEN without `--no-force-exclude`, red with it | GREEN | same |
Only the top-level scope empties both discovery commands; `[lint]` empties `check`, `[format]` empties
`format --check`, so in those two the job would still redden on the other step. `[format]` is where a
line appended to `ruff.toml` lands, by TOML rules. The last row is the whole reason for the flag.
`include = []`, `extend-exclude` and a nested `scripts/ruff.toml` were tried too, and are equally
inert against the explicit form. The empty-list arm is the anti-vacuity
check — `ruff check .` over no files exits **0** with a stderr warning, so an emptied population is a
green gate, not a red one. Enumerating from git also covers tracked-but-gitignored files, which
discovery skips (`git add -f` under an ignored path is established practice here).
**The rule set is not covered, and that is a stated limit rather than an oversight.** `select = []`
silences every selected rule, so the `ruff check` step goes green over any lint violation (a syntax
error still reds) while still printing a reassuring file count. `ruff format --check` is unaffected, because formatting is not rule-selected. So the population
arm makes an emptied *file* set loud, nothing makes an emptied *rule* set loud, and half the gate is
killable by a config edit only a reviewer catches.
Both steps were witnessed red on the runner before merge, not argued to work — **on the body that
shipped**: run 2173 job 9176 (`❌ Failure - Main Lint scripts (ruff check)` on an `F401`) and run 2170
job 9163 (`❌ Failure - Main Lint scripts (ruff format --check)`), printing `Linting 34 tracked
Python files` and `Format-checking 34 tracked Python files` — the population arm executing (34 = the
33 tracked files plus the probe; the merged tree has 33). Each came from a temporary probe commit
reverted before merge. Two probes are needed, not one: a check-dirty file stops the job
before the format step ever runs. Earlier reds against the previous, discovery-based bodies were
discarded rather than cited — a proof belongs to the code that ran, not to its predecessor.
**Lint runs early in the job, ahead of the jq preflight.** `Preflight jq version` is a hard `--expect`
tripwire; a lint step behind it stops running for as long as the jq contract is broken, under a red
that names jq. Ordering is the difference between a gate that is skipped and one that is not. The
`git` half of `Preflight external tools` stays *ahead* of the lint steps, because they consume `git`:
without it, a missing git arrives as an empty population and both steps report a population problem
instead of the missing tool.
**`pyright` is deliberately NOT gated.** Its only findings here are `reportMissingImports` for
`etv_client` in `scripts/scripted-schedules/entrypoint.py`, resolvable only inside that script's
deploy environment. Gating it would put a node toolchain on the git-only `small` lane to find nothing.
Revisit when this repo grows a typed Python surface — the reason is the cost/finding ratio today, not
a judgement that type checking does not belong.
**The pin is the second half of the fix.** An unpinned `pip install ruff` re-introduces exactly the
divergence the config closes, one layer up: the verdict becomes a function of *when* the job ran. Same
argument as the `jq` pin in the same job (`ci.jq-version-contract`), and the same consequence — a bump
is a PR someone reads. `pytest` and `pyyaml` in the same job stay unpinned, and the
asymmetry is the point rather than an oversight: a pytest release does not add assertions to your
suite, a ruff release adds rules to your lint.
@@ -5,7 +5,7 @@ status: active
since: '2026-07-26'
supersedes: none
superseded-by: none
rule: 'The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it.'
rule: 'The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest pyyaml`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`; since #780 it also runs a pinned ruff over a `git ls-files` population first), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it.'
signals: 'scripts/tests never ran in CI, pytest not in any workflow, python test suite local-only, decorative test, decisions-guard runs the code not the tests, script-tests job, small lane pytest, negative control CI goes red · paths: `.gitea/workflows/pr-checks.yml`, `scripts/tests/`, `docs/ci-cd.md` · issues: #631, #610, #621, #622, #542'
mechanics: '`.gitea/workflows/pr-checks.yml` -> `script-tests`; `docs/ci-cd.md` -> "`script-tests` job"'
---
@@ -29,7 +29,7 @@ it lives in, so **a job under a standing ignore-rule can host no real gate.**
This does not conflict with `ci.ui-e2e-harness` ("never their own job"). That record folds UI-E2E
into `functional-e2e` because the specs need an app the job has *already booted* — sharing expensive
setup. Here there is no shared setup to reuse (a checkout plus `pip install pytest pyyaml`), and the
setup. Here there is no shared setup to reuse (a checkout plus `pip install pytest pyyaml` and a pinned ruff), and the
sibling job carries an ignore-rule. Same question, opposite answers, for stated reasons.
**Unconditional, not path-filtered.** The suite's real input set spans more than `scripts/`:
@@ -104,3 +104,9 @@ from a proxy ("fewer than we asked for", "jq didn't complain").
It is not yet a *required* status check — `main` requires only `Build & test (.NET)`,
`EF migration integrity` and `review-verdict/h10`. It reddens the run; promoting it to required is a
branch-protection change left deliberately separate.
Since #780 the job also lints Python before pytest, so its display name is
`Script lint and tests (ruff + pytest)`. It does **not** invoke `ruff check .` — the invocation and
the reasons for its exact shape are `ci.python-lint-ruff-config-committed`. Why the lint lives here
rather than in a job of its own: it needs the same `setup-python`, it costs seconds, and a second job would double the dispatch overhead
this file exists to keep small.
+38
View File
@@ -445,6 +445,44 @@ machine lints differently, or not at all. One of the two must move — either th
committed ruff config and enforces it, or the global instruction stops claiming this repo enforces
something it does not.
**Resolved 2026-08-21 (#780): the repo moved.** `ruff.toml` is committed at the root and the
`script-tests` job runs `ruff check` + `ruff format --check` under a pinned `ruff==0.12.11` over an
explicit population from `git ls-files` — not `ruff check .`, which an `exclude` in the right config
scope silently empties into a green run. The pre-fix state, reproducible rather than
asserted — against `706674272`, the base this landed on, with the committed config dropped in:
```
mkdir -p ~/scratch/m780 && git archive 706674272 | tar -x -C ~/scratch/m780
git show 01f7a89e8:ruff.toml > ~/scratch/m780/ruff.toml # a sha: the file is not on main pre-merge
cd ~/scratch/m780 && ruff check . ; ruff format --check . # ruff 0.12.11
# -> Found 74 errors. / 20 files would be reformatted, 13 files already formatted
```
(`;` not `&&``ruff check` exits 1, which would swallow the second command. Not `/tmp`: macOS purges
it. Redirect the config into place *before* running anything: an empty `ruff.toml` is valid, so a
failed `git show` leaves ruff silently using its own defaults and printing a different number.)
The row above measured **47** eight days earlier against the operator's global config; the tree grew
and the configs differ, so the two numbers are not comparable and neither supersedes the other. Two
of the 74 are `RUF100` on suppressions that were already in the tree before this change — they exist
in this count only because the committed config enables that rule.
Of the 74, **57 were fixed in code** (most of them by the `ruff format` pass itself, which splits the
40 semicolon statements) and **17 carry a per-site `# noqa` with its reason inline**: 8 `S105` on stub
credentials handed to the real hooks by `scripts/tests`, 9 `E501` on one-line JSON and shell fixtures.
The `S105`s are deliberately per-site rather than a directory exemption, so a real credential pasted
into a fixture later still reddens the gate. Only `S101` is exempted directory-wide for `scripts/tests/**`,
because a test suite asserts.
`RUF100` is selected, which is what keeps that split honest: a `# noqa` that suppresses nothing still
reads as a suppression, and it is invisible without this rule. Three were live when it was switched
on: one on a site that had already been fixed in code, plus the two counted above — one whose rule had
stopped firing, one for a rule this config never enables.
`pyright` stayed ungated: its only findings are the `etv_client` imports in the row above, and gating
it would put a node toolchain on the git-only `small` lane to find nothing. Rationale, the exemption
list and the measured exclude matrix: `ci.python-lint-ruff-config-committed`.
### 5.3 Configured vs actually invoked
Measured over the session transcript corpus (811 files under
+59
View File
@@ -0,0 +1,59 @@
# Ruff configuration for this repo's Python surface (all of it lives under `scripts/`).
#
# WHY THIS FILE EXISTS (ersatztv#780). Without a committed config, ruff falls back to whatever
# `~/.config/ruff/ruff.toml` the operator's machine happens to have — so a second machine lints this
# repo differently, or not at all. That is the environment-divergence class #643/#647/#648 (a shell
# gate whose behaviour was a function of an untested interpreter version) and #512 (a test that
# passed on a fast laptop and flaked on a starved CI VM). The settings below are pinned HERE so the
# lint verdict is a property of the repo, not of the machine.
#
# It is enforced by the `script-tests` job (`Script lint and tests (ruff + pytest)`) in
# .gitea/workflows/pr-checks.yml. A config nobody runs is the same divergence one step later.
#
# That job does NOT invoke `ruff check .`: it passes an explicit population from `git ls-files` with
# `--no-force-exclude`. An `exclude` added to this file silently empties a discovery-based run into a
# GREEN one — a top-level `exclude` empties both commands, one under `[lint]` empties `check`, one
# under `[format]` (where an appended line lands, by TOML rules) empties `format --check`. Adding
# `exclude` here will therefore not do what you expect, which is the point. The measured matrix is in
# `ci.python-lint-ruff-config-committed`.
#
# `pyright` is deliberately NOT gated: its only findings here are `reportMissingImports` for
# `etv_client` in scripts/scripted-schedules/entrypoint.py, which resolves only inside that script's
# deploy environment, and gating it would put a node toolchain on the git-only `small` lane for zero
# real findings. Revisit if this repo grows a typed Python surface.
target-version = "py311"
line-length = 120
[lint]
select = [
"E", # pycodestyle errors
"W", # pycodestyle warnings
"F", # pyflakes
"I", # isort
"B", # flake8-bugbear
"UP", # pyupgrade
"SIM", # flake8-simplify
"S", # flake8-bandit (security)
# RUF100 is load-bearing, not tidiness: every `# noqa` below is an assertion that a real finding
# is being suppressed for a stated reason, and without this a suppression that suppresses nothing
# stays in the file reading as one. #780 did exactly that mid-branch — a `# noqa: UP031` on a site
# the same branch had already fixed in code — and found two more already in the tree: one whose
# rule had stopped firing, one for a rule this config never enables.
"RUF100",
]
ignore = [
"S603", # subprocess call - check for execution of untrusted input (too noisy for scripts)
"S607", # starting a process with a partial executable path
]
[lint.per-file-ignores]
# scripts/tests asserts, so S101 would fire on every test. S105 is deliberately NOT exempted here:
# the eight sites that trip it (`env["ETV_GITEA_TOKEN"] = "stub"`) carry a per-site `# noqa: S105`
# instead, so a real credential pasted into a fixture next year still reddens the gate. A directory
# blanket would have given up hardcoded-credential coverage over the largest Python surface in the
# repo, permanently, to suppress eight known lines.
"scripts/tests/**" = ["S101"]
[format]
quote-style = "double"
+2 -7
View File
@@ -19,7 +19,7 @@ from datetime import date
from pathlib import Path
from typing import NamedTuple
import scripts.decisions_lib as dl # noqa: E402 (run with PYTHONPATH=. or as module)
import scripts.decisions_lib as dl # (run with PYTHONPATH=. or as module)
SKIP_HEADINGS = dl.SKIP_HEADINGS # single source of truth
# `signals` is required alongside the lifecycle fields: the `**Signals:**` line (plus `key:`) is what
@@ -469,12 +469,7 @@ def _is_stripped_index(path: Path, archive_dir: Path, recs: list) -> bool:
file there is exempt only if it actually LOOKS like a stripped index: exactly one keyless
record whose heading is one of the known generated ones (`dl.SKIP_HEADINGS`).
"""
return (
path.parent == archive_dir
and len(recs) == 1
and not recs[0].key
and recs[0].heading in dl.SKIP_HEADINGS
)
return path.parent == archive_dir and len(recs) == 1 and not recs[0].key and recs[0].heading in dl.SKIP_HEADINGS
def record_wing_faults(records_dir: Path | None = None, archive_dir: Path | None = None) -> list[str]:
+2 -6
View File
@@ -62,9 +62,7 @@ def render(spec: dict) -> str:
"`scripts/update-openapi.sh`.*"
)
lines.append("")
lines.append(
f"{len(spec.get('paths', {}))} endpoints, {len(operations)} operations."
)
lines.append(f"{len(spec.get('paths', {}))} endpoints, {len(operations)} operations.")
lines.append("")
for tag in tags:
@@ -73,9 +71,7 @@ def render(spec: dict) -> str:
lines.append("| Method | Path | Operation | Summary |")
lines.append("|---|---|---|---|")
for op in sorted(grouped[tag], key=lambda o: (o["path"], o["method"])):
lines.append(
f"| {op['method']} | `{op['path']}` | {op['operationId']} | {op['summary']} |"
)
lines.append(f"| {op['method']} | `{op['path']}` | {op['operationId']} | {op['summary']} |")
lines.append("")
return "\n".join(lines).rstrip("\n") + "\n"
+33 -15
View File
@@ -37,6 +37,7 @@ explicitly NOT bounded; that is the accepted limit stated above.
from __future__ import annotations
import contextlib
import json
import os
import secrets
@@ -47,6 +48,7 @@ import sys
import threading
import time
def fail(msg: str, code: int) -> int:
print(f"FAIL: {msg}")
return code
@@ -60,15 +62,19 @@ def main() -> int:
i = 0
while i < len(argv):
if argv[i] == "--expect-server" and i + 1 < len(argv):
expect_server = argv[i + 1]; i += 2
expect_server = argv[i + 1]
i += 2
elif argv[i] == "--expect-tool" and i + 1 < len(argv):
expect_tools.append(argv[i + 1]); i += 2
expect_tools.append(argv[i + 1])
i += 2
else:
positional.append(argv[i]); i += 1
positional.append(argv[i])
i += 1
if len(positional) < 2:
return fail("usage: mcp_smoke.py <.mcp.json> <server> [timeout] "
"[--expect-server NAME] [--expect-tool NAME]...", 2)
return fail(
"usage: mcp_smoke.py <.mcp.json> <server> [timeout] [--expect-server NAME] [--expect-tool NAME]...", 2
)
cfg_path, server = positional[0], positional[1]
if len(positional) > 2:
try:
@@ -136,8 +142,12 @@ def main() -> int:
try:
proc = subprocess.Popen(
[resolved, *args], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL, env=env, cwd=workdir,
[resolved, *args],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
env=env,
cwd=workdir,
start_new_session=True, # own process group, so children die with us
)
except OSError as exc:
@@ -222,10 +232,8 @@ def main() -> int:
os.killpg(pgid, sig)
except OSError:
break # no group members left
try:
with contextlib.suppress(subprocess.TimeoutExpired):
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
pass
time.sleep(0.2)
else:
for sig in (signal.SIGTERM, signal.SIGKILL):
@@ -245,9 +253,20 @@ def main() -> int:
id_tools = secrets.randbelow(2**31 - 1000) + 1000
while id_tools == id_init:
id_tools = secrets.randbelow(2**31 - 1000) + 1000
init = expect(id_init, {"jsonrpc": "2.0", "id": id_init, "method": "initialize", "params": {
"protocolVersion": "2024-11-05", "capabilities": {},
"clientInfo": {"name": "mcp-smoke", "version": "0"}}}, deadline)
init = expect(
id_init,
{
"jsonrpc": "2.0",
"id": id_init,
"method": "initialize",
"params": {
"protocolVersion": "2024-11-05",
"capabilities": {},
"clientInfo": {"name": "mcp-smoke", "version": "0"},
},
},
deadline,
)
if init is None:
return fail(f"no 'initialize' response within {budget}s (server did not start)", 9)
if "error" in init:
@@ -263,8 +282,7 @@ def main() -> int:
return fail(f"wrong server: expected '{expect_server}', got '{actual}'", 13)
send({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}})
listed = expect(id_tools, {"jsonrpc": "2.0", "id": id_tools,
"method": "tools/list", "params": {}}, deadline)
listed = expect(id_tools, {"jsonrpc": "2.0", "id": id_tools, "method": "tools/list", "params": {}}, deadline)
if listed is None:
return fail(f"no 'tools/list' response within {budget}s", 10)
lresult = listed.get("result")
+9 -9
View File
@@ -3,18 +3,18 @@
import argparse
import importlib
import sys
from uuid import UUID
import etv_client
from etv_client.api import ScriptedScheduleApi
def main():
parser = argparse.ArgumentParser(description="Run an ETV scripted schedule")
parser.add_argument('host', help="The ETV host (e.g., http://localhost:8409)")
parser.add_argument('build_id', type=UUID, help="The build ID for the playout")
parser.add_argument('mode', choices=['reset', 'continue'], help="The playout build mode")
parser.add_argument('script_name', help="The name of the script module to use (e.g., one)")
parser.add_argument("host", help="The ETV host (e.g., http://localhost:8409)")
parser.add_argument("build_id", type=UUID, help="The build ID for the playout")
parser.add_argument("mode", choices=["reset", "continue"], help="The playout build mode")
parser.add_argument("script_name", help="The name of the script module to use (e.g., one)")
known_args, unknown_args = parser.parse_known_args()
@@ -28,9 +28,9 @@ def main():
with etv_client.ApiClient(configuration) as api_client:
try:
define_content = getattr(script_module, 'define_content')
reset_playout = getattr(script_module, 'reset_playout')
build_playout = getattr(script_module, 'build_playout')
define_content = script_module.define_content
reset_playout = script_module.reset_playout
build_playout = script_module.build_playout
api_instance = ScriptedScheduleApi(api_client)
@@ -48,6 +48,6 @@ def main():
except AttributeError as e:
print(f"Error: the '{known_args.script_name}' script is missing a required function. {e}")
if __name__ == "__main__":
main()
+1 -3
View File
@@ -24,6 +24,4 @@ import pytest
@pytest.fixture(autouse=True)
def isolate_hook_fire_log(tmp_path_factory, monkeypatch):
monkeypatch.setenv(
"ETV_HOOK_FIRE_LOG_DIR", str(tmp_path_factory.mktemp("hook-fire-log"))
)
monkeypatch.setenv("ETV_HOOK_FIRE_LOG_DIR", str(tmp_path_factory.mktemp("hook-fire-log")))
+27 -14
View File
@@ -36,9 +36,17 @@ BOM = b"\xef\xbb\xbf"
def _git(cwd: Path, *args: str) -> None:
subprocess.run(
["git", *args], cwd=str(cwd), check=True, capture_output=True,
env={**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e"},
["git", *args],
cwd=str(cwd),
check=True,
capture_output=True,
env={
**os.environ,
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@e",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@e",
},
)
@@ -54,12 +62,18 @@ def _repo_with(tmp_path: Path, name: str, content: bytes) -> Path:
def _run(hook: Path, repo: Path, env: dict) -> tuple[int, bytes]:
payload = json.dumps({
"session_id": "s", "hook_event_name": "PreToolUse", "tool_name": "Bash",
"cwd": str(repo), "tool_input": {"command": "git commit -m x"},
})
p = subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
cwd=str(repo), env=env, timeout=60)
payload = json.dumps(
{
"session_id": "s",
"hook_event_name": "PreToolUse",
"tool_name": "Bash",
"cwd": str(repo),
"tool_input": {"command": "git commit -m x"},
}
)
p = subprocess.run(
["bash", str(hook)], input=payload.encode(), capture_output=True, cwd=str(repo), env=env, timeout=60
)
return p.returncode, p.stdout
@@ -105,8 +119,9 @@ def _path_without_xxd(tmp_path: Path) -> dict:
def test_the_fixture_really_stages_a_BOM(tmp_path):
repo = _repo_with(tmp_path, "Bad.cs", BOM + b"class A {}\n")
assert (repo / "Bad.cs").read_bytes()[:3] == BOM
staged = subprocess.run(["git", "diff", "--name-only", "--cached"], cwd=str(repo),
capture_output=True, text=True).stdout.split()
staged = subprocess.run(
["git", "diff", "--name-only", "--cached"], cwd=str(repo), capture_output=True, text=True
).stdout.split()
assert staged == ["Bad.cs"], f"nothing was staged, so the guard would have nothing to read: {staged}"
@@ -179,6 +194,4 @@ def test_DISARMING_the_BOM_comparison_stops_detection(tmp_path):
rc, out = _run(mutated, repo, dict(os.environ))
assert rc == 0
assert out == b"", (
f"disarming the BOM comparison did not stop detection, so it is not load-bearing: {out!r}"
)
assert out == b"", f"disarming the BOM comparison did not stop detection, so it is not load-bearing: {out!r}"
+38 -34
View File
@@ -115,13 +115,13 @@ def _guard_buckets(job: str):
argv = _guard(job)["run"].split()
assert "--always" in argv and "--gated" in argv, argv
a, g = argv.index("--always"), argv.index("--gated")
return argv[a + 1:g], argv[g + 1:]
return argv[a + 1 : g], argv[g + 1 :]
# Mirrors the `if:` every gated step in these jobs carries. Compared as a normalised string rather
# than by parsing the expression: what matters is that a step's gating and the guard's bucketing are
# the SAME condition, and any rewrite of one that is not mirrored in the other should be loud.
SKIP_GATE = ("steps.detect.outputs.docs_only!='true'&&steps.revalidate.outputs.skip!='true'")
SKIP_GATE = "steps.detect.outputs.docs_only!='true'&&steps.revalidate.outputs.skip!='true'"
def _is_gated(step) -> bool:
@@ -211,17 +211,14 @@ def test_every_consequential_run_step_marks_itself_as_its_FIRST_act(job):
# prefix as a preceding command and reddens every correctly-written step.
lines = s["run"].splitlines()
at = next(i for i, ln in enumerate(lines) if _MARK.search(ln))
preceding = [
ln.strip() for ln in lines[:at]
if ln.strip() and not ln.strip().startswith("#")
]
preceding = [ln.strip() for ln in lines[:at] if ln.strip() and not ln.strip().startswith("#")]
if [ln for ln in preceding if not ln.startswith("set -")]:
late.append((s.get("name", "?"), preceding))
assert not missing, (
f"these run: steps of the REQUIRED job '{job}' do not record that they executed: {missing}. "
"A step the runner drops concludes success, so without a marker its non-execution takes the "
"whole required context green having done no work (ersatztv#756). Add "
'`\"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh\" mark <key>` as the step\'s first line and '
'`"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark <key>` as the step\'s first line and '
"the key to the guard step's --always/--gated list."
)
assert not late, (
@@ -357,8 +354,10 @@ def test_the_guards_OWN_body_cannot_be_dropped_by_the_mechanism_it_guards_agains
# a red here blocks every merge through the combined status, so brittleness is a real cost and
# not a free strictness win. Whitespace inside the delimiters is normalised for the same reason.
env = {k: re.sub(r"\s+", "", str(v)) for k, v in (guard.get("env") or {}).items()}
for name, want in (("ETV_DOCS_ONLY", "${{steps.detect.outputs.docs_only}}"),
("ETV_REVALIDATE_SKIP", "${{steps.revalidate.outputs.skip}}")):
for name, want in (
("ETV_DOCS_ONLY", "${{steps.detect.outputs.docs_only}}"),
("ETV_REVALIDATE_SKIP", "${{steps.revalidate.outputs.skip}}"),
):
assert env.get(name) == want, (
f"the '{job}' guard's env: has {name}={guard.get('env', {}).get(name)!r}, expected the "
f"output the gated steps' own `if:` reads ({want}). A typo here is SILENT rather than "
@@ -427,12 +426,10 @@ def _env(tmp_path, **extra):
def _run(script: str, env):
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env,
capture_output=True, text=True)
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env, capture_output=True, text=True)
@pytest.mark.parametrize("gate", GATE_VALUES_IN_THE_WILD,
ids=["gate-false", "gate-empty", "gate-unset"])
@pytest.mark.parametrize("gate", GATE_VALUES_IN_THE_WILD, ids=["gate-false", "gate-empty", "gate-unset"])
@pytest.mark.parametrize("job", MARKED_JOBS)
def test_the_guard_PASSES_when_every_step_marked_itself(job, gate, tmp_path):
"""The positive control. Without it, a guard that always failed would satisfy every case below.
@@ -458,7 +455,8 @@ def test_the_guard_PASSES_when_every_step_marked_itself(job, gate, tmp_path):
# inverted provenance test would otherwise ship silently, and the operator reading this line to
# settle the promotion question would read it wrong.
assert f"Marker identity: job={job} run=424242 attempt=7 (from the runner)" in r.stdout, (
f"the guard misreported its marker identity: {r.stdout!r}")
f"the guard misreported its marker identity: {r.stdout!r}"
)
@pytest.mark.parametrize("job", MARKED_JOBS)
@@ -534,8 +532,7 @@ def test_an_EMPTY_gate_value_requires_the_gated_steps(job, tmp_path):
what is required.
"""
guard = _guard(job)["run"]
r = _run("\n".join(["set -e", guard]),
_env(tmp_path, GITHUB_JOB=job, ETV_DOCS_ONLY="", ETV_REVALIDATE_SKIP=""))
r = _run("\n".join(["set -e", guard]), _env(tmp_path, GITHUB_JOB=job, ETV_DOCS_ONLY="", ETV_REVALIDATE_SKIP=""))
assert r.returncode != 0
assert _guard_buckets(job)[1][-1] in (r.stdout + r.stderr), (
f"empty gate values were read as a skip, so the gated steps went unchecked: {r.stdout}"
@@ -571,8 +568,7 @@ def test_a_STALE_marker_from_another_run_cannot_satisfy_the_guard(tmp_path):
# ...nor may the OTHER job in the same run inherit them.
sibling = _env(tmp_path, GITHUB_JOB="migrations", GITHUB_RUN_ID="111", GITHUB_RUN_ATTEMPT="1")
assert _run(_guard("migrations")["run"], sibling).returncode != 0, (
"the two required jobs share one marker file, so one job's markers answer for the other's "
"dropped steps"
"the two required jobs share one marker file, so one job's markers answer for the other's dropped steps"
)
@@ -611,8 +607,7 @@ def test_a_key_is_matched_WHOLE_not_as_a_substring(tmp_path):
assert _run(f"{SCRIPT} mark web-build && {SCRIPT} mark web-test", env).returncode == 0
r = _run(f"{SCRIPT} assert --always build", env)
assert r.returncode != 0, (
"the key 'build' was satisfied by a marker for 'web-build' — a dropped `dotnet build` would "
"pass unnoticed"
"the key 'build' was satisfied by a marker for 'web-build' — a dropped `dotnet build` would pass unnoticed"
)
@@ -638,9 +633,11 @@ def test_a_degraded_run_IDENTITY_refuses_rather_than_sharing_a_marker_path(tmp_p
f"{r.stdout}{r.stderr}"
)
assert "cannot identify this run" in (r.stdout + r.stderr), (
f"refused, but without naming the cause: {r.stdout!r} {r.stderr!r}")
f"refused, but without naming the cause: {r.stdout!r} {r.stderr!r}"
)
assert not list(tmp_path.iterdir()), (
"a degraded-identity `mark` still created a marker file somewhere under RUNNER_TEMP")
"a degraded-identity `mark` still created a marker file somewhere under RUNNER_TEMP"
)
def test_the_marker_identity_is_REPORTED_on_stdout_every_run(tmp_path):
@@ -661,12 +658,15 @@ def test_the_marker_identity_is_REPORTED_on_stdout_every_run(tmp_path):
documented contract (the record's `mechanics:`), and a future reader is told to trust it.
"""
marks = [_mark_line(s) for s, _ in _marked("test")]
r = _run("\n".join(["set -e", *marks, _guard("test")["run"]]),
_env(tmp_path, GITHUB_RUN_ID="1916", GITHUB_RUN_ATTEMPT="4"))
r = _run(
"\n".join(["set -e", *marks, _guard("test")["run"]]),
_env(tmp_path, GITHUB_RUN_ID="1916", GITHUB_RUN_ATTEMPT="4"),
)
assert r.returncode == 0, r.stdout + r.stderr
assert "Marker identity: job=test run=1916 attempt=4 (from the runner)" in r.stdout, (
"the guard did not report the identity its marker path was actually keyed on, so a reader "
f"cannot audit the keying from a run log: {r.stdout!r}")
f"cannot audit the keying from a run log: {r.stdout!r}"
)
def test_a_skip_gate_that_empties_the_expected_set_REFUSES(tmp_path):
@@ -681,15 +681,16 @@ def test_a_skip_gate_that_empties_the_expected_set_REFUSES(tmp_path):
that it proved everything while proving nothing" is the failure this whole file exists to remove.
"""
r = _run(f"{SCRIPT} assert --always --gated foo", _env(tmp_path, ETV_DOCS_ONLY="true"))
assert r.returncode != 0, (
f"the guard passed with an empty post-gate expectation set: {r.stdout!r}")
assert r.returncode != 0, f"the guard passed with an empty post-gate expectation set: {r.stdout!r}"
assert "no expected keys" in (r.stdout + r.stderr).lower() or "NO expected keys" in r.stderr
@pytest.mark.parametrize("revalidate", ["true", "false", "", None],
ids=lambda v: f"reval-{v if v is not None else 'unset'}")
@pytest.mark.parametrize("docs_only", ["true", "false", "", None],
ids=lambda v: f"docs-{v if v is not None else 'unset'}")
@pytest.mark.parametrize(
"revalidate", ["true", "false", "", None], ids=lambda v: f"reval-{v if v is not None else 'unset'}"
)
@pytest.mark.parametrize(
"docs_only", ["true", "false", "", None], ids=lambda v: f"docs-{v if v is not None else 'unset'}"
)
def test_the_skip_gate_over_the_WHOLE_value_matrix(docs_only, revalidate, tmp_path):
"""Every combination of the two gate values, not just the diagonal — cold review's last finding.
@@ -713,11 +714,14 @@ def test_the_skip_gate_over_the_WHOLE_value_matrix(docs_only, revalidate, tmp_pa
job = "test"
always, gated = _guard_buckets(job)
marks = [_mark_line(s) for s, k in _marked(job) if k in always]
r = _run("\n".join(["set -e", *marks, _guard(job)["run"]]),
_env(tmp_path, ETV_DOCS_ONLY=docs_only, ETV_REVALIDATE_SKIP=revalidate))
r = _run(
"\n".join(["set -e", *marks, _guard(job)["run"]]),
_env(tmp_path, ETV_DOCS_ONLY=docs_only, ETV_REVALIDATE_SKIP=revalidate),
)
should_skip = docs_only == "true" or revalidate == "true"
assert (r.returncode == 0) is should_skip, (
f"with docs_only={docs_only!r} and revalidate={revalidate!r} the guard "
f"{'passed' if r.returncode == 0 else 'failed'}, expected it to "
f"{'skip the gated keys' if should_skip else 'require them'}. The gate must treat a value as "
"a skip if and only if it is exactly `true` in EITHER variable.\n" + r.stdout + r.stderr)
"a skip if and only if it is exactly `true` in EITHER variable.\n" + r.stdout + r.stderr
)
@@ -198,8 +198,7 @@ def test_the_registry_partitions_every_job_in_the_workflow():
)
assert not (TOOLCHAIN_JOBS & BARE_RUNNER_JOBS), "a job cannot be in both lists"
assert all_jobs == TOOLCHAIN_JOBS | BARE_RUNNER_JOBS, (
f"the registry names jobs that do not exist: "
f"{sorted((TOOLCHAIN_JOBS | BARE_RUNNER_JOBS) - all_jobs)}"
f"the registry names jobs that do not exist: {sorted((TOOLCHAIN_JOBS | BARE_RUNNER_JOBS) - all_jobs)}"
)
@@ -242,8 +241,7 @@ def test_a_single_job_losing_its_pin_is_DETECTED(doc):
empty list would satisfy the live assertion above and prove nothing which is how #621 and
#685 both shipped."""
assert pin_population_faults(doc), (
"the population check accepted a workflow in which a container job no longer runs the "
"pinned toolchain image"
"the population check accepted a workflow in which a container job no longer runs the pinned toolchain image"
)
+23 -25
View File
@@ -328,8 +328,7 @@ def _run_the_real_scan_body(repo: Path, tmp_path: Path):
(repo / _NONCE_FILE).write_text(nonce)
env[_FENCE] = nonce
(tmp_path / "runner").mkdir(exist_ok=True)
return subprocess.run(["bash", "-c", body], cwd=repo, env=env,
capture_output=True, text=True)
return subprocess.run(["bash", "-c", body], cwd=repo, env=env, capture_output=True, text=True)
@_nested
@@ -358,7 +357,7 @@ def test_the_scan_step_REALLY_FAILS_on_a_poisoned_workflow(tmp_path):
text = wf.read_text()
anchor = ' IMG="${IMAGE}:${SMOKE_SHORT_SHA}"'
assert anchor in text, "anchor for the poison is gone — rewrite this control"
wf.write_text(text.replace(anchor, ' # ${{ steps.meta.outputs.short }}\n' + anchor, 1))
wf.write_text(text.replace(anchor, " # ${{ steps.meta.outputs.short }}\n" + anchor, 1))
res = _run_the_real_scan_body(repo, tmp_path)
assert res.returncode != 0, (
@@ -403,8 +402,10 @@ def test_the_PROOF_SCRIPT_itself_refuses_when_the_ban_is_deselected(tmp_path):
)
res = subprocess.run(
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True, text=True,
cwd=repo,
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True,
text=True,
)
assert res.returncode != 0, (
"ci-prove-ban-detects.sh vouched for the gate while the ban test was deselected at the "
@@ -446,19 +447,16 @@ def test_the_PROOF_SCRIPT_refuses_when_the_WRONG_test_fails(tmp_path):
# landing on the "not enforcing" branch instead of the one under test. (First draft of this test
# did exactly that and was red for the wrong reason.)
ban = repo / BAN_TEST_FILE
ban.write_text(
ban.read_text()
+ "\n\ndef test_an_unrelated_failure_for_this_probe():\n assert False\n"
)
ban.write_text(ban.read_text() + "\n\ndef test_an_unrelated_failure_for_this_probe():\n assert False\n")
res = subprocess.run(
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True, text=True,
cwd=repo,
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True,
text=True,
)
combined = res.stdout + res.stderr
assert res.returncode != 0, (
f"the script read an unrelated test's failure as proof.\n{combined}"
)
assert res.returncode != 0, f"the script read an unrelated test's failure as proof.\n{combined}"
# THE SPECIFIC branch. This scenario is built to land on "wrong test failed" (exit 1, no `[build]`
# failure); accepting "could not prove anything" too would let it drift onto the exit-5 branch and
# silently cover a branch it was not written for, while still looking green.
@@ -474,15 +472,17 @@ def test_the_PROOF_SCRIPT_passes_on_a_clean_tree(tmp_path):
repo = _repo_copy(tmp_path)
res = subprocess.run(
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True, text=True,
cwd=repo,
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
capture_output=True,
text=True,
)
assert res.returncode == 0, (
f"ci-prove-ban-detects.sh failed on a clean tree.\nstdout:\n{res.stdout}\nstderr:\n{res.stderr}"
)
assert (repo / ".gitea" / "workflows" / "docker-build.yml").read_text() == (
WORKFLOW.read_text()
), "the script did not restore the workflow file it poisoned"
assert (repo / ".gitea" / "workflows" / "docker-build.yml").read_text() == (WORKFLOW.read_text()), (
"the script did not restore the workflow file it poisoned"
)
def test_the_pytest_invocation_cannot_DESELECT_or_swallow_its_result():
@@ -494,7 +494,7 @@ def test_the_pytest_invocation_cannot_DESELECT_or_swallow_its_result():
# Only the tokens AFTER `pytest` are pytest's own arguments. Checking the whole line would flag
# the `-m` in `python3 -m pytest`, which is how the interpreter is invoked — a false positive
# that would make this test red on the correct command.
args = tokens[tokens.index("pytest") + 1:]
args = tokens[tokens.index("pytest") + 1 :]
banned = {"-k", "-m", "--deselect", "--ignore", "--collect-only", "--co"}
assert not (banned & set(args)), f"pytest invocation may deselect tests: {line!r}"
for op in ("||", "&&", ";", "|"):
@@ -527,8 +527,7 @@ def test_no_step_in_the_scan_job_is_advisory():
@pytest.mark.parametrize(
"step_name",
[s.get("name", "?") for s in yaml.safe_load(WORKFLOW.read_text())["jobs"][JOB]["steps"]
if s.get("run")],
[s.get("name", "?") for s in yaml.safe_load(WORKFLOW.read_text())["jobs"][JOB]["steps"] if s.get("run")],
)
def test_every_run_body_in_the_scan_job_is_delimiter_free(step_name):
"""The guard must not be vulnerable to the defect it guards against.
@@ -561,7 +560,7 @@ def test_the_guard_expectations_match_the_markers_exactly():
restated here, so adding a step without a marker is a red."""
argv = _guard()["run"].split()
assert "--always" in argv, argv
always = argv[argv.index("--always") + 1:]
always = argv[argv.index("--always") + 1 :]
assert "--gated" not in argv, "every step in this job is unconditional; there is nothing to gate"
assert sorted(always) == sorted(k for _, k in _marked())
@@ -601,8 +600,7 @@ def _env(tmp_path, **extra):
def _run(script: str, env):
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env,
capture_output=True, text=True)
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env, capture_output=True, text=True)
def test_the_guard_PASSES_when_every_step_ran(tmp_path):
+11 -10
View File
@@ -744,7 +744,9 @@ def test_wing_faults_block_scalar_record_fails_loudly(tmp_path):
this corpus's very long `rule:` values — makes the whole record silently invisible."""
records, archive = _wing(tmp_path)
bad = records / "ci" / "blockscalar.md"
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\n a long rule wrapped\n over two lines\n"))
bad.write_text(
_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\n a long rule wrapped\n over two lines\n")
)
# Precondition: this really is the silent-vanish case, not some other parse error.
assert dl.parse_file(bad) == [], "expected the reader to drop the record entirely"
@@ -805,8 +807,10 @@ def test_wing_faults_exempts_stripped_legacy_archive_files(tmp_path):
def test_validate_surfaces_wing_faults_as_errors(tmp_path):
"""Faults must arrive as validator ERRORS (exit 1), not warnings."""
errs = _v([_rec(key="ci.a", source=Path("docs/decisions/records/ci/a.md"), heading="A")],
wing_faults=["docs/decisions/records/ci/x.md: parsed to 0 records, expected exactly 1"])
errs = _v(
[_rec(key="ci.a", source=Path("docs/decisions/records/ci/a.md"), heading="A")],
wing_faults=["docs/decisions/records/ci/x.md: parsed to 0 records, expected exactly 1"],
)
assert any("x.md" in e for e in errs), errs
@@ -826,7 +830,7 @@ def test_wing_faults_sees_a_DEEPER_nested_archive_record(tmp_path):
one level down'."""
records, archive = _wing(tmp_path)
(records / "ci" / "good.md").write_text(_GOOD)
(archive / "api.md").write_text("# api\n\n## Records formerly in this file\n") # still exempt
(archive / "api.md").write_text("# api\n\n## Records formerly in this file\n") # still exempt
deep = archive / "ci" / "sub"
deep.mkdir(parents=True)
(deep / "broken.md").write_text("# not a record\n")
@@ -887,8 +891,7 @@ def test_junk_frontmatter_key_from_a_split_value_is_faulted(tmp_path):
PyYAML rejects this input, so the hand reader is more permissive than the writer."""
records, archive = _wing(tmp_path)
bad = records / "ci" / "corrupt.md"
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n",
"rule: >-\nthe real rule: with a colon\n"))
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\nthe real rule: with a colon\n"))
recs = dl.parse_file(bad)
assert len(recs) == 1 and recs[0].key, "precondition: this parses to one KEYED record"
assert recs[0].rule == ">-", f"precondition: the real value was truncated, got {recs[0].rule!r}"
@@ -903,7 +906,7 @@ def test_an_empty_or_missing_record_wing_is_LOUD(tmp_path):
missing = dv.record_wing_faults(tmp_path / "nope" / "records", tmp_path / "nope" / "archive")
assert missing and "missing or contains no" in missing[0], missing
records, archive = _wing(tmp_path) # exists but holds no *.md
records, archive = _wing(tmp_path) # exists but holds no *.md
empty = dv.record_wing_faults(records, archive)
assert empty and "missing or contains no" in empty[0], empty
@@ -1437,7 +1440,6 @@ def test_no_budget_flag_means_no_retirement_warning(capsys):
assert "is RETIRED and was IGNORED" not in capsys.readouterr().err
def test_main_reports_ceiling_drift_as_a_NOTICE_and_still_exits_0(capsys):
"""The fine claim's live wiring (#688): the drift notice must fire, and must NOT turn the run
red the entire point of the v5 split.
@@ -1511,8 +1513,7 @@ def test_main_actually_REPORTS_the_ceiling_and_the_trend(capsys):
err = capsys.readouterr().err
assert "prose lines across" in err, "the aggregate trend notice must always print"
over = [r.key for r in dl.all_active_records()
if r.key and dv.record_prose_lines(r) > dv.RECORD_CEILING_DEFAULT]
over = [r.key for r in dl.all_active_records() if r.key and dv.record_prose_lines(r) > dv.RECORD_CEILING_DEFAULT]
warned = "exceed the" in err and "prose ceiling" in err
assert warned is bool(over), f"ceiling warning printed={warned} but {len(over)} record(s) are over it"
if over:
+20 -22
View File
@@ -72,16 +72,17 @@ def derived_guard_files() -> set[str]:
not a guard, and globbing would drag in every helper and make the inventory a chore that gets
rubber-stamped.
"""
found = {
str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")
} | {
str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()
} | {
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
# invoked and none is individually named anywhere. Globbing is the only derivation that
# matches how they actually run.
str(p.relative_to(REPO_ROOT)) for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
}
found = (
{str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
| {str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()}
| {
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
# invoked and none is individually named anywhere. Globbing is the only derivation that
# matches how they actually run.
str(p.relative_to(REPO_ROOT))
for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
}
)
callers = list(WORKFLOWS_DIR.glob("*.yml")) + list(HOOKS_DIR.glob("*.sh"))
callers += [p for p in HUSKY_DIR.iterdir() if p.is_file()]
@@ -113,13 +114,8 @@ def wired_hook_files() -> set[str]:
text = (REPO_ROOT / ".claude" / "settings.json").read_text()
for husky in HUSKY_DIR.iterdir():
if husky.is_file():
text += "\n".join(
line for line in husky.read_text().splitlines()
if not line.lstrip().startswith("#")
)
return {
str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text
}
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
return {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text}
def inventory_rows() -> list[tuple[str, str, str, str]]:
@@ -235,9 +231,7 @@ def test_proof_rows_do_not_themselves_claim_a_proof():
for guard, kind, proof, ref in inventory_rows():
if kind == "PROOF":
assert proof == "NONE", f"{guard} is PROOF but claims Proof {proof}"
assert guard.startswith("scripts/tests/"), (
f"{guard} is marked PROOF but does not live in scripts/tests/"
)
assert guard.startswith("scripts/tests/"), f"{guard} is marked PROOF but does not live in scripts/tests/"
assert ref in ("", "-", ""), f"{guard}: PROOF rows carry no proof ref"
@@ -283,8 +277,12 @@ def test_the_summary_counts_match_the_table():
)
claimed = tuple(int(g) for g in m.groups())
actual = (
kinds["GUARD"], kinds["TOOLING"], kinds["PROOF"],
grades["MUTATION"], grades["BEHAVIOUR-ONLY"], grades["NONE"],
kinds["GUARD"],
kinds["TOOLING"],
kinds["PROOF"],
grades["MUTATION"],
grades["BEHAVIOUR-ONLY"],
grades["NONE"],
)
assert claimed == actual, (
f"the summary claims (guards, tooling, proofs, mutation, behaviour-only, none) = {claimed} "
+322 -204
View File
@@ -27,9 +27,9 @@ that is fine alone and lethal in context.
from __future__ import annotations
import os
import pty
import re
import subprocess
import pty
from pathlib import Path
import pytest
@@ -103,9 +103,7 @@ def instrumentation_faults(text: str, name: str) -> list[str]:
if not mode:
faults.append(f"{name}: etv_hook_fire_begin names no stdout mode")
elif mode.group(1) != want:
faults.append(
f"{name}: begins in {mode.group(1)!r} mode but its wiring implies {want!r}"
)
faults.append(f"{name}: begins in {mode.group(1)!r} mode but its wiring implies {want!r}")
# Order is the whole point: 8 of the hooks slurp stdin with `input=$(cat)`, and a begin
# placed after that read would find the pipe already drained — recording a fire with no
@@ -133,8 +131,9 @@ def strip_instrumentation(text: str) -> str:
out.pop()
continue
if skipping:
if line.startswith(("#", "ETV_HOOK_FIRE_LIB=", "[ -r ", "type etv_hook_fire_begin",
"etv_hook_fire_begin ")):
if line.startswith(
("#", "ETV_HOOK_FIRE_LIB=", "[ -r ", "type etv_hook_fire_begin", "etv_hook_fire_begin ")
):
continue
skipping = False
out.append(line)
@@ -173,8 +172,15 @@ def test_the_stripper_removes_EXACTLY_the_preamble_and_nothing_else():
"""
import difflib
allowed = ("# ersatztv#776", "# git hook:", "# Claude hook:", "ETV_HOOK_FIRE_LIB=",
"[ -r \"$ETV_HOOK_FIRE_LIB\"", "type etv_hook_fire_begin", "etv_hook_fire_begin ")
allowed = (
"# ersatztv#776",
"# git hook:",
"# Claude hook:",
"ETV_HOOK_FIRE_LIB=",
'[ -r "$ETV_HOOK_FIRE_LIB"',
"type etv_hook_fire_begin",
"etv_hook_fire_begin ",
)
for hook in hook_scripts():
cur = hook.read_text().splitlines(keepends=True)
stripped = strip_instrumentation("".join(cur)).splitlines(keepends=True)
@@ -189,7 +195,6 @@ def test_the_stripper_removes_EXACTLY_the_preamble_and_nothing_else():
)
def test_the_stripper_actually_strips():
"""The A/B control and the mutation are the same function. If it were a no-op, the differential
test would compare each hook against itself and pass on a wrapper that breaks everything."""
@@ -211,8 +216,9 @@ def test_every_hook_reports_that_it_fired():
for hook in hook_scripts():
faults += instrumentation_faults(hook.read_text(), hook.stem)
assert not faults, (
"these hooks do not report their own execution:\n " + "\n ".join(faults) +
"\n\nAdd the three-line preamble after the `set -` line and BEFORE any stdin read. A hook "
"these hooks do not report their own execution:\n "
+ "\n ".join(faults)
+ "\n\nAdd the three-line preamble after the `set -` line and BEFORE any stdin read. A hook "
"that does not report is one whose firing we can only infer, which is ersatztv#776."
)
@@ -235,7 +241,7 @@ def test_a_hook_that_LOSES_its_instrumentation_is_DETECTED():
def test_begin_placed_AFTER_the_stdin_read_is_DETECTED():
"""The subtler mutation: present but too late. Ordering is the property that makes it work."""
late = 'set -euo pipefail\ninput=$(cat)\nETV_HOOK_FIRE_LIB="x"\n[ -r "$ETV_HOOK_FIRE_LIB" ] && . "$ETV_HOOK_FIRE_LIB" || true\netv_hook_fire_begin demo "" capture || true\n'
late = 'set -euo pipefail\ninput=$(cat)\nETV_HOOK_FIRE_LIB="x"\n[ -r "$ETV_HOOK_FIRE_LIB" ] && . "$ETV_HOOK_FIRE_LIB" || true\netv_hook_fire_begin demo "" capture || true\n' # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
faults = instrumentation_faults(late, "demo")
assert any("reads stdin before" in f for f in faults), (
"a begin call placed after `input=$(cat)` was accepted. It would record a fire with no "
@@ -257,15 +263,15 @@ def test_begin_placed_AFTER_the_stdin_read_is_DETECTED():
# issue `deny`. An A/B over silent allows proves transparency on the one path where there is nothing
# to be transparent about.
PAYLOADS = {
"bash-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ETV_UPDATE_GOLDENS=1 dotnet test"}}',
"bash-allow": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ls -la"}}',
"git-commit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git commit -m x"}}',
"bash-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ETV_UPDATE_GOLDENS=1 dotnet test"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"bash-allow": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ls -la"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"git-commit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git commit -m x"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"nav-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__x__navigate","cwd":"%(cwd)s","tool_input":{"url":"http://h/iptv/channels.m3u"}}',
"agent-no-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p"}}',
"agent-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p","model":"sonnet"}}',
"merge": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__gitea__pull_request_write","cwd":"%(cwd)s","tool_input":{"method":"merge","owner":"timothy","repo":"ersatztv","pull_number":1}}',
"worktree-add": '{"session_id":"s","hook_event_name":"PostToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git worktree add /tmp/nope-%(nonce)s HEAD"}}',
"ui-edit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Edit","cwd":"%(cwd)s","tool_input":{"file_path":"web/src/screens/Channels.tsx"}}',
"agent-no-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"agent-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p","model":"sonnet"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"merge": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__gitea__pull_request_write","cwd":"%(cwd)s","tool_input":{"method":"merge","owner":"timothy","repo":"ersatztv","pull_number":1}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"worktree-add": '{"session_id":"s","hook_event_name":"PostToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git worktree add /tmp/nope-%(nonce)s HEAD"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"ui-edit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Edit","cwd":"%(cwd)s","tool_input":{"file_path":"web/src/screens/Channels.tsx"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
"empty": "",
"garbage": "not json at all",
}
@@ -300,17 +306,23 @@ def sandbox(tmp_path_factory):
def _run(script: Path, payload: str, env: dict, cwd: Path, arg: str | None = None):
cmd = ["bash", str(script)] + ([arg] if arg else [])
p = subprocess.run(
cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=env, timeout=60
)
p = subprocess.run(cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=env, timeout=60)
return p.returncode, p.stdout
def _git(cwd: Path, *args: str) -> None:
subprocess.run(
["git", *args], cwd=str(cwd), check=True, capture_output=True,
env={**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e",
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e"},
["git", *args],
cwd=str(cwd),
check=True,
capture_output=True,
env={
**os.environ,
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@e",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@e",
},
)
@@ -328,18 +340,23 @@ def positives(sandbox, tmp_path_factory):
def pay(**kw) -> str:
import json as _json
return _json.dumps({"session_id": "me", "hook_event_name": "PreToolUse",
"tool_name": "Bash", **kw})
return _json.dumps({"session_id": "me", "hook_event_name": "PreToolUse", "tool_name": "Bash", **kw})
# --- worktree-guard: a marker naming ANOTHER session, on a `git commit` -------------------
wt = w / "sibling"
wt.mkdir()
_git(wt, "init", "-q", ".")
(wt / ".claude-worktree-owner").write_text("SOME-OTHER-SESSION\n")
cases["pretooluse-worktree-guard"] = [(
"foreign-marker-deny",
pay(cwd=str(wt), tool_input={"command": "git commit -m x"}), None, base_env, wt,
)]
cases["pretooluse-worktree-guard"] = [
(
"foreign-marker-deny",
pay(cwd=str(wt), tool_input={"command": "git commit -m x"}),
None,
base_env,
wt,
)
]
# --- bom-guard: repo path must match `*ersatztv*`, with a staged BOM-carrying .cs ----------
br = w / "ersatztv-scratch"
@@ -347,10 +364,15 @@ def positives(sandbox, tmp_path_factory):
_git(br, "init", "-q", ".")
(br / "Bad.cs").write_bytes(b"\xef\xbb\xbfclass A {}\n")
_git(br, "add", "Bad.cs")
cases["pretooluse-bom-guard"] = [(
"staged-bom-deny",
pay(cwd=str(br), tool_input={"command": "git commit -m x"}), None, base_env, br,
)]
cases["pretooluse-bom-guard"] = [
(
"staged-bom-deny",
pay(cwd=str(br), tool_input={"command": "git commit -m x"}),
None,
base_env,
br,
)
]
# --- agent-ram: both thresholds, via a stubbed `memory_pressure` ---------------------------
ram_cases = []
@@ -362,11 +384,15 @@ def positives(sandbox, tmp_path_factory):
stub.chmod(0o755)
env = dict(base_env)
env["PATH"] = f"{bindir}:{os.environ['PATH']}"
ram_cases.append((
f"free-{pct}pct-{label}",
pay(tool_name="Agent", cwd=str(w), tool_input={"prompt": "p", "model": "sonnet"}),
None, env, w,
))
ram_cases.append(
(
f"free-{pct}pct-{label}",
pay(tool_name="Agent", cwd=str(w), tool_input={"prompt": "p", "model": "sonnet"}),
None,
env,
w,
)
)
cases["pretooluse-agent-ram"] = ram_cases
# --- decisions-guard: a validator that prints and blocks, and one that prints and passes ---
@@ -409,12 +435,16 @@ def positives(sandbox, tmp_path_factory):
_git(ahead, "add", "-A")
_git(ahead, "commit", "-qm", "advance main")
_git(ahead, "push", "-q", "origin", "main")
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(behind),
capture_output=True, text=True).stdout.strip()
cases["prepush-rebase-check"] = [(
"behind-origin-main-blocks",
f"refs/heads/feat {head} refs/heads/feat {'0' * 40}\n", None, base_env, behind,
)]
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(behind), capture_output=True, text=True).stdout.strip()
cases["prepush-rebase-check"] = [
(
"behind-origin-main-blocks",
f"refs/heads/feat {head} refs/heads/feat {'0' * 40}\n",
None,
base_env,
behind,
)
]
# clean-worktree-check: a file both MODIFIED in the tree and present in the pushed set.
dirty = clone("dirty")
@@ -422,13 +452,16 @@ def positives(sandbox, tmp_path_factory):
_git(dirty, "add", "-A")
_git(dirty, "commit", "-qm", "change file")
(dirty / "file.txt").write_text("uncommitted change\n")
dhead = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dirty),
capture_output=True, text=True).stdout.strip()
cases["prepush-clean-worktree-check"] = [(
"dirty-file-in-pushed-set-blocks",
f"refs/heads/main {dhead} refs/heads/main {'0' * 40}\n", None, base_env, dirty,
)]
dhead = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dirty), capture_output=True, text=True).stdout.strip()
cases["prepush-clean-worktree-check"] = [
(
"dirty-file-in-pushed-set-blocks",
f"refs/heads/main {dhead} refs/heads/main {'0' * 40}\n",
None,
base_env,
dirty,
)
]
# --- prepush-donewhen: a push to main closing an issue with an unticked Done-when box -------
#
@@ -443,7 +476,7 @@ def positives(sandbox, tmp_path_factory):
body = _json.dumps({"body": "## Done-when\n\n- [ ] not finished\n- [x] finished\n"}).encode()
class _Stub(http.server.BaseHTTPRequestHandler):
def do_GET(self): # noqa: N802
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
@@ -457,20 +490,23 @@ def positives(sandbox, tmp_path_factory):
threading.Thread(target=srv.serve_forever, daemon=True).start()
dw = clone("donewhen")
(dw / "src.cs").write_text("class A {}\n") # NOT docs-only, so the exemption does not apply
(dw / "src.cs").write_text("class A {}\n") # NOT docs-only, so the exemption does not apply
_git(dw, "add", "-A")
_git(dw, "commit", "-qm", "feat: thing\n\nfixes #999")
dw_head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dw),
capture_output=True, text=True).stdout.strip()
dw_prev = subprocess.run(["git", "rev-parse", "HEAD~1"], cwd=str(dw),
capture_output=True, text=True).stdout.strip()
dw_head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dw), capture_output=True, text=True).stdout.strip()
dw_prev = subprocess.run(["git", "rev-parse", "HEAD~1"], cwd=str(dw), capture_output=True, text=True).stdout.strip()
dw_env = dict(base_env)
dw_env["ETV_GITEA_URL"] = f"http://127.0.0.1:{srv.server_address[1]}"
dw_env["ETV_GITEA_BASICAUTH"] = "stub:stub"
cases["prepush-donewhen"] = [(
"unticked-donewhen-blocks-push-to-main",
f"refs/heads/main {dw_head} refs/heads/main {dw_prev}\n", None, dw_env, dw,
)]
cases["prepush-donewhen"] = [
(
"unticked-donewhen-blocks-push-to-main",
f"refs/heads/main {dw_head} refs/heads/main {dw_prev}\n",
None,
dw_env,
dw,
)
]
return cases
@@ -482,10 +518,15 @@ def _ab_cases(hook: Path, sandbox, positives) -> list[tuple]:
args = [None, "start", "finish"] if hook.stem == "design-sync-reminder" else [None]
for name, template in PAYLOADS.items():
for arg in args:
cases.append((
f"matrix:{name}:{arg}", template % {"cwd": str(root), "nonce": name},
arg, env, root,
))
cases.append(
(
f"matrix:{name}:{arg}",
template % {"cwd": str(root), "nonce": name},
arg,
env,
root,
)
)
for label, payload, arg, penv, cwd in positives.get(hook.stem, []):
cases.append((f"positive:{label}", payload, arg, penv, cwd))
return cases
@@ -513,8 +554,7 @@ def _ab_run(hook: Path, sandbox, case, tag_root: str):
def one(script: Path, t: str):
cmd = ["bash", str(script)] + ([arg] if arg else [])
p = subprocess.run(cmd, input=payload.encode(), capture_output=True,
cwd=str(cwd), env=side(t), timeout=90)
p = subprocess.run(cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=side(t), timeout=90)
return p.returncode, p.stdout, p.stderr
return one(control, "control"), one(hook, "instrumented")
@@ -613,11 +653,16 @@ def test_the_worktree_markers_SIDE_EFFECT_is_unchanged(sandbox, tmp_path):
script = tmp_path / "control-marker.sh"
script.write_text(strip_instrumentation(hook.read_text()))
import json as _json
payload = _json.dumps({
"session_id": "SESSION-XYZ", "hook_event_name": "PostToolUse", "tool_name": "Bash",
"cwd": str(repo),
"tool_input": {"command": f"git worktree add {target} HEAD"},
})
payload = _json.dumps(
{
"session_id": "SESSION-XYZ",
"hook_event_name": "PostToolUse",
"tool_name": "Bash",
"cwd": str(repo),
"tool_input": {"command": f"git worktree add {target} HEAD"},
}
)
_git(repo, "worktree", "add", "-q", str(target), "HEAD")
_run(script, payload, env, repo)
marker = target / ".claude-worktree-owner"
@@ -641,15 +686,13 @@ def test_stdout_is_replayed_BYTE_EXACT(sandbox):
hook.write_text(
"#!/usr/bin/env bash\nset -euo pipefail\n"
f'. "{SINK}"\n'
"etv_hook_fire_begin trailing \"\" capture || true\n"
'etv_hook_fire_begin trailing "" capture || true\n'
"input=$(cat)\n"
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}\n\n"' + "\n"
)
rc, out = _run(hook, '{"tool_name":"Bash"}', env, root)
assert rc == 0
assert out == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}\n\n', (
f"trailing bytes were altered: {out!r}"
)
assert out == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}\n\n', f"trailing bytes were altered: {out!r}"
@pytest.mark.parametrize("code", [0, 1, 2, 3])
@@ -704,8 +747,12 @@ def test_a_TTY_stdin_is_not_slurped(sandbox):
master, slave = pty.openpty()
try:
p = subprocess.Popen(
["bash", str(hook)], stdin=slave, stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL, cwd=str(root), env=env,
["bash", str(hook)],
stdin=slave,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
cwd=str(root),
env=env,
)
os.close(slave)
try:
@@ -751,11 +798,11 @@ def test_logging_failure_does_not_break_the_hook(sandbox):
@pytest.mark.parametrize(
"emit,mode,expected",
[
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}', "capture", "deny"),
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"ask\"}}', "capture", "ask"),
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"allow\"}}', "capture", "allow"),
(r'{\"hookSpecificOutput\":{\"additionalContext\":\"hi\"}}', "capture", "context"),
(r'{\"decision\":\"block\",\"reason\":\"r\"}', "capture", "block"),
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}", "capture", "deny"),
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"ask\"}}", "capture", "ask"),
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"allow\"}}", "capture", "allow"),
(r"{\"hookSpecificOutput\":{\"additionalContext\":\"hi\"}}", "capture", "context"),
(r"{\"decision\":\"block\",\"reason\":\"r\"}", "capture", "block"),
("", "capture", "no-op"),
("", "stream", "pass"),
],
@@ -822,7 +869,10 @@ def test_the_report_names_hooks_that_NEVER_fired(sandbox):
)
p = subprocess.run(
["bash", str(SINK), "report", "--all", "--dir", str(logdir)],
capture_output=True, cwd=str(REPO_ROOT), env=env, timeout=60,
capture_output=True,
cwd=str(REPO_ROOT),
env=env,
timeout=60,
)
out = p.stdout.decode()
assert p.returncode == 0, p.stderr.decode()
@@ -859,7 +909,10 @@ def test_the_report_REFUSES_an_empty_population(tmp_path, sandbox):
runenv.pop("CLAUDE_PROJECT_DIR", None)
p = subprocess.run(
["bash", str(copied), "report", "--all", "--dir", str(tmp_path / "log")],
capture_output=True, cwd=str(tmp_path), env=runenv, timeout=60,
capture_output=True,
cwd=str(tmp_path),
env=runenv,
timeout=60,
)
assert p.returncode == 2, (
"the report exited 0 over an empty hook population. It must refuse rather than print a "
@@ -895,12 +948,11 @@ def test_stderr_is_NOT_silenced(sandbox):
'printf "IMPORTANT DIAGNOSTIC\\n" >&2\n'
"exit 1\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}',
capture_output=True, cwd=str(root), env=env, timeout=60)
assert p.returncode == 1
assert b"IMPORTANT DIAGNOSTIC" in p.stderr, (
f"the hook's stderr was swallowed by the instrumentation: {p.stderr!r}"
p = subprocess.run(
["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True, cwd=str(root), env=env, timeout=60
)
assert p.returncode == 1
assert b"IMPORTANT DIAGNOSTIC" in p.stderr, f"the hook's stderr was swallowed by the instrumentation: {p.stderr!r}"
def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
@@ -923,8 +975,14 @@ def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
)
tmp = root / "vanish-tmp"
tmp.mkdir(exist_ok=True)
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
cwd=str(root), env={**env, "TMPDIR": str(tmp)}, timeout=60)
p = subprocess.run(
["bash", str(hook)],
input=b'{"tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "TMPDIR": str(tmp)},
timeout=60,
)
assert p.returncode == 0
assert b'"permissionDecision":"deny"' in p.stdout, (
"the guard's deny was DISCARDED when its stdout scratch file vanished mid-run. The hook "
@@ -932,24 +990,22 @@ def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
)
@pytest.mark.parametrize(
"emit,code,expected",
[
# A NON-CANONICAL value is recorded as non-canonical, not laundered into a valid decision.
# This case went both ways before settling: first filed as unclassified prose, then
# lowercased into a clean `deny` (manufacturing a decision the harness may never honour).
(r'{\"permissionDecision\":\"Deny\"}', 0, "unrecognized"),
(r"{\"permissionDecision\":\"Deny\"}", 0, "unrecognized"),
# A non-zero status neither ERASES a printed decision nor annotates it. The status has its
# own field; the decision field states the decision and nothing else.
(r'{\"permissionDecision\":\"deny\"}', 1, "deny"),
(r"{\"permissionDecision\":\"deny\"}", 1, "deny"),
# A failure with nothing classifiable IS an error.
("", 1, "error"),
# Exit 2 is the harness's block channel and DOMINATES the printed JSON. Recording the
# printed `allow` would report a permit for a call that was actually refused — the one
# direction a log of security decisions must never be wrong in.
(r'{\"permissionDecision\":\"allow\"}', 2, "deny-exit2"),
(r"{\"permissionDecision\":\"allow\"}", 2, "deny-exit2"),
],
)
def test_the_classifier_does_not_MISREPORT(sandbox, emit, code, expected):
@@ -961,12 +1017,16 @@ def test_the_classifier_does_not_MISREPORT(sandbox, emit, code, expected):
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{SINK}"\n'
'etv_hook_fire_begin cls "" capture || true\n'
"input=$(cat)\n"
+ (f'printf "{emit}"\n' if emit else "")
+ f"exit {code}\n"
"input=$(cat)\n" + (f'printf "{emit}"\n' if emit else "") + f"exit {code}\n"
)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env=runenv,
timeout=60,
)
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root), env=runenv, timeout=60)
rec = [ln for ln in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in ln]
assert f'"decision":"{expected}"' in rec[0], f"expected {expected}, got {rec[0]}"
@@ -983,9 +1043,14 @@ def test_the_word_additionalContext_in_PROSE_is_not_a_decision(sandbox):
"input=$(cat)\n"
'printf "note: this hook does not use additionalContext at all\\n"\n'
)
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
rec = [ln for ln in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in ln]
assert '"decision":"output"' in rec[0], f"prose was classified as a decision: {rec[0]}"
@@ -1001,14 +1066,17 @@ def test_the_session_id_cannot_ESCAPE_the_log_directory(sandbox, tmp_path):
logdir = tmp_path / "logs"
hook = tmp_path / "esc.sh"
hook.write_text(
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{SINK}"\n'
'etv_hook_fire_begin esc "" capture || true\n'
"input=$(cat)\n"
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin esc "" capture || true\ninput=$(cat)\n'
)
evil = '{"session_id":"../../escaped","tool_name":"Bash"}'
subprocess.run(["bash", str(hook)], input=evil.encode(), capture_output=True,
cwd=str(tmp_path), env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
subprocess.run(
["bash", str(hook)],
input=evil.encode(),
capture_output=True,
cwd=str(tmp_path),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
written = list(logdir.glob("*.jsonl"))
assert written, "nothing was logged at all"
for f in written:
@@ -1040,8 +1108,14 @@ def test_the_suite_does_not_write_to_the_PRODUCTION_log(sandbox):
'etv_hook_fire_begin prodcheck "" capture || true\n'
"input=$(cat)\n"
)
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root), env=env, timeout=60)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env=env,
timeout=60,
)
after = sorted(p.stat().st_mtime_ns for p in default.glob("*.jsonl")) if default.exists() else []
assert before == after, "a test run modified the production hook-fire log"
@@ -1075,16 +1149,17 @@ def test_DELETING_the_replay_makes_the_differential_go_RED(sandbox, positives, t
"input=$(cat)\n"
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
cwd=str(root), env={**env, "TMPDIR": str(tmp_path)}, timeout=60)
assert p.stdout == b"", (
"the mutation did not actually disarm the replay, so this proof establishes nothing: "
f"stdout={p.stdout!r}"
p = subprocess.run(
["bash", str(hook)],
input=b'{"tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "TMPDIR": str(tmp_path)},
timeout=60,
)
assert p.stdout == b"", (
f"the mutation did not actually disarm the replay, so this proof establishes nothing: stdout={p.stdout!r}"
)
def test_no_exec_in_the_sink_carries_a_STDERR_REDIRECT():
@@ -1123,17 +1198,20 @@ def test_the_log_and_the_REPLAY_agree_when_the_hook_uses_fd_4(sandbox):
f'. "{SINK}"\n'
'etv_hook_fire_begin fd4 "" capture || true\n'
"input=$(cat)\n"
'exec 4</dev/null\n' # the hook takes fd 4 for itself
"exec 4</dev/null\n" # the hook takes fd 4 for itself
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
p = subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
assert b'"permissionDecision":"deny"' in p.stdout, f"replay lost the decision: {p.stdout!r}"
exits = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
assert '"decision":"deny"' in exits[0], (
f"the harness saw a deny but the log recorded something else: {exits[0]}"
)
assert '"decision":"deny"' in exits[0], f"the harness saw a deny but the log recorded something else: {exits[0]}"
def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
@@ -1160,9 +1238,14 @@ def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
"input=$(cat)\n"
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(tmp_path),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
p = subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(tmp_path),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
assert p.returncode == 0
assert p.stdout == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}'
assert p.stderr == b"", (
@@ -1171,10 +1254,6 @@ def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
)
def test_NUL_bytes_in_hook_output_survive(sandbox):
"""A shell variable cannot hold a NUL, so replaying through `$(...)` silently drops them.
@@ -1192,8 +1271,9 @@ def test_NUL_bytes_in_hook_output_survive(sandbox):
"input=$(cat)\n"
r"printf 'a\000b\n'" + "\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
cwd=str(root), env=env, timeout=60)
p = subprocess.run(
["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True, cwd=str(root), env=env, timeout=60
)
assert p.stdout == b"a\x00b\n", f"NUL-containing output was mangled: {p.stdout!r}"
assert b"null byte" not in p.stderr, f"a warning leaked to the harness: {p.stderr!r}"
@@ -1202,8 +1282,8 @@ def test_NUL_bytes_in_hook_output_survive(sandbox):
"emit,code,expected",
[
# Every non-canonical value, not just the ones a restricted character class admits.
(r'{\"permissionDecision\":\"deny2\"}', 0, "unrecognized"),
(r'{\"permissionDecision\":\"deny_now\"}', 0, "unrecognized"),
(r"{\"permissionDecision\":\"deny2\"}", 0, "unrecognized"),
(r"{\"permissionDecision\":\"deny_now\"}", 0, "unrecognized"),
# Unclassified output plus a FAILING exit is an error, not `output`: the report histograms
# the decision, so filing it as `output` hid the failure entirely.
("diagnostic text", 1, "error"),
@@ -1217,13 +1297,16 @@ def test_odd_values_and_failing_exits_are_not_LAUNDERED(sandbox, emit, code, exp
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{SINK}"\n'
'etv_hook_fire_begin odd "" capture || true\n'
"input=$(cat)\n"
+ (f'printf "{emit}"\n' if emit else "")
+ f"exit {code}\n"
"input=$(cat)\n" + (f'printf "{emit}"\n' if emit else "") + f"exit {code}\n"
)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
rec = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
assert f'"decision":"{expected}"' in rec[0], f"expected {expected}, got {rec[0]}"
@@ -1247,9 +1330,12 @@ def test_an_INHERITED_flushed_flag_does_not_disable_reporting(sandbox):
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
)
p = subprocess.run(
["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}', capture_output=True,
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir), "ETV_HOOK_FIRE_FLUSHED": "1"}, timeout=60,
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir), "ETV_HOOK_FIRE_FLUSHED": "1"},
timeout=60,
)
assert p.stdout == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}', (
f"an inherited FLUSHED=1 left stdout redirected and swallowed the decision: {p.stdout!r}"
@@ -1284,9 +1370,7 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
)
instrumented = root / f"sigab-{signame}.sh"
instrumented.write_text(
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{SINK}"\n'
f'etv_hook_fire_begin sigab "" stream || true\n' + body
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin sigab "" stream || true\n' + body
)
control = root / f"sigab-control-{signame}.sh"
control.write_text("#!/usr/bin/env bash\nset -uo pipefail\n" + body)
@@ -1302,9 +1386,15 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
# anything. Measured: pid-only gives control 4.0s and trapped 4.1s; group signalling
# gives control 0.002s and trapped 0.054s. A supervisor kills the group, so this is
# also the shape that actually occurs.
p = subprocess.Popen(["bash", str(script)], stdin=fh, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, cwd=str(root), env=env,
start_new_session=True)
p = subprocess.Popen(
["bash", str(script)],
stdin=fh,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd=str(root),
env=env,
start_new_session=True,
)
time.sleep(1.0)
t0 = time.monotonic()
os.killpg(os.getpgid(p.pid), sig)
@@ -1312,7 +1402,8 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
results[tag] = (p.returncode, time.monotonic() - t0, out, err)
(rc_a, dt_a, out_a, err_a), (rc_b, dt_b, out_b, err_b) = (
results["control"], results["instrumented"],
results["control"],
results["instrumented"],
)
assert out_a == out_b, f"SIG{signame}: stdout differs under signal: {out_a!r} vs {out_b!r}"
@@ -1329,13 +1420,11 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
# require one of bash's four signal words followed by `:` or whitespace, so an ordinary
# diagnostic still fails the comparison.
return b"\n".join(
ln for ln in e.split(b"\n")
if not re.search(rb"(^|\s)(Terminated|Hangup|Interrupt|Killed)(:|\s|$)", ln)
ln for ln in e.split(b"\n") if not re.search(rb"(^|\s)(Terminated|Hangup|Interrupt|Killed)(:|\s|$)", ln)
)
assert strip_jobnotice(err_a) == strip_jobnotice(err_b), (
f"SIG{signame}: stderr differs under signal beyond bash's job-control notice: "
f"{err_a!r} vs {err_b!r}"
f"SIG{signame}: stderr differs under signal beyond bash's job-control notice: {err_a!r} vs {err_b!r}"
)
assert rc_a == rc_b, (
f"SIG{signame}: exit status differs, control={rc_a} instrumented={rc_b}. git and the "
@@ -1348,14 +1437,15 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
)
@pytest.mark.parametrize("locale_env", [
{"LANG": "en_US.UTF-8"},
{"LC_CTYPE": "en_US.UTF-8"},
{"LC_CTYPE": "UTF-8"}, # macOS Terminal's default, and the case that broke the fix
{"LC_ALL": "en_US.UTF-8"},
])
@pytest.mark.parametrize(
"locale_env",
[
{"LANG": "en_US.UTF-8"},
{"LC_CTYPE": "en_US.UTF-8"},
{"LC_CTYPE": "UTF-8"}, # macOS Terminal's default, and the case that broke the fix
{"LC_ALL": "en_US.UTF-8"},
],
)
def test_an_INVALID_UTF8_byte_in_a_decision_is_still_classified(tmp_path, locale_env):
"""`local LC_ALL=C` does not export, so the child `sed`/`tr` never saw it — the fix was INERT.
@@ -1387,10 +1477,15 @@ def test_an_INVALID_UTF8_byte_in_a_decision_is_still_classified(tmp_path, locale
'printf \'{"hookSpecificOutput":{"permissionDecision":"deny",'
'"permissionDecisionReason":"caf\\xe9"}}\'\n'
)
env = {"PATH": os.environ["PATH"], "HOME": os.environ["HOME"],
"ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(tmp_path), env=env, timeout=60)
env = {"PATH": os.environ["PATH"], "HOME": os.environ["HOME"], "ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
p = subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(tmp_path),
env=env,
timeout=60,
)
assert p.stderr == b"", f"{locale_env}: a locale diagnostic leaked to the harness: {p.stderr!r}"
record = (logdir / "s1.jsonl").read_text()
@@ -1419,16 +1514,24 @@ def test_a_NESTED_identity_field_does_not_outrank_the_TOP_LEVEL_one(tmp_path):
"input=$(cat)\n"
)
import json as _json
payload = _json.dumps({
"session_id": "TOP-SESSION",
"hook_event_name": "PreToolUse",
"tool_name": "TopTool",
"tool_input": {"filler": "x" * 1000},
"tool_response": {"session_id": "NESTED-SESSION", "tool_name": "NestedTool"},
})
subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
cwd=str(tmp_path), env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60)
payload = _json.dumps(
{
"session_id": "TOP-SESSION",
"hook_event_name": "PreToolUse",
"tool_name": "TopTool",
"tool_input": {"filler": "x" * 1000},
"tool_response": {"session_id": "NESTED-SESSION", "tool_name": "NestedTool"},
}
)
subprocess.run(
["bash", str(hook)],
input=payload.encode(),
capture_output=True,
cwd=str(tmp_path),
env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
assert (logdir / "TOP-SESSION.jsonl").exists(), (
f"records filed under the wrong session: {[f.name for f in logdir.glob('*.jsonl')]}"
@@ -1451,9 +1554,14 @@ def test_a_PRESENT_but_empty_decision_is_not_laundered(sandbox):
"input=$(cat)\n"
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"\"}}"' + "\n"
)
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=60,
)
rec = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
assert '"decision":"unrecognized"' in rec[0], f"an empty decision was laundered: {rec[0]}"
@@ -1473,21 +1581,25 @@ def test_identity_fields_BEYOND_the_fast_path_cap_are_still_found(tmp_path):
hook = tmp_path / "big.sh"
hook.write_text(
"#!/usr/bin/env bash\nset -uo pipefail\n"
f'. "{SINK}"\n'
'etv_hook_fire_begin big "" capture || true\n'
"input=$(cat)\n"
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin big "" capture || true\ninput=$(cat)\n'
)
logdir = tmp_path / "biglog"
payload = _json.dumps({
"tool_input": {"filler": "x" * 400_000},
"session_id": "TOP-SESSION",
"hook_event_name": "PreToolUse",
"tool_name": "TopTool",
})
subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
cwd=str(tmp_path), env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=90)
payload = _json.dumps(
{
"tool_input": {"filler": "x" * 400_000},
"session_id": "TOP-SESSION",
"hook_event_name": "PreToolUse",
"tool_name": "TopTool",
}
)
subprocess.run(
["bash", str(hook)],
input=payload.encode(),
capture_output=True,
cwd=str(tmp_path),
env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
timeout=90,
)
assert (logdir / "TOP-SESSION.jsonl").exists(), (
"identity beyond the fast-path cap was not found, so the fire filed under "
@@ -1510,11 +1622,17 @@ def test_the_field_helper_does_not_LEAK_into_the_hooks_namespace(sandbox):
f'. "{SINK}"\n'
'etv_hook_fire_begin ns "" capture || true\n'
"input=$(cat)\n"
'if declare -F _etv_field >/dev/null 2>&1; then echo LEAKED_etv_field; fi\n'
'if declare -F etv_hook_fire__field >/dev/null 2>&1; then echo LEAKED_namespaced; fi\n'
"if declare -F _etv_field >/dev/null 2>&1; then echo LEAKED_etv_field; fi\n"
"if declare -F etv_hook_fire__field >/dev/null 2>&1; then echo LEAKED_namespaced; fi\n"
"echo CLEAN\n"
)
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True, cwd=str(root), env=env, timeout=60)
p = subprocess.run(
["bash", str(hook)],
input=b'{"session_id":"s1","tool_name":"Bash"}',
capture_output=True,
cwd=str(root),
env=env,
timeout=60,
)
assert b"LEAKED" not in p.stdout, f"a helper leaked into the hook's namespace: {p.stdout!r}"
assert b"CLEAN" in p.stdout
+72 -58
View File
@@ -51,12 +51,12 @@ def preflight(tmp_path):
so every test passed.
"""
shim = bindir / "jq"
body = "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then\n"
body = '#!/bin/sh\nif [ "$1" = "--version" ]; then\n'
if version_line:
body += ' printf "%%s\\n" %s\n' % _shq(version_line)
body += f' printf "%s\\n" {_shq(version_line)}\n'
if stderr:
body += ' printf "%%s\\n" %s >&2\n' % _shq(stderr)
body += " exit %d\nfi\nexit 0\n" % exit_code
body += f' printf "%s\\n" {_shq(stderr)} >&2\n'
body += f" exit {exit_code}\nfi\nexit 0\n"
shim.write_text(body)
shim.chmod(0o755)
@@ -73,8 +73,7 @@ def preflight(tmp_path):
# but jq itself (`command -v` is a builtin, and bash is invoked by absolute path), so
# there is nothing to keep.
env["PATH"] = str(bindir)
return subprocess.run([BASH, str(SCRIPT), *args],
env=env, capture_output=True, text=True)
return subprocess.run([BASH, str(SCRIPT), *args], env=env, capture_output=True, text=True)
def run_bytes(self, *args):
"""Same, but WITHOUT text mode.
@@ -87,8 +86,7 @@ def preflight(tmp_path):
"""
env = dict(os.environ)
env["PATH"] = str(bindir)
return subprocess.run([BASH, str(SCRIPT), *args],
env=env, capture_output=True)
return subprocess.run([BASH, str(SCRIPT), *args], env=env, capture_output=True)
return Handle()
@@ -172,11 +170,15 @@ def test_expect_without_a_value_is_a_usage_error_WITH_output(preflight):
# the script exited 0 having asserted NOTHING. That is this script's own stated failure mode,
# reproduced inside itself, which is why these cases are pinned rather than left to inspection.
@pytest.mark.parametrize("version_line", [
"jq version 1.6", # some distro wrappers print this form
"JQ-1.6",
"jq-1.6-dirty",
])
@pytest.mark.parametrize(
"version_line",
[
"jq version 1.6", # some distro wrappers print this form
"JQ-1.6",
"jq-1.6-dirty",
],
)
def test_unusual_but_parseable_version_forms_are_accepted(preflight, version_line):
preflight.with_jq(version_line)
r = preflight.run()
@@ -190,7 +192,8 @@ def test_unparseable_version_fails_CLOSED_rather_than_asserting_nothing(prefligh
r = preflight.run()
assert r.returncode == 1, (
f"{version_line!r} exited {r.returncode}: an unparsed version must never reach — or "
"silently skip — the floor assertion")
"silently skip — the floor assertion"
)
assert "could not parse" in r.stderr
@@ -202,20 +205,21 @@ def test_a_jq_that_cannot_START_fails_closed(preflight):
discarding the exit status, so that message became the parse input, `2.34` matched, and the floor
was certified green on a jq that cannot run at all.
"""
preflight.with_jq(
"", stderr="jq: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found",
exit_code=127)
preflight.with_jq("", stderr="jq: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found", exit_code=127)
r = preflight.run()
assert r.returncode == 1
assert "cannot run" in r.stderr
assert "parsed 2.34" not in r.stdout, "stderr must never be parsed as a version"
@pytest.mark.parametrize("version_line", [
"warning: something 3.14", # a noise line carrying a plausible number
"2026.07.26 jq-1.6", # a date prefix, which outranks the real version if unanchored
"jq-master-v0.0.0-1.6",
])
@pytest.mark.parametrize(
"version_line",
[
"warning: something 3.14", # a noise line carrying a plausible number
"2026.07.26 jq-1.6", # a date prefix, which outranks the real version if unanchored
"jq-master-v0.0.0-1.6",
],
)
def test_a_number_that_is_not_the_VERSION_is_not_accepted_as_one(preflight, version_line):
"""Matching the first `<digits>.<digits>` ANYWHERE let a prefix win over the real version.
`2026.07.26 jq-1.6` parsed as 2026.07 and sailed over the floor. The pattern is anchored to the
@@ -226,10 +230,13 @@ def test_a_number_that_is_not_the_VERSION_is_not_accepted_as_one(preflight, vers
assert "could not parse" in r.stderr
@pytest.mark.parametrize("version_line", [
"jq-99999999999999999999999.0",
"jq-1.99999999999999999999999",
])
@pytest.mark.parametrize(
"version_line",
[
"jq-99999999999999999999999.0",
"jq-1.99999999999999999999999",
],
)
def test_an_OUT_OF_RANGE_digit_run_fails_closed(preflight, version_line):
"""The round-1 fail-open mechanism, resurrected via an over-long number.
@@ -247,25 +254,28 @@ def test_an_OUT_OF_RANGE_digit_run_fails_closed(preflight, version_line):
assert r.returncode == 1, f"{version_line!r} exited 0 — the floor was not asserted"
@pytest.mark.parametrize("version_line", [
# Killed by the SEPARATOR restriction (a blank separator must be followed by `version`).
"jq\n2.34: cannot load shared library",
"jq\n\n\n99.9",
"jq -- 2.34 (real jq-1.6)",
"jq\t\t9.9",
# Killed ONLY by the first-line slice + `[[:blank:]]`. These carry the literal word `version`,
# so the separator restriction is satisfied and cannot save us — the newline must be excluded
# from the separator class AND the parse confined to line one.
#
# Without these, a round-5 mutation check found that reverting BOTH of those changes together
# (`[[:blank:]]`→`[[:space:]]` and parsing `$raw` instead of `$first`) left the whole suite
# GREEN: the four cases above are all killed by the separator alone, so they attributed the fix
# to the wrong layer. A test that passes for the wrong reason is how the previous three rounds
# each shipped a defect.
"jq\nversion\n9.9",
"jq\nversion 9.9",
"jq \n version \n 9.9",
])
@pytest.mark.parametrize(
"version_line",
[
# Killed by the SEPARATOR restriction (a blank separator must be followed by `version`).
"jq\n2.34: cannot load shared library",
"jq\n\n\n99.9",
"jq -- 2.34 (real jq-1.6)",
"jq\t\t9.9",
# Killed ONLY by the first-line slice + `[[:blank:]]`. These carry the literal word `version`,
# so the separator restriction is satisfied and cannot save us — the newline must be excluded
# from the separator class AND the parse confined to line one.
#
# Without these, a round-5 mutation check found that reverting BOTH of those changes together
# (`[[:blank:]]`→`[[:space:]]` and parsing `$raw` instead of `$first`) left the whole suite
# GREEN: the four cases above are all killed by the separator alone, so they attributed the fix
# to the wrong layer. A test that passes for the wrong reason is how the previous three rounds
# each shipped a defect.
"jq\nversion\n9.9",
"jq\nversion 9.9",
"jq \n version \n 9.9",
],
)
def test_a_number_AFTER_the_jq_token_is_not_reachable_across_filler(preflight, version_line):
"""Two independent layers keep a stray number from being read as the version, and both are
pinned here: the separator must be one of the forms real jq emits (`jq-1.6` / `jq version 1.6`),
@@ -306,14 +316,17 @@ def test_the_observability_line_stays_on_ONE_line(preflight):
assert "trailing noise" not in r.stdout
@pytest.mark.parametrize("version_line,expected", [
("jq-1.6 (Debian 1.6-2.1)", "1.6"), # distro packaging suffix
("jq-1.10", "1.10"), # two-digit minor: must compare numerically, not lexically
("jq-1.7.1", "1.7"),
("jq-1.6.0", "1.6"),
("jq-v1.6", "1.6"),
("JQ-1.6", "1.6"),
])
@pytest.mark.parametrize(
"version_line,expected",
[
("jq-1.6 (Debian 1.6-2.1)", "1.6"), # distro packaging suffix
("jq-1.10", "1.10"), # two-digit minor: must compare numerically, not lexically
("jq-1.7.1", "1.7"),
("jq-1.6.0", "1.6"),
("jq-v1.6", "1.6"),
("JQ-1.6", "1.6"),
],
)
def test_legitimate_forms_still_parse_to_the_right_version(preflight, version_line, expected):
preflight.with_jq(version_line)
r = preflight.run()
@@ -323,6 +336,7 @@ def test_legitimate_forms_still_parse_to_the_right_version(preflight, version_li
# --- Wiring guards: the preflight is worthless if a caller silently stops running it ------------
def test_script_tests_pins_the_jq_version():
"""The pin is the tripwire, so its presence is asserted rather than merely commented.
@@ -333,8 +347,7 @@ def test_script_tests_pins_the_jq_version():
is `main`, which does not yet contain `scripts/jq-preflight.sh`.
"""
pr_checks = (WORKFLOWS / "pr-checks.yml").read_text()
assert "jq-preflight.sh --expect" in pr_checks, \
"script-tests must pin the jq version — that pin is the tripwire"
assert "jq-preflight.sh --expect" in pr_checks, "script-tests must pin the jq version — that pin is the tripwire"
def test_review_verdict_never_pins_a_jq_version():
@@ -344,6 +357,7 @@ def test_review_verdict_never_pins_a_jq_version():
follow-up PR adds the floor-only call rather than being a comment someone can miss.
"""
review_verdict = (WORKFLOWS / "review-verdict.yml").read_text()
assert "jq-preflight.sh --expect" not in review_verdict, \
("review-verdict.yml must NOT pin a jq version: it writes the required review-verdict/h10 "
"status, so a pin would deadlock every merge on a jq bump (ersatztv#648)")
assert "jq-preflight.sh --expect" not in review_verdict, (
"review-verdict.yml must NOT pin a jq version: it writes the required review-verdict/h10 "
"status, so a pin would deadlock every merge on a jq bump (ersatztv#648)"
)
+26 -21
View File
@@ -33,7 +33,7 @@ SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
# The PR is deliberately NOT docs-only: the docs-only exemption short-circuits the whole gate, so a
# docs PR would never reach the base check and the tests would pass without exercising it.
CURL_SHIM = r'''#!/usr/bin/env python3
CURL_SHIM = r"""#!/usr/bin/env python3
import json, os, sys, pathlib, urllib.parse
state = pathlib.Path(os.environ["STUB_DIR"])
@@ -75,14 +75,18 @@ if "/pulls/" in url:
sys.exit(0)
print("{}")
'''
"""
@pytest.fixture
def hook(tmp_path):
bindir = tmp_path / "bin"; bindir.mkdir()
curl = bindir / "curl"; curl.write_text(CURL_SHIM); curl.chmod(0o755)
state = tmp_path / "state"; state.mkdir()
bindir = tmp_path / "bin"
bindir.mkdir()
curl = bindir / "curl"
curl.write_text(CURL_SHIM)
curl.chmod(0o755)
state = tmp_path / "state"
state.mkdir()
(state / "live_base").write_text("main")
(state / "verdict_desc").write_text("Review-verdict: MERGEABLE @ a9e3e23 (base: main)")
@@ -90,7 +94,7 @@ def hook(tmp_path):
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["STUB_SHA"] = SHA
env["ETV_GITEA_TOKEN"] = "stub"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env.pop("ETV_GITEA_BASICAUTH", None)
@@ -103,10 +107,8 @@ def hook(tmp_path):
(state / "verdict_desc").write_text(desc)
def decision(self):
payload = {"tool_input": {"method": "merge", "owner": "timothy",
"repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
env=env, capture_output=True, text=True)
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
assert r.returncode == 0, r.stderr
if not r.stdout.strip():
return None
@@ -124,21 +126,24 @@ def test_a_retargeted_base_denies_a_verdict_formed_against_the_old_one(hook):
reason = hook.reason()
assert "deny" in reason, "a verdict formed against a different base was allowed to stand"
assert "release/26.4" in reason and "main" in reason, (
"the deny must name both bases; a reader cannot act on 'the base changed'")
"the deny must name both bases; a reader cannot act on 'the base changed'"
)
def test_positive_control_an_unchanged_base_does_not_trigger_the_base_deny(hook):
"""Without this, the test above could pass because the hook denies on every path — which it
very nearly does, since this PR is non-docs and the rest of the gate is unstubbed."""
reason = hook.reason()
assert "ersatztv#632" not in reason, (
"the base check fired on a PR whose base never moved")
assert "ersatztv#632" not in reason, "the base check fired on a PR whose base never moved"
@pytest.mark.parametrize("desc", [
"Review-verdict: MERGEABLE @ a9e3e23", # posted before #632
"NONE", # no verdict status on this head at all
])
@pytest.mark.parametrize(
"desc",
[
"Review-verdict: MERGEABLE @ a9e3e23", # posted before #632
"NONE", # no verdict status on this head at all
],
)
def test_a_verdict_with_no_recorded_base_gets_no_opinion(hook, desc):
"""Graceful adoption. Denying here would block every in-flight PR the day this lands, and the
window closes on its own: verdicts are per-head and short-lived, so every verdict posted after
@@ -151,7 +156,8 @@ def test_a_verdict_with_no_recorded_base_gets_no_opinion(hook, desc):
hook.set_live_base("release/26.4")
hook.set_verdict_description(desc)
assert "base" not in hook.reason(), (
"a pre-#632 verdict drew a base-related decision for a field it could not have carried")
"a pre-#632 verdict drew a base-related decision for a field it could not have carried"
)
@pytest.mark.parametrize("failure", ["SCALAR-ROW", "NONSTRING-DESC"])
@@ -173,7 +179,7 @@ def test_a_malformed_status_MEMBER_asks_too(hook, failure):
@pytest.mark.parametrize("failure", ["TRANSPORT-ERROR", "GARBAGE"])
def test_an_UNREADABLE_status_response_asks_rather_than_skipping_the_check(hook, failure):
""""Could not check" is a third outcome, not a quiet synonym for "no base recorded".
""" "Could not check" is a third outcome, not a quiet synonym for "no base recorded".
The first draft collapsed the two: an unreadable status response produced an empty
`recorded_base`, took the graceful-adoption path, and skipped validation in silence after
@@ -202,5 +208,4 @@ def test_the_comparator_is_the_base_REF_not_its_tip_sha():
A base branch that merely ADVANCES must be silent here; rebasing onto it moves the head sha,
which the per-sha binding already covers."""
assert ".base.ref" in HOOK.read_text(), "the hook must compare the base BRANCH, not its tip sha"
assert ".base.sha" not in HOOK.read_text(), (
"comparing base.sha deadlocks every open PR whenever main advances")
assert ".base.sha" not in HOOK.read_text(), "comparing base.sha deadlocks every open PR whenever main advances"
+64 -52
View File
@@ -33,7 +33,7 @@ HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
# Serves paged `pulls/N/files`, plus the minimal PR object the hook reads first.
CURL_SHIM = r'''#!/usr/bin/env python3
CURL_SHIM = r"""#!/usr/bin/env python3
import json, os, sys, pathlib, urllib.parse
state = pathlib.Path(os.environ["STUB_DIR"])
@@ -88,7 +88,7 @@ if "/pulls/" in url:
sys.exit(0)
print("{}")
'''
"""
def _rows(paths):
@@ -97,15 +97,19 @@ def _rows(paths):
@pytest.fixture
def hook(tmp_path):
bindir = tmp_path / "bin"; bindir.mkdir()
shim = bindir / "curl"; shim.write_text(CURL_SHIM); shim.chmod(0o755)
state = tmp_path / "state"; state.mkdir()
bindir = tmp_path / "bin"
bindir.mkdir()
shim = bindir / "curl"
shim.write_text(CURL_SHIM)
shim.chmod(0o755)
state = tmp_path / "state"
state.mkdir()
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["STUB_SHA"] = SHA
env["ETV_GITEA_TOKEN"] = "stub"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env.pop("ETV_GITEA_BASICAUTH", None)
@@ -114,10 +118,10 @@ def hook(tmp_path):
(state / "pages.json").write_text(json.dumps(list(pages)))
def run(self):
payload = {"tool_input": {"method": "merge", "owner": "timothy",
"repo": "ersatztv", "pull_number": 42}}
return subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
env=env, capture_output=True, text=True)
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
return subprocess.run(
["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True
)
def exempted(self):
"""Exempt == passthrough == exit 0 with no decision JSON."""
@@ -140,16 +144,14 @@ def test_code_pr_is_not_exempt(hook):
def test_protected_path_on_a_LATER_page_is_still_seen(hook):
"""The #619 shape: 50 docs files on page 1, code hiding on page 2."""
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
_rows(["scripts/decisions_lib.py"]))
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["scripts/decisions_lib.py"]))
assert hook.exempted() is False
def test_full_first_page_alone_does_not_end_enumeration(hook):
"""A full 50-row page must trigger a second fetch, not terminate the loop."""
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
_rows(["docs/tail.md"]))
assert hook.exempted() is True # genuinely all docs, but only provable by reading page 2
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["docs/tail.md"]))
assert hook.exempted() is True # genuinely all docs, but only provable by reading page 2
def test_rename_of_code_into_docs_is_not_exempt(hook):
@@ -161,15 +163,15 @@ def test_rename_of_code_into_docs_is_not_exempt(hook):
exempt. That is the hook's contract and differs from `review-verdict.yml`'s stricter
PROTECTED list, which must never auto-post a green status for those paths.
"""
hook.set_pages([{"filename": "docs/innocuous-note.md", "status": "renamed",
"previous_filename": "ErsatzTV/Program.cs"}])
hook.set_pages(
[{"filename": "docs/innocuous-note.md", "status": "renamed", "previous_filename": "ErsatzTV/Program.cs"}]
)
assert hook.exempted() is False
def test_rename_between_two_exempt_paths_stays_exempt(hook):
"""Guards the above: reading previous_filename must not over-trigger on legitimate moves."""
hook.set_pages([{"filename": "docs/b.md", "status": "renamed",
"previous_filename": "docs/a.md"}])
hook.set_pages([{"filename": "docs/b.md", "status": "renamed", "previous_filename": "docs/a.md"}])
assert hook.exempted() is True
@@ -232,14 +234,15 @@ def test_malformed_rename_row_withholds_the_exemption(hook):
Real Gitea always populates it (verified by constructing a rename), so this is the
malformed-2xx class the guard claims to fail closed on; the claim should match the behaviour.
"""
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
[{"filename": "docs/moved.md", "status": "renamed"}])
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), [{"filename": "docs/moved.md", "status": "renamed"}])
assert hook.exempted() is False
def test_rename_row_with_empty_previous_filename_withholds_the_exemption(hook):
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
[{"filename": "docs/moved.md", "status": "renamed", "previous_filename": ""}])
hook.set_pages(
_rows([f"docs/f{i}.md" for i in range(50)]),
[{"filename": "docs/moved.md", "status": "renamed", "previous_filename": ""}],
)
assert hook.exempted() is False
@@ -254,9 +257,13 @@ def test_ordinary_row_without_previous_filename_is_still_valid(hook):
`test_a_rename_disguised_by_an_unknown_status_is_rejected[None]`. The statusless row this test
used to carry was incidental to what it is actually pinning.
"""
hook.set_pages([{"filename": "docs/a.md", "status": "modified"},
{"filename": "docs/b.md", "status": "added"},
{"filename": "docs/c.md", "status": "changed"}])
hook.set_pages(
[
{"filename": "docs/a.md", "status": "modified"},
{"filename": "docs/b.md", "status": "added"},
{"filename": "docs/c.md", "status": "changed"},
]
)
assert hook.exempted() is True
@@ -312,16 +319,20 @@ sys.exit(127)
@pytest.fixture
def hook_jq16(tmp_path):
"""Same harness as `hook`, plus a jq shim emulating jq 1.6's empty-input exit status."""
bindir = tmp_path / "bin"; bindir.mkdir()
(bindir / "curl").write_text(CURL_SHIM); (bindir / "curl").chmod(0o755)
(bindir / "jq").write_text(_JQ16_SHIM); (bindir / "jq").chmod(0o755)
state = tmp_path / "state"; state.mkdir()
bindir = tmp_path / "bin"
bindir.mkdir()
(bindir / "curl").write_text(CURL_SHIM)
(bindir / "curl").chmod(0o755)
(bindir / "jq").write_text(_JQ16_SHIM)
(bindir / "jq").chmod(0o755)
state = tmp_path / "state"
state.mkdir()
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["STUB_SHA"] = SHA
env["ETV_GITEA_TOKEN"] = "stub"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env.pop("ETV_GITEA_BASICAUTH", None)
@@ -330,10 +341,8 @@ def hook_jq16(tmp_path):
(state / "pages.json").write_text(json.dumps(list(pages)))
def exempted(self):
payload = {"tool_input": {"method": "merge", "owner": "timothy",
"repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
env=env, capture_output=True, text=True)
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
assert r.returncode == 0, r.stderr
return r.stdout.strip() == ""
@@ -394,8 +403,9 @@ def test_newline_in_previous_filename_is_also_rejected(hook):
rejects on its own merits, so the test passed with the newline guard entirely removed it
asserted the outcome without ever exercising the mechanism. That is the same
filter-hides-the-defect trap the guard itself is about."""
hook.set_pages([{"filename": "docs/ok.md", "previous_filename": "safe.md\ndocs/Program.cs",
"status": "renamed"}], [])
hook.set_pages(
[{"filename": "docs/ok.md", "previous_filename": "safe.md\ndocs/Program.cs", "status": "renamed"}], []
)
assert hook.exempted() is False
@@ -406,8 +416,7 @@ def test_previous_filename_is_validated_on_NON_renamed_rows_too(hook, status):
`chunk` emits `(.previous_filename // empty)` for EVERY row regardless of `.status`, but the
field was validated only when `.status == "renamed"`. A row marked `modified` (or Gitea's
distinct `copied`) carrying a newline in `previous_filename` was reproducibly exempted."""
hook.set_pages([{"filename": "docs/ok.md", "status": status,
"previous_filename": "safe.md\ndocs/Program.cs"}], [])
hook.set_pages([{"filename": "docs/ok.md", "status": status, "previous_filename": "safe.md\ndocs/Program.cs"}], [])
assert hook.exempted() is False
@@ -421,21 +430,18 @@ def test_dotdot_path_component_is_rejected(hook):
def test_legitimate_rename_within_docs_still_exempts(hook):
"""Positive control: the tightened row schema must not break a real docs-only rename."""
hook.set_pages([{"filename": "docs/b.md", "status": "renamed",
"previous_filename": "docs/a.md"}], [])
hook.set_pages([{"filename": "docs/b.md", "status": "renamed", "previous_filename": "docs/a.md"}], [])
assert hook.exempted() is True
def test_short_NONTERMINAL_page_does_not_end_the_enumeration(hook):
""""Fewer rows than we asked for" must not be read as "last page".
""" "Fewer rows than we asked for" must not be read as "last page".
Gitea caps `limit` at the server-wide MAX_RESPONSE_ITEMS (default 50, configurable) and may
return fewer rows than requested. A 30-row docs page followed by a page of code would otherwise
complete the enumeration over a PARTIAL list the same fail-open, reached with no transport
error at all. Only a validated EMPTY page may terminate it."""
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]),
_rows(["ErsatzTV/Program.cs"]),
[])
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]), _rows(["ErsatzTV/Program.cs"]), [])
assert hook.exempted() is False
@@ -473,9 +479,14 @@ def test_gitea_real_status_values_are_accepted(hook):
"""Positive control for the closed set. The real Gitea 1.25.4 value for an edit is `changed`,
NOT `modified` a closed allow-list built from the wrong vocabulary would reject every real
docs-only PR, which is a far worse failure than the hole it closes."""
hook.set_pages([{"filename": "docs/a.md", "status": "changed"},
{"filename": "docs/b.md", "status": "added"},
{"filename": "docs/c.md", "status": "deleted"}], [])
hook.set_pages(
[
{"filename": "docs/a.md", "status": "changed"},
{"filename": "docs/b.md", "status": "added"},
{"filename": "docs/c.md", "status": "deleted"},
],
[],
)
assert hook.exempted() is True
@@ -483,6 +494,7 @@ def test_gitea_real_status_values_are_accepted(hook):
# The round-3 `..` finding was an anchor subversion, and mutating the anchors showed no test
# covered them: dropping `^` from the docs/ alternative, or `$` from `.md`, both survived.
def test_docs_must_be_a_PREFIX_not_a_substring(hook):
"""Dropping `^` would exempt `ErsatzTV/docs/Program.cs`."""
hook.set_pages([{"filename": "ErsatzTV/docs/Program.cs", "status": "changed"}], [])
@@ -517,7 +529,6 @@ def test_object_valued_status_is_also_rejected(hook):
assert hook.exempted() is False
# --- The `grep -q` / pipefail inversion, on the ADVISORY side (ersatztv#698) --------------------
#
# Round-2 cross-family review noted the enforced gate gained large-input regression tests while the
@@ -530,6 +541,7 @@ def test_object_valued_status_is_also_rejected(hook):
# negated docs-only test. ~171KB is needed to cross the threshold; every other test in this file uses a
# handful of short paths, which is exactly why the class was invisible here.
def _many_docs(n=1900):
return [f"docs/{'d' * 40}-{i:040d}.md" for i in range(n)]
@@ -539,12 +551,12 @@ def test_a_LARGE_pr_containing_a_code_file_is_NOT_exempt(hook):
bulk of ~171KB still to write."""
hook.set_pages(_rows(["A.cs", *_many_docs()]))
assert hook.exempted() is False, (
"a large PR containing A.cs was granted the docs-only exemption — the predicate inverted")
"a large PR containing A.cs was granted the docs-only exemption — the predicate inverted"
)
def test_positive_control_a_LARGE_genuinely_docs_only_pr_IS_still_exempt(hook):
"""Guards the opposite failure: if large lists merely errored, the test above would pass while the
hook prompted on every big docs PR. Without this, 'fixed' and 'broken' are indistinguishable."""
hook.set_pages(_rows(_many_docs()))
assert hook.exempted() is True, (
"a large but genuinely docs-only PR lost its exemption")
assert hook.exempted() is True, "a large but genuinely docs-only PR lost its exemption"
@@ -45,7 +45,7 @@ SHORT = SHA[:7]
# The PR is deliberately NOT docs-only: the docs-only exemption short-circuits the whole gate before
# the scheduled-merge branch is reached, so a docs PR would pass these tests without ever running
# the code under test.
CURL_SHIM = r'''#!/usr/bin/env python3
CURL_SHIM = r"""#!/usr/bin/env python3
import json, os, sys, pathlib, urllib.parse
state = pathlib.Path(os.environ["STUB_DIR"])
@@ -242,14 +242,18 @@ if "/pulls/" in url:
sys.exit(0)
print("{}")
'''
"""
@pytest.fixture
def hook(tmp_path):
bindir = tmp_path / "bin"; bindir.mkdir()
curl = bindir / "curl"; curl.write_text(CURL_SHIM); curl.chmod(0o755)
state = tmp_path / "state"; state.mkdir()
bindir = tmp_path / "bin"
bindir.mkdir()
curl = bindir / "curl"
curl.write_text(CURL_SHIM)
curl.chmod(0o755)
state = tmp_path / "state"
state.mkdir()
(state / "bp").write_text("GUARDED")
env = dict(os.environ)
@@ -257,7 +261,7 @@ def hook(tmp_path):
env["STUB_DIR"] = str(state)
env["STUB_SHA"] = SHA
env["STUB_SHORT"] = SHORT
env["ETV_GITEA_TOKEN"] = "stub"
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env["CLAUDE_PROJECT_DIR"] = str(REPO_ROOT)
env.pop("ETV_GITEA_BASICAUTH", None)
@@ -283,11 +287,16 @@ def hook(tmp_path):
return f.read_text().splitlines() if f.exists() else []
def decision(self, scheduled=True):
payload = {"tool_input": {"method": "merge", "owner": "timothy",
"repo": "ersatztv", "pull_number": 42,
"merge_when_checks_succeed": scheduled}}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
env=env, capture_output=True, text=True)
payload = {
"tool_input": {
"method": "merge",
"owner": "timothy",
"repo": "ersatztv",
"pull_number": 42,
"merge_when_checks_succeed": scheduled,
}
}
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
assert r.returncode == 0, r.stderr
if not r.stdout.strip():
return None
@@ -307,9 +316,11 @@ def test_a_base_without_the_required_check_denies_a_SCHEDULED_merge(hook):
reason = hook.reason()
assert "deny" in reason, (
"a scheduled auto-merge was armed with no 'review-verdict/h10' required check on the base — "
"that is ersatztv#622's hole reopened")
"that is ersatztv#622's hole reopened"
)
assert "review-verdict/h10" in reason, (
"the deny must name the missing context; a reader cannot act on 'branch protection is wrong'")
"the deny must name the missing context; a reader cannot act on 'branch protection is wrong'"
)
def test_status_checks_disabled_wholesale_also_denies(hook):
@@ -318,8 +329,7 @@ def test_status_checks_disabled_wholesale_also_denies(hook):
check-the-label-not-the-capability shape (#697/#698) this repo has paid for twice."""
hook.set_branch_protection("STATUS-CHECK-OFF")
reason = hook.reason()
assert "deny" in reason, (
"status checks were disabled entirely and the listed context was read as protection anyway")
assert "deny" in reason, "status checks were disabled entirely and the listed context was read as protection anyway"
def test_positive_control_a_guarded_base_REACHES_the_check_and_still_auto_grants(hook):
@@ -334,7 +344,8 @@ def test_positive_control_a_guarded_base_REACHES_the_check_and_still_auto_grants
decision = hook.decision()
assert hook.branch_protection_urls(), (
"the guarded case never reached the branch-protection endpoint, so the other tests are not "
"exercising the code they claim to")
"exercising the code they claim to"
)
assert decision is not None, "the hook passed through instead of auto-granting"
verdict = decision["hookSpecificOutput"]["permissionDecision"]
assert verdict == "allow", f"a fully-satisfied gate did not auto-grant (got {verdict!r})"
@@ -354,9 +365,11 @@ def test_a_context_name_that_merely_CONTAINS_the_required_one_does_not_satisfy_i
hook.set_branch_protection(shape)
reason = hook.reason()
assert "allow" not in reason or "deny" in reason or "ask" in reason, (
f"payload shape {shape} auto-granted a scheduled merge")
f"payload shape {shape} auto-granted a scheduled merge"
)
assert "satisfied" not in reason, (
f"a context name that merely contains 'review-verdict/h10' ({shape}) was accepted as it")
f"a context name that merely contains 'review-verdict/h10' ({shape}) was accepted as it"
)
@pytest.mark.parametrize("shape", ["EMPTY-CONTEXTS", "NULL-CONTEXTS"])
@@ -366,8 +379,7 @@ def test_an_empty_or_null_contexts_list_DENIES_rather_than_asking(hook, shape):
unknown-shape ask alongside genuinely unreadable payloads."""
hook.set_branch_protection(shape)
reason = hook.reason()
assert "deny" in reason, (
f"{shape} was treated as unreadable rather than as a confirmed absent required check")
assert "deny" in reason, f"{shape} was treated as unreadable rather than as a confirmed absent required check"
def test_a_FALSE_contexts_value_asks_rather_than_denying(hook):
@@ -400,7 +412,8 @@ def test_a_malformed_contexts_MEMBER_asks_rather_than_denying_with_the_wrong_rea
reason = hook.reason()
assert "ask" in reason, "a malformed contexts member produced a decision instead of a question"
assert "NOT a required status check" not in reason, (
"an unreadable payload was reported as a confirmed missing required check")
"an unreadable payload was reported as a confirmed missing required check"
)
def test_a_base_with_NO_branch_protection_at_all_denies_rather_than_asking(hook):
@@ -417,11 +430,10 @@ def test_a_base_with_NO_branch_protection_at_all_denies_rather_than_asking(hook)
"""
hook.set_branch_protection("EMPTY-LIST")
reason = hook.reason()
assert "deny" in reason, (
"a base with no branch protection at all did not deny a scheduled auto-merge")
assert "deny" in reason, "a base with no branch protection at all did not deny a scheduled auto-merge"
assert "none matches" in reason, (
"the deny must distinguish 'the list was read and nothing governs this base' from "
"'could not read'")
"the deny must distinguish 'the list was read and nothing governs this base' from 'could not read'"
)
def test_a_403_asks_because_it_says_only_that_we_could_not_look(hook):
@@ -431,13 +443,12 @@ def test_a_403_asks_because_it_says_only_that_we_could_not_look(hook):
hook.set_branch_protection("FORBIDDEN")
reason = hook.reason()
assert "ask" in reason, "a 403 was treated as evidence about the protection"
assert "none matches" not in reason, (
"a 403 was reported as a confirmed absence of branch protection")
assert "none matches" not in reason, "a 403 was reported as a confirmed absence of branch protection"
@pytest.mark.parametrize("failure", ["TRANSPORT-ERROR", "GARBAGE", "EMPTY", "LIST-404"])
def test_an_UNREADABLE_branch_protection_asks_rather_than_denying_or_passing(hook, failure):
""""Could not check" is a third outcome, not a synonym for either neighbour.
""" "Could not check" is a third outcome, not a synonym for either neighbour.
Denying would strand every scheduled merge on a Gitea hiccup or on credentials without the
repo-admin scope this endpoint needs. Passing would be worse: it would restore the exact
@@ -446,10 +457,8 @@ def test_an_UNREADABLE_branch_protection_asks_rather_than_denying_or_passing(hoo
"""
hook.set_branch_protection(failure)
reason = hook.reason()
assert "ask" in reason, (
f"an unreadable branch-protection response ({failure}) did not fall through to a human")
assert "branch-protection rules" in reason, (
"the ask must name what could not be checked")
assert "ask" in reason, f"an unreadable branch-protection response ({failure}) did not fall through to a human"
assert "branch-protection rules" in reason, "the ask must name what could not be checked"
def test_an_IMMEDIATE_merge_is_not_subjected_to_this_check(hook):
@@ -465,7 +474,8 @@ def test_an_IMMEDIATE_merge_is_not_subjected_to_this_check(hook):
hook.set_branch_protection("MISSING-CONTEXT")
reason = hook.reason(scheduled=False)
assert "NOT a required status check" not in reason, (
"the required-check deny fired on an immediate merge, which has no post-scheduling window")
"the required-check deny fired on an immediate merge, which has no post-scheduling window"
)
def test_a_NON_BOOLEAN_enable_status_check_asks(hook):
@@ -504,8 +514,8 @@ def test_a_SCALAR_status_row_asks_instead_of_killing_the_hook(hook):
hook.set_branch_protection("SCALAR-STATUS-ROW")
decision = hook.decision()
assert decision is not None, (
"the hook emitted no decision at all for a malformed statuses payload — it neither granted, "
"denied nor asked")
"the hook emitted no decision at all for a malformed statuses payload — it neither granted, denied nor asked"
)
assert decision["hookSpecificOutput"]["permissionDecision"] == "ask"
@@ -521,8 +531,7 @@ def test_a_PERSISTENT_retarget_denies_on_the_IMMEDIATE_path_too(hook):
hook.drop_recorded_base()
hook.set_retarget("scratch")
reason = hook.reason(scheduled=False)
assert "deny" in reason, (
"an immediate merge was auto-granted after the PR was retargeted mid-evaluation")
assert "deny" in reason, "an immediate merge was auto-granted after the PR was retargeted mid-evaluation"
assert "scratch" in reason and "main" in reason
@@ -540,10 +549,10 @@ def test_a_PERSISTENT_retarget_after_the_first_read_denies(hook):
hook.drop_recorded_base()
hook.set_retarget("scratch")
reason = hook.reason()
assert "deny" in reason, (
"the PR was retargeted mid-evaluation and the gate still granted on the original base")
assert "deny" in reason, "the PR was retargeted mid-evaluation and the gate still granted on the original base"
assert "scratch" in reason and "main" in reason, (
"the deny must name both branches; a reader cannot act on 'the base changed'")
"the deny must name both branches; a reader cannot act on 'the base changed'"
)
def test_a_NON_STRING_status_reaches_the_scheduled_validator(hook):
@@ -573,13 +582,12 @@ def test_a_base_that_a_GLOB_rule_could_govern_ASKS(hook):
"""
hook.set_branch_protection("GLOB-RULE")
reason = hook.reason()
assert "ask" in reason, (
"a base a glob rule could govern was decided rather than referred to a human")
assert "ask" in reason, "a base a glob rule could govern was decided rather than referred to a human"
assert "could govern it" in reason, (
"the ask came from the generic could-not-read arm, not the undecidable-glob arm — those "
"are different outcomes and a crashed classifier must not pass as a correct classification")
assert "none matches" not in reason, (
"a base a glob rule could govern was reported as having no protection at all")
"are different outcomes and a crashed classifier must not pass as a correct classification"
)
assert "none matches" not in reason, "a base a glob rule could govern was reported as having no protection at all"
def test_an_unreadable_rule_LIST_asks_rather_than_denying(hook):
@@ -606,8 +614,7 @@ def test_the_protection_lookup_puts_NO_ref_in_the_url(hook):
urls = hook.branch_protection_urls()
assert urls, "the branch-protection endpoint was never requested"
for u in urls:
assert u.rstrip("/").endswith("/branch_protections"), (
f"a ref reached the branch-protection URL: {u}")
assert u.rstrip("/").endswith("/branch_protections"), f"a ref reached the branch-protection URL: {u}"
def test_a_rule_name_with_REGEX_METACHARACTERS_does_not_match_a_different_base(hook):
@@ -624,11 +631,11 @@ def test_a_rule_name_with_REGEX_METACHARACTERS_does_not_match_a_different_base(h
"""
hook.set_branch_protection("REGEX-META-RULE")
reason = hook.reason()
assert "deny" in reason, (
"a rule named 'mai.' was regex-matched against base 'main' and read as protection")
assert "deny" in reason, "a rule named 'mai.' was regex-matched against base 'main' and read as protection"
assert "none matches" in reason, (
"'mai.' contains no GLOB metacharacter, so it is decidable: it simply does not govern "
"'main', and the base is genuinely unprotected")
"'main', and the base is genuinely unprotected"
)
def test_a_GLOB_rule_whose_literal_part_has_a_metacharacter_still_MATCHES(hook):
@@ -645,17 +652,19 @@ def test_a_GLOB_rule_whose_literal_part_has_a_metacharacter_still_MATCHES(hook):
rather than reporting the base as unprotected.
"""
hook.set_base("release/26.4")
hook.drop_recorded_base() # keep the #632 comparison out of this test's way
hook.drop_recorded_base() # keep the #632 comparison out of this test's way
hook.set_branch_protection("GLOB-WITH-DOT-RULE")
reason = hook.reason()
assert "ask" in reason, (
"the glob rule 'release/26.*' could govern base 'release/26.4', which is undecidable here "
"and must ask")
"the glob rule 'release/26.*' could govern base 'release/26.4', which is undecidable here and must ask"
)
assert "could govern it" in reason, (
"the ask must come from the undecidable-glob arm, not from the classifier failing")
"the ask must come from the undecidable-glob arm, not from the classifier failing"
)
assert "none matches" not in reason, (
"a base a glob rule could govern was reported as entirely unprotected — the over-escaping "
"failure this test exists to catch")
"failure this test exists to catch"
)
def test_a_rule_name_containing_a_CHAR_CLASS_bracket_does_not_crash_the_matcher(hook):
@@ -667,12 +676,13 @@ def test_a_rule_name_containing_a_CHAR_CLASS_bracket_does_not_crash_the_matcher(
assert reason, "the hook emitted no decision at all"
assert "ask" not in reason, (
"the exact-name rule was decidable and must have been honoured; an ask here means the "
"classifier failed rather than classified")
"classifier failed rather than classified"
)
assert "none matches" not in reason, (
"one rule with a bracket in its name discarded the whole list, including the exact-name "
"rule that actually protects this base")
assert "deny" not in reason, (
"an exact-name rule requiring review-verdict/h10 was present and was not honoured")
"rule that actually protects this base"
)
assert "deny" not in reason, "an exact-name rule requiring review-verdict/h10 was present and was not honoured"
def test_a_plain_rule_name_is_matched_CASE_INSENSITIVELY(hook):
@@ -683,9 +693,11 @@ def test_a_plain_rule_name_is_matched_CASE_INSENSITIVELY(hook):
reason = hook.reason()
assert "ask" not in reason, (
"the case-folded exact rule was decidable and must have been honoured; an ask means the "
"classifier failed rather than classified")
"classifier failed rather than classified"
)
assert "none matches" not in reason, (
"a rule named 'MAIN' governs base 'main' in Gitea but was missed by a case-sensitive compare")
"a rule named 'MAIN' governs base 'main' in Gitea but was missed by a case-sensitive compare"
)
assert "deny" not in reason
@@ -707,12 +719,13 @@ def test_a_BACKSLASH_ESCAPED_metacharacter_in_a_rule_name_is_undecidable_not_abs
hook.set_branch_protection("ESCAPED-META-RULE")
reason = hook.reason()
assert "none matches" not in reason, (
"a rule whose escaped brace governs this base was reported as unable to govern it")
assert "ask" in reason, (
"an escaped-metacharacter rule is undecidable here and must ask")
"a rule whose escaped brace governs this base was reported as unable to govern it"
)
assert "ask" in reason, "an escaped-metacharacter rule is undecidable here and must ask"
assert "could govern it" in reason, (
"the ask must come from the undecidable-glob arm, not from the classifier failing — this "
"test hits the same arm as its two siblings and needs the same pin")
"test hits the same arm as its two siblings and needs the same pin"
)
def test_the_precedence_check_runs_even_when_an_exactly_named_rule_EXISTS(hook):
@@ -732,13 +745,13 @@ def test_the_precedence_check_runs_even_when_an_exactly_named_rule_EXISTS(hook):
"""
hook.set_branch_protection("EXACT-PLUS-GLOB-RULE")
reason = hook.reason()
assert "ask" in reason, (
"an exactly-named rule was trusted without asking which rule Gitea would actually apply")
assert "ask" in reason, "an exactly-named rule was trusted without asking which rule Gitea would actually apply"
assert "could govern it" in reason
# And the by-name endpoint must not be consulted at all — its existence is what split the paths.
for u in hook.branch_protection_urls():
assert u.rstrip("/").endswith("/branch_protections"), (
f"the by-name lookup is back, and with it the unguarded path: {u}")
f"the by-name lookup is back, and with it the unguarded path: {u}"
)
def test_a_GLOB_rule_that_could_outrank_an_exact_one_wins_and_ASKS(hook):
@@ -758,9 +771,11 @@ def test_a_GLOB_rule_that_could_outrank_an_exact_one_wins_and_ASKS(hook):
reason = hook.reason()
assert "ask" in reason, (
"an exact rule was trusted while a glob rule could outrank it — the gate granted on a base "
"whose enforced rule it never identified")
"whose enforced rule it never identified"
)
assert "could govern it" in reason, (
"the ask must come from the undecidable-glob arm, not from the classifier failing")
"the ask must come from the undecidable-glob arm, not from the classifier failing"
)
def test_two_rules_differing_only_in_CASE_are_undecidable(hook):
@@ -769,8 +784,7 @@ def test_two_rules_differing_only_in_CASE_are_undecidable(hook):
whose enforced rule was never identified the same defect as the arm order, one level down."""
hook.set_branch_protection("TWO-CASE-VARIANT-RULES")
reason = hook.reason()
assert "ask" in reason, (
"two fold-equal rules disagree about review-verdict/h10 and one was picked by list order")
assert "ask" in reason, "two fold-equal rules disagree about review-verdict/h10 and one was picked by list order"
assert "could govern it" in reason
@@ -784,7 +798,8 @@ def test_a_NON_ASCII_rule_or_base_is_undecidable_rather_than_fold_compared(hook)
hook.set_branch_protection("NONASCII-RULE")
reason = hook.reason()
assert "none matches" not in reason, (
"a rule that folds equal to this base under EqualFold was reported as unable to govern it")
"a rule that folds equal to this base under EqualFold was reported as unable to govern it"
)
assert "ask" in reason
@@ -803,8 +818,7 @@ def test_an_HTTP_404_on_the_LIST_read_asks_and_does_not_claim_the_list_was_read(
hook.set_branch_protection("LIST-404")
reason = hook.reason()
assert "ask" in reason, "an unreadable repo was treated as evidence about the base"
assert "none matches" not in reason, (
"a 404 read claimed the full rule list had been read and matched nothing")
assert "none matches" not in reason, "a 404 read claimed the full rule list had been read and matched nothing"
@pytest.mark.parametrize("shape", ["UNPARSEABLE-RULES", "GARBAGE", "EMPTY"])
@@ -825,6 +839,5 @@ def test_a_200_the_classifier_cannot_PARSE_asks_without_blaming_the_transport(ho
hook.set_branch_protection(shape)
reason = hook.reason()
assert "ask" in reason, "an unparseable rule list produced a decision instead of a question"
assert "could not parse" in reason, (
"the ask blamed the transport for a 200 the classifier simply could not read")
assert "could not parse" in reason, "the ask blamed the transport for a 200 the classifier simply could not read"
assert "unreachable" not in reason
+13 -10
View File
@@ -104,7 +104,7 @@ def gitea(tmp_path):
env = dict(os.environ)
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
env["STUB_DIR"] = str(state)
env["ETV_GITEA_TOKEN"] = "stub-token"
env["ETV_GITEA_TOKEN"] = "stub-token" # noqa: S105 - deliberately fake; the real credential comes from the environment
env["ETV_GITEA_URL"] = "http://gitea.example"
env["ETV_GITEA_REPO"] = "timothy/ersatztv"
env.pop("ETV_GITEA_BASICAUTH", None)
@@ -127,7 +127,9 @@ def gitea(tmp_path):
def run(self, *args):
return subprocess.run(
["bash", str(SCRIPT), *args],
env=env, capture_output=True, text=True,
env=env,
capture_output=True,
text=True,
)
def posts(self):
@@ -229,6 +231,7 @@ def test_unreachable_pr_is_an_error_not_a_silent_success(gitea):
# --- Cross-checks against the hook's own condition-(c) parser -----------------------------------
def _classify(body: str, head: str) -> str:
"""Run the REAL H10 classifier over a comment body — no Python mirror of the grammar.
@@ -239,9 +242,7 @@ def _classify(body: str, head: str) -> str:
passing here while the shell drifted.
"""
payload = json.dumps([{"body": body}])
p = subprocess.run(
["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True
)
p = subprocess.run(["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True)
assert p.returncode == 0, f"classifier errored: {p.stderr}"
return p.stdout.strip()
@@ -280,6 +281,7 @@ def test_note_cannot_forge_a_second_verdict_line(gitea):
# mirror case: the head sha and the status both hold still while the effective DIFF changes, so the
# verdict keeps reading green for a review nobody performed against that base.
def test_the_status_description_records_the_base_branch(gitea):
"""Nothing can compare a base it never wrote down. This field is what the hook reads back."""
assert gitea.run("42", "MERGEABLE").returncode == 0
@@ -333,7 +335,8 @@ def test_a_failed_HEAD_RECHECK_writes_no_status(gitea):
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0
assert gitea.statuses() == [], (
"a status was written even though the head/base re-read failed — nothing was confirmed")
"a status was written even though the head/base re-read failed — nothing was confirmed"
)
# --- write-side polarity, the half the #774 rescue initially missed ------------------------------
@@ -404,8 +407,8 @@ def test_a_reread_that_LOSES_a_field_refuses_instead_of_posting(head_seq, base_s
gitea.set_base_sequence(*base_seq)
result = gitea.run("42", "MERGEABLE")
assert result.returncode != 0, (
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed "
"nothing about it")
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed nothing about it"
)
assert not gitea.statuses(), (
f"a status was written despite the re-read carrying no {field} — this is the fail-open "
"the -n conjunct created")
f"a status was written despite the re-read carrying no {field} — this is the fail-open the -n conjunct created"
)
File diff suppressed because it is too large Load Diff
+46 -27
View File
@@ -35,14 +35,18 @@ PROVE_FIX = Path(os.environ.get("PROVE_FIX_PATH") or (REPO_ROOT / "scripts" / "p
def _git(repo: Path, *args: str) -> str:
return subprocess.run(
["git", "-C", str(repo), *args],
check=True, capture_output=True, text=True,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def _run(repo: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
["bash", str(PROVE_FIX), "--repo", str(repo), *args],
capture_output=True, text=True, cwd=str(repo),
capture_output=True,
text=True,
cwd=str(repo),
)
@@ -61,9 +65,7 @@ def fixrepo(tmp_path: Path) -> Path:
# --- commit 1: the bug, plus a test that cannot see it
(repo / "calc.py").write_text("def add(a, b):\n return a - b # bug\n")
(repo / "scripts" / "tests" / "test_unrelated.py").write_text(
"def test_unrelated():\n assert 1 + 1 == 2\n"
)
(repo / "scripts" / "tests" / "test_unrelated.py").write_text("def test_unrelated():\n assert 1 + 1 == 2\n")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "initial: buggy add, unrelated test")
@@ -73,8 +75,7 @@ def fixrepo(tmp_path: Path) -> Path:
"from calc import add\n\n\ndef test_add():\n assert add(2, 3) == 5\n"
)
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m",
"fix: add() returned a difference\n\nProves: scripts/tests/test_add.py")
_git(repo, "commit", "-q", "-m", "fix: add() returned a difference\n\nProves: scripts/tests/test_add.py")
return repo
@@ -143,7 +144,8 @@ def test_added_code_file_is_removed_not_checked_out(tmp_path: Path) -> None:
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "scripts" / "tests" / "keep.py").write_text("# placeholder\n")
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "initial")
# the fix ADDS helper.py (it has no parent version) and a test that needs it
(repo / "helper.py").write_text("def shout(s):\n return s.upper()\n")
@@ -165,7 +167,8 @@ def test_root_commit_REFUSES(tmp_path: Path) -> None:
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "a.py").write_text("x = 1\n")
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "root")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "root")
r = _run(repo, "HEAD", "scripts/tests")
assert r.returncode == 5
assert "root commit" in r.stderr
@@ -185,9 +188,10 @@ def test_runs_under_the_system_bash(fixrepo: Path) -> None:
# developer Mac. The 3.2 hazard (mapfile yielding an empty array) is what motivated it.
ver = subprocess.run([str(system_bash), "--version"], capture_output=True, text=True).stdout
r = subprocess.run(
[str(system_bash), str(PROVE_FIX), "--repo", str(fixrepo), "HEAD",
"scripts/tests/test_add.py"],
capture_output=True, text=True, cwd=str(fixrepo),
[str(system_bash), str(PROVE_FIX), "--repo", str(fixrepo), "HEAD", "scripts/tests/test_add.py"],
capture_output=True,
text=True,
cwd=str(fixrepo),
)
assert r.returncode == 0, (
f"prove-fix.sh must work under the system bash ({ver.splitlines()[0] if ver else '?'}); "
@@ -204,8 +208,7 @@ def test_control_failure_REFUSES(fixrepo: Path) -> None:
)
(fixrepo / "calc.py").write_text("def add(a, b):\n return a + b # unchanged\n")
_git(fixrepo, "add", "-A")
_git(fixrepo, "commit", "-q", "-m",
"fix: with an already-failing test\n\nProves: scripts/tests/test_broken.py")
_git(fixrepo, "commit", "-q", "-m", "fix: with an already-failing test\n\nProves: scripts/tests/test_broken.py")
r = _run(fixrepo, "HEAD")
assert r.returncode == 6, f"expected 6 (control failed), got {r.returncode}\n{r.stdout}\n{r.stderr}"
assert "control FAILED" in r.stderr
@@ -248,9 +251,19 @@ def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test(
env = {**os.environ, "PROVE_FIX_PATH": str(mutant), "PROVE_FIX_MUTATION_RUN": "1"}
nested = subprocess.run(
["python3", "-m", "pytest", f"{Path(__file__).name}::test_unrelated_test_is_UNPROVEN",
"-q", "-p", "no:cacheprovider"],
cwd=str(Path(__file__).parent), env=env, capture_output=True, text=True,
[
"python3",
"-m",
"pytest",
f"{Path(__file__).name}::test_unrelated_test_is_UNPROVEN",
"-q",
"-p",
"no:cacheprovider",
],
cwd=str(Path(__file__).parent),
env=env,
capture_output=True,
text=True,
)
out = nested.stdout + nested.stderr
assert nested.returncode == 1, (
@@ -282,17 +295,20 @@ def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
_git(repo, "config", "user.name", "T")
(repo / "mod.py").write_text("VALUE = 1\n")
(repo / "scripts" / "tests" / "test_slow.py").write_text(
"import time\nfrom mod import VALUE\n\n\n"
"def test_slow():\n time.sleep(20)\n assert VALUE == 2\n"
"import time\nfrom mod import VALUE\n\n\ndef test_slow():\n time.sleep(20)\n assert VALUE == 2\n"
)
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "initial")
(repo / "mod.py").write_text("VALUE = 2\n")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "fix: bump\n\nProves: scripts/tests/test_slow.py")
proc = subprocess.Popen(
["bash", str(PROVE_FIX), "--repo", str(repo), "HEAD"],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd=str(repo),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
cwd=str(repo),
)
time.sleep(4) # inside the control run, which sleeps 20s
assert proc.poll() is None, "the run finished before it could be signalled; test is void"
@@ -305,9 +321,7 @@ def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
# bugs landing on the same observable is not a witnessed fix. Cold review measured that
# pair failing to separate old from new; rc==5 AND the handler's own message do separate
# them.
assert proc.returncode == 5, (
f"a signalled run must exit 5 from on_signal, got {proc.returncode}\n{out}\n{err}"
)
assert proc.returncode == 5, f"a signalled run must exit 5 from on_signal, got {proc.returncode}\n{out}\n{err}"
assert "interrupted by signal" in err, (
f"expected the signal handler's own message, so this test cannot be satisfied by an "
f"unrelated later failure:\n{err}"
@@ -335,7 +349,8 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
_git(repo, "config", "user.email", "t@example.com")
_git(repo, "config", "user.name", "T")
(repo / "scripts" / "tests" / "keep.py").write_text("# placeholder\n")
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "initial")
(repo / "added.py").write_text("def val():\n return 7\n")
(repo / "scripts" / "tests" / "test_added.py").write_text(
"from added import val\n\n\ndef test_val():\n assert val() == 7\n"
@@ -343,7 +358,8 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "feat: add val\n\nProves: scripts/tests/test_added.py")
shim = tmp_path / "bin"; shim.mkdir()
shim = tmp_path / "bin"
shim.mkdir()
counter = tmp_path / "count"
(shim / "git").write_text(
"#!/bin/sh\n"
@@ -358,7 +374,10 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
env = {**os.environ, "PATH": f"{shim}:{os.environ['PATH']}"}
r = subprocess.run(
["bash", str(PROVE_FIX), "--repo", str(repo), "HEAD"],
capture_output=True, text=True, cwd=str(repo), env=env,
capture_output=True,
text=True,
cwd=str(repo),
env=env,
)
assert "PROVEN" not in r.stdout, (
"a phase whose worktree does not exist cannot witness anything; pre-fix this "
+28 -23
View File
@@ -109,7 +109,8 @@ def _tracked_files() -> list[str]:
"""
proc = subprocess.run(
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
capture_output=True, check=True,
capture_output=True,
check=True,
)
return [p for p in proc.stdout.decode().split("\0") if p]
@@ -150,7 +151,8 @@ def _inventory_section(text: str) -> str:
raise AssertionError(
f"{INVENTORY.name} has no {heading!r} heading. Row parsing is bounded by "
"'## The inventory' and '## Limits'; renaming or reordering either one would "
"silently change which rows are checked, so it fails here instead.")
"silently change which rows are checked, so it fails here instead."
)
start = text.index("## The inventory")
try:
end = text.index("## Limits", start)
@@ -161,7 +163,8 @@ def _inventory_section(text: str) -> str:
# either returns an index >= start or raises. A guard that cannot execute proves nothing.
raise AssertionError(
f"{INVENTORY.name}: '## Limits' precedes '## The inventory', so the parsed window "
"would be empty and every completeness assertion would pass vacuously.") from None
"would be empty and every completeness assertion would pass vacuously."
) from None
return text[start:end]
@@ -188,10 +191,11 @@ def test_anti_vacuity_the_derivation_and_the_table_both_found_something():
sites = inventory_sites()
assert len(population) >= 40, (
f"derived only {len(population)} in-scope files — the globs are broken, not the repo "
"(the scope held 59 files on 2026-08-16, and it only grows)")
"(the scope held 59 files on 2026-08-16, and it only grows)"
)
assert len(sites) >= 40, (
f"parsed only {len(sites)} rows out of the inventory — the row regex has drifted from the "
"table format")
f"parsed only {len(sites)} rows out of the inventory — the row regex has drifted from the table format"
)
def test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file():
@@ -208,9 +212,9 @@ def test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file():
phantom = sorted(sites - population)
assert not missing, (
"in scope but absent from docs/remote-state-inventory.md (classify each as "
f"PINNED / CAS / UNSAFE-KNOWN / N/A): {missing}")
assert not phantom, (
f"listed in docs/remote-state-inventory.md but no such in-scope file exists: {phantom}")
f"PINNED / CAS / UNSAFE-KNOWN / N/A): {missing}"
)
assert not phantom, f"listed in docs/remote-state-inventory.md but no such in-scope file exists: {phantom}"
def test_MUTATION_PROOF_a_dropped_row_and_a_phantom_row_are_both_detected():
@@ -228,27 +232,28 @@ def test_MUTATION_PROOF_a_dropped_row_and_a_phantom_row_are_both_detected():
population = derived_population()
victim = sorted(population)[0]
dropped = "\n".join(
line for line in text.splitlines() if not line.startswith(f"| `{victim}`"))
dropped = "\n".join(line for line in text.splitlines() if not line.startswith(f"| `{victim}`"))
assert victim not in inventory_sites(dropped), (
f"the mutation did not actually remove {victim}; the proof below would be vacuous")
f"the mutation did not actually remove {victim}; the proof below would be vacuous"
)
assert population - inventory_sites(dropped), (
"a row was removed from the inventory and the comparison still reported complete coverage")
"a row was removed from the inventory and the comparison still reported complete coverage"
)
# Inserted INSIDE the inventory section, not appended to the file: rows are parsed only between
# "## The inventory" and "## Limits", so appending at the end would test nothing.
phantom = text.replace(
"## Limits",
"| `scripts/does-not-exist.sh` — invented | `PINNED` | n/a |\n\n## Limits", 1)
phantom = text.replace("## Limits", "| `scripts/does-not-exist.sh` — invented | `PINNED` | n/a |\n\n## Limits", 1)
assert inventory_sites(phantom) - population == {"scripts/does-not-exist.sh"}, (
"a row naming a file that does not exist was not reported as phantom")
"a row naming a file that does not exist was not reported as phantom"
)
def test_every_class_cell_comes_from_the_closed_vocabulary():
bad = sorted({cls for _, cls in inventory_rows() if cls not in CLASSES})
assert not bad, (
f"unknown classification(s) {bad}; allowed: {sorted(CLASSES)}. A typo here would silently "
"create a state nobody reviews.")
"create a state nobody reviews."
)
def test_every_unsafe_row_states_why_the_residual_is_accepted():
@@ -268,8 +273,7 @@ def test_every_unsafe_row_states_why_the_residual_is_accepted():
site, note = cells[0], cells[-1]
if len(note) < 120:
thin.append(site[:60])
assert not thin, (
f"UNSAFE-KNOWN row(s) with no stated justification: {thin}")
assert not thin, f"UNSAFE-KNOWN row(s) with no stated justification: {thin}"
def test_the_population_never_includes_a_file_git_does_not_track(monkeypatch):
@@ -296,14 +300,15 @@ def test_the_population_never_includes_a_file_git_does_not_track(monkeypatch):
)
# Patch the module object this test is running inside, whatever name it was imported under.
import sys
mod = sys.modules[__name__]
monkeypatch.setattr(mod, "_tracked_files", lambda: tracked)
assert (REPO_ROOT / victim).is_file(), (
f"{victim} must still exist on disk for this proof to mean anything")
assert (REPO_ROOT / victim).is_file(), f"{victim} must still exist on disk for this proof to mean anything"
assert victim not in derived_population(), (
f"{victim} is on disk and matches the scope, but git no longer tracks it — it must not enter "
"the population, or untracked build output can redden this guard again")
"the population, or untracked build output can redden this guard again"
)
def test_every_derived_member_is_tracked():