feat(780): commit a ruff config and enforce it in CI (#813)
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 28s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m29s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 5m54s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 5m59s
Build ErsatzTV Image / Build & push image (amd64) (push) Failing after 15s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (push) Skipped
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (push) Skipped
Build ErsatzTV Image / Delimiter ban (release path) (push) Successful in 28s
Build ErsatzTV Image / Build & test (.NET) (push) Successful in 8m29s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (push) Successful in 5m54s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (push) Successful in 5m59s
Build ErsatzTV Image / Build & push image (amd64) (push) Failing after 15s
Python lint here was a property of the operator's laptop: the global instructions
say to run ruff, no workflow ran it, and with no committed config ruff fell back
to whichever ~/.config/ruff/ruff.toml the machine happened to have.
- ruff.toml at the root, pinned ruff==0.12.11 in the script-tests job.
- Both lint steps pass an EXPLICIT population from `git ls-files` with
`--no-force-exclude`, never `ruff check .` — an `exclude` empties a
discovery-based run into a GREEN one (top level empties both commands, [lint]
empties check, [format] empties format --check), and `ruff check .` over zero
files exits 0 with only a stderr warning. Guarded by an empty-population arm.
- Tree clean: 74 findings at 706674272, 57 fixed in code, 17 per-site noqa with
reasons inline. S105 deliberately per-site, not a directory blanket. RUF100
selected so a suppression that suppresses nothing is itself a finding.
- pyright stays ungated; reasoning in the record.
Both steps witnessed red on the runner against the shipped bodies: run 2173 job
9176 (ruff check) and run 2170 job 9163 (ruff format --check).
Docs: new record ci.python-lint-ruff-config-committed, ci.script-tests-job
cross-ref, docs/ci-cd.md (also correcting a stale ~190-tests/~10s figure to the
measured 773 tests / ~4.5 min), docs/defect-shapes-773.md §5.2 resolved.
fixes #780
Co-authored-by: Timothy <timothy@noreply.gitea.tblindustries.be>
This commit was merged in pull request #813.
This commit is contained in:
@@ -359,7 +359,7 @@ jobs:
|
||||
[ "$failed" -eq 0 ]
|
||||
|
||||
script-tests:
|
||||
name: Script tests (pytest)
|
||||
name: Script lint and tests (ruff + pytest)
|
||||
runs-on: small
|
||||
if: github.event_name == 'pull_request'
|
||||
steps:
|
||||
@@ -369,6 +369,90 @@ jobs:
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.x'
|
||||
# Preflight, not an install (ersatztv#390 removed run-time `apt-get` from CI on purpose).
|
||||
# Two consumers need `git`: the lint steps below derive their population from `git ls-files`,
|
||||
# and test_post_review_verdict.py / test_merge_consent_exemption.py exec the REAL
|
||||
# post-review-verdict.sh / pretooluse-merge-consent.sh. `curl` those tests shim on PATH; `jq`
|
||||
# and `git` they do NOT. It stays AHEAD of the lint steps, not merely ahead of pytest: without
|
||||
# it, a missing git reaches the lint steps as an empty population, which they report as a
|
||||
# population problem. One actionable line beats a misdirected one, and beats the wall of
|
||||
# unattributable assertion failures the suite produces without git.
|
||||
- name: Preflight external tools
|
||||
run: |
|
||||
if ! command -v git >/dev/null 2>&1; then
|
||||
echo "::error::script-tests needs git on PATH but it is absent. The lint steps derive" \
|
||||
"their population from it and the suite execs real shell scripts that use it." \
|
||||
"Bake it into the runner image rather than apt-get installing here (ersatztv#390)."
|
||||
exit 1
|
||||
fi
|
||||
echo "Preflight OK: $(git --version)"
|
||||
# ersatztv#780. Lint runs EARLY — after the git preflight it depends on, but before the test
|
||||
# dependencies, the jq preflight and the ~4-minute pytest run. A style red therefore arrives in
|
||||
# seconds, and, more importantly, the lint does not sit behind `Preflight jq version`: that is
|
||||
# an `--expect` tripwire, so a runner jq bump would take the lint dark for as long as the jq
|
||||
# contract is broken, under a red that says "jq".
|
||||
#
|
||||
# The version is PINNED: an unpinned ruff makes the verdict a function of whenever the job ran
|
||||
# — the same environment-divergence the committed ruff.toml exists to close. Bumping it is a
|
||||
# deliberate PR (new rules may fire), exactly like the jq pin below. `pytest`/`pyyaml` are
|
||||
# deliberately NOT pinned: a pytest release does not add assertions to your suite, a ruff
|
||||
# release adds rules to your lint.
|
||||
- name: Install ruff
|
||||
run: python3 -m pip install --disable-pip-version-check --quiet 'ruff==0.12.11'
|
||||
# POPULATION. Both steps lint an EXPLICIT list from `git ls-files`, never `ruff check .`, and
|
||||
# pass `--no-force-exclude`. Measured with ruff 0.12.11 and `exclude = ["scripts/**"]` — a
|
||||
# per-FILE pattern, because `exclude` matches per file: a bare `["scripts"]` still works at the
|
||||
# top level but matches nothing under `[lint]`/`[format]`. The subject is a planted tracked file
|
||||
# holding an unused import, a hardcoded credential and a formatting error. GREEN means the gate
|
||||
# was silently off:
|
||||
#
|
||||
# DISCOVERY FORM EXPLICIT FORM (what ships)
|
||||
# exclude scope check . format --check . check format --check
|
||||
# top-level GREEN GREEN red red
|
||||
# [lint] GREEN red red red
|
||||
# [format] red GREEN red red
|
||||
# top + force-exclude GREEN GREEN red red <- with the flag
|
||||
# GREEN GREEN <- without it
|
||||
#
|
||||
# Only the top-level scope empties BOTH discovery commands; `[lint]` empties `check` and
|
||||
# `[format]` empties `format --check`, so in those two the job would still redden on the other
|
||||
# step. `[format]` is where a line appended to ruff.toml lands, by TOML rules. `include = []`,
|
||||
# `extend-exclude` and a nested `scripts/ruff.toml` behave the same way and are equally inert
|
||||
# against the explicit form. The last row is the whole reason for `--no-force-exclude`:
|
||||
# `force-exclude = true` re-applies excludes to explicitly-passed paths, and is the one setting
|
||||
# that reaches explicitly-passed paths at all.
|
||||
#
|
||||
# `ruff check .` over an empty tree exits **0** with only a stderr warning, so every GREEN above
|
||||
# is a gate that was switched off without a red.
|
||||
#
|
||||
# This also derives the population from source rather than from the filesystem
|
||||
# (docs/decisions/records/testing/guard-derives-population-from-source.md) and covers
|
||||
# tracked-but-gitignored files, which `ruff check .` skips. The empty-population arm is the
|
||||
# anti-vacuity check: a completeness check whose population is empty reports that it proved
|
||||
# everything. What it does NOT cover: an emptied RULE set. `select = []` silences every selected
|
||||
# rule, so the `ruff check` step goes green over any lint violation (a syntax error still reds)
|
||||
# while printing a reassuring file count.
|
||||
# `ruff format --check` is unaffected, because formatting is not rule-selected. So half the
|
||||
# gate is killable by a config edit, and only a human reading that edit catches it.
|
||||
- name: Lint scripts (ruff check)
|
||||
run: |
|
||||
mapfile -d '' -t PYFILES < <(git ls-files -z '*.py' '*.pyi' '*.ipynb')
|
||||
if [ "${#PYFILES[@]}" -eq 0 ]; then
|
||||
echo "::error::the lint population is EMPTY — git tracks no Python files. Either the" \
|
||||
"checkout is wrong or the glob is. A lint over nothing passes; see ersatztv#780."
|
||||
exit 1
|
||||
fi
|
||||
echo "Linting ${#PYFILES[@]} tracked Python files"
|
||||
python3 -m ruff check --no-force-exclude -- "${PYFILES[@]}"
|
||||
- name: Lint scripts (ruff format --check)
|
||||
run: |
|
||||
mapfile -d '' -t PYFILES < <(git ls-files -z '*.py' '*.pyi' '*.ipynb')
|
||||
if [ "${#PYFILES[@]}" -eq 0 ]; then
|
||||
echo "::error::the format population is EMPTY — git tracks no Python files. See ersatztv#780."
|
||||
exit 1
|
||||
fi
|
||||
echo "Format-checking ${#PYFILES[@]} tracked Python files"
|
||||
python3 -m ruff format --check --no-force-exclude -- "${PYFILES[@]}"
|
||||
# pytest + PyYAML. PyYAML is NOT a contradiction of the dependency-free decisions READ path:
|
||||
# `decisions_lib._read_frontmatter` is hand-written precisely so validation runs where nothing
|
||||
# is installed, but the one-shot WRITE path `migrate_decisions_split.py` uses PyYAML by
|
||||
@@ -379,20 +463,6 @@ jobs:
|
||||
# went red in CI on a collection error.
|
||||
- name: Install test dependencies
|
||||
run: python3 -m pip install --disable-pip-version-check --quiet pytest pyyaml
|
||||
# Preflight, not an install (ersatztv#390 removed run-time `apt-get` from CI on purpose).
|
||||
# test_post_review_verdict.py and test_merge_consent_exemption.py exec the REAL
|
||||
# post-review-verdict.sh / pretooluse-merge-consent.sh, which shell out to `jq` ~26 times.
|
||||
# `curl` those tests shim on PATH; `jq` they do NOT. If it were missing, the suite would fail
|
||||
# as ~20 opaque assertion errors — this turns that into one actionable line.
|
||||
- name: Preflight external tools
|
||||
run: |
|
||||
if ! command -v git >/dev/null 2>&1; then
|
||||
echo "::error::script-tests needs git on PATH but it is absent. The suite execs real" \
|
||||
"shell scripts that use it. Bake it into the runner image rather than apt-get" \
|
||||
"installing here (see ersatztv#390)."
|
||||
exit 1
|
||||
fi
|
||||
echo "Preflight OK: $(git --version)"
|
||||
# jq gets its OWN step because its VERSION, not merely its presence, is load-bearing
|
||||
# (ersatztv#648). `--expect` makes this a TRIPWIRE: scripts/tests exercises the jq 1.6 code path
|
||||
# only because this runner ships 1.6, so an upgrade would silently delete that coverage — and
|
||||
|
||||
+29
-3
@@ -928,14 +928,14 @@ this job (`prove-fix.sh` exits 5 on a harness/git failure or a signal, and the j
|
||||
error), so what the exit-code discipline buys is the other way round — a **green** here means a claim
|
||||
was witnessed, never that a run was cancelled or broke.
|
||||
|
||||
### `script-tests` job (`Script tests (pytest)`, PR-only — in `pr-checks.yml`)
|
||||
### `script-tests` job (`Script lint and tests (ruff + pytest)`, PR-only — in `pr-checks.yml`)
|
||||
|
||||
> Reddens the run on failure, but like the other `pr-checks.yml` gates it is **not** one of the
|
||||
> three required status checks on `main` (`Build & test (.NET)`, `EF migration integrity`,
|
||||
> `review-verdict/h10`). Promoting it to required is a branch-protection change, tracked separately.
|
||||
|
||||
Runs the repository's Python test suite: `PYTHONPATH=. python3 -m pytest scripts/tests -q`
|
||||
(~190 tests at time of writing, ~10s; the suite grows, so treat the figure as indicative). It covers the decision-corpus parser/validator/catalog builder, the ersatztv#610
|
||||
(773 tests at `706674272`, ~4.5 min; the suite grows fast — it was ~190 tests / ~10s when this job landed in #631 — so treat the figure as a dated snapshot, not a budget). It covers the decision-corpus parser/validator/catalog builder, the ersatztv#610
|
||||
migration-equivalence harness, the merge-consent exemption logic and the ersatztv#622 review-verdict
|
||||
poster.
|
||||
|
||||
@@ -956,7 +956,33 @@ name keeps a real failure unambiguous.
|
||||
input set spans more than one directory — `test_post_review_verdict.py` and
|
||||
`test_merge_consent_exemption.py` execute the real `scripts/post-review-verdict.sh` and
|
||||
`.claude/hooks/pretooluse-merge-consent.sh` — so a `scripts/**` filter would silently miss a
|
||||
`.claude/hooks/**` edit. At ~10s, a filter buys nothing but drift.
|
||||
`.claude/hooks/**` edit. The reason is the input set, not the cost: the suite was ~10s when that was
|
||||
decided and is now ~4.5 min, and it would still be wrong to filter on `scripts/**`.
|
||||
|
||||
**It also lints (ersatztv#780).** Early in the job it installs a **pinned** `ruff==0.12.11` and runs
|
||||
`ruff check` and `ruff format --check` against the repo-root `ruff.toml`. Five things are deliberate:
|
||||
|
||||
- The config is **committed**. Without it ruff falls back to whatever `~/.config/ruff/ruff.toml` the
|
||||
operator's machine has, so a second machine lints this repo differently or not at all.
|
||||
- The version is **pinned** — an unpinned install makes the verdict a function of when the job ran,
|
||||
the same divergence one layer up, and the same argument as the `jq` pin below. `pytest`/`pyyaml`
|
||||
stay unpinned on purpose: a pytest release does not add assertions to your suite, a ruff release
|
||||
adds rules to your lint.
|
||||
- Lint runs **before the jq preflight**, and after the `git` one. `Preflight jq version` is a hard
|
||||
`--expect` tripwire; a lint sitting behind it goes dark for as long as the jq contract is broken,
|
||||
under a red that says "jq". `Preflight external tools` stays ahead, because the lint steps consume
|
||||
`git` — without it a missing git reaches them as an empty population and they blame the glob.
|
||||
- Neither step is `ruff check .`. Both pass an **explicit population** from
|
||||
`git ls-files -z '*.py' '*.pyi' '*.ipynb'` with `--no-force-exclude`, and fail if that list is
|
||||
empty. Discovery-based invocation is silently emptied by an `exclude` in the right config scope —
|
||||
top level empties both commands, `[lint]` empties `check`, `[format]` empties `format --check`
|
||||
(and `[format]` is where an appended line lands) — and
|
||||
`ruff check .` over zero files exits **0** with only a stderr warning, so the failure mode is a
|
||||
green gate. The measured matrix is in `ci.python-lint-ruff-config-committed`.
|
||||
- `RUF100` is selected, so a `# noqa` that no longer matches anything is itself a finding.
|
||||
|
||||
`pyright` is not gated; the reasoning and the exemption list are in
|
||||
`ci.python-lint-ruff-config-committed`.
|
||||
|
||||
**Dependencies: `pytest` and `pyyaml`** — the complete third-party set across `scripts/`, established
|
||||
by an AST import scan rather than by reading the files that looked relevant. PyYAML does **not**
|
||||
|
||||
@@ -56,10 +56,11 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `ci.monitor-armed-at-pr-open` | Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work. | 2026-07-21 | [link](records/ci/monitor-armed-at-pr-open.md) |
|
||||
| `ci.no-host-health-gating` | Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs. | 2026-07-21 | [link](records/ci/no-host-health-gating.md) |
|
||||
| `ci.peak-anon-measurement` | The `test` job's headline memory figure is a sampled high-water mark of cgroup `anon`, produced by `scripts/ci-peak-anon.sh`; `memory.peak` and the end-of-job `anon`/`file` split are kept only as a cache-inflated reference. | 2026-07-19 | [link](records/ci/peak-anon-measurement.md) |
|
||||
| `ci.python-lint-ruff-config-committed` | The repo commits `ruff.toml`, and the `script-tests` job runs `ruff check` + `ruff format --check` under a PINNED ruff over an EXPLICIT population from `git ls-files`, never `ruff check .`. Never rely on `~/.config/ruff/ruff.toml`, and never add a lint rule to the config without making the tree clean against it in the same PR. | 2026-08-21 | [link](records/ci/python-lint-ruff-config-committed.md) |
|
||||
| `ci.required-job-step-execution-markers` | A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`'s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job's LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step's `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships. The ban is enforced on the RELEASE PATH itself, not only in review (#767): a `scan` job runs the PyYAML-based ban test and `build` lists it in `needs:`, so a delimiter means `build` never runs and no image is published. A guard STEP inside `build` was tried first and is wrong — a step cannot protect the job it publishes from, and "my body has no opener so I cannot be dropped" is circular when only the PR-only test enforces that. The pytest in `script-tests` remains, but it is `on: pull_request` and not a required context, so it alone left the tag path unchecked. | 2026-08-10 | [link](records/ci/required-job-step-execution-markers.md) |
|
||||
| `ci.root-screenshot-guard` | The Husky `pre-commit` hook refuses a staged root-level `*.png` (belt-and-suspenders with the `.gitignore` rule); nested `*.png` real assets are unaffected. | 2026-07-12 | [link](records/ci/root-screenshot-guard.md) |
|
||||
| `ci.runner-placement` | No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's). | 2026-07-17 | [link](records/ci/runner-placement.md) |
|
||||
| `ci.script-tests-job` | The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it. | 2026-07-26 | [link](records/ci/script-tests-job.md) |
|
||||
| `ci.script-tests-job` | The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest pyyaml`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`; since #780 it also runs a pinned ruff over a `git ls-files` population first), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it. | 2026-07-26 | [link](records/ci/script-tests-job.md) |
|
||||
| `ci.shared-pr-file-enumeration` | A PR's complete set of changed file paths is computed by exactly one implementation, `scripts/pr-changed-files.sh`, called by both `.claude/hooks/pretooluse-merge-consent.sh` (advisory — a failure falls through to a human prompt) and `.gitea/workflows/review-verdict.yml` (enforced — a failure must fail closed, because a match here posts the branch-protection-required `review-verdict/h10` status with nobody in the loop). The script owns exhaustiveness (pagination, rename/path validation, head-sha binding, base-ref binding — see `ci.exemption-provenance` — and base-TIP binding, #707: the ref answers "did this PR RETARGET", the tip answers "did the base ADVANCE mid-enumeration", and only the second can see `/pulls/{n}/files` recomputing each offset-paged page against a moved base and dropping a path out of an already-consumed range; both ends of the window are bound, and an advance BEFORE the window is deliberately not an error, or ordinary churn on `main` would fail every open PR) and returns exit 0 only for a verified-complete list; it does NOT classify paths — each caller keeps its own docs-only allow-list, and the two allow-lists differ on purpose and stay separate. | 2026-07-26 | [link](records/ci/shared-pr-file-enumeration.md) |
|
||||
| `ci.small-lane-git-only` | `runs-on: small` is defined by what a job does (git-only), not its usual runtime; the two `docker build` jobs (docker-build.yml, ci-image.yml) move to `ubuntu-latest` because their worst-case memory, not median runtime, was pinning the small lane's per-slot cap. | 2026-07-20 | [link](records/ci/small-lane-git-only.md) |
|
||||
| `ci.ui-e2e-harness` | The UI-interactive E2E flows run as headless Playwright specs (`web/e2e/*.spec.ts`, driven by `scripts/e2e-ui.sh`) in a **second step of the existing advisory `functional-e2e` job**, never their own job; the browser is `chromium-headless-shell` **baked into the CI toolchain image** (`docker/ci/Dockerfile`, `PLAYWRIGHT_VERSION` kept equal to `web/package.json`'s EXACT `@playwright/test` pin), never installed per run; specs are `serial` with `retries: 0` and assert only contracts the curl harness structurally cannot reach. | 2026-07-25 | [link](records/ci/ui-e2e-harness.md) |
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
key: ci.python-lint-ruff-config-committed
|
||||
title: 2026-08-21 — Python lint is a committed ruff.toml enforced in CI, not the operator machine's global config (#780)
|
||||
status: active
|
||||
since: '2026-08-21'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: The repo commits `ruff.toml`, and the `script-tests` job runs `ruff check` + `ruff format --check` under a PINNED ruff over an EXPLICIT population from `git ls-files`, never `ruff check .`. Never rely on `~/.config/ruff/ruff.toml`, and never add a lint rule to the config without making the tree clean against it in the same PR.
|
||||
signals: 'ruff · pyright · python lint · `ruff format --check` · lint passes on my machine but not yours · no repo lint config · S105 on a test stub credential · paths: `ruff.toml`, `.gitea/workflows/pr-checks.yml` · issues: #780, #773, #648, #512'
|
||||
mechanics: 'Config at repo root; `.gitea/workflows/pr-checks.yml` -> `script-tests` pins `ruff==0.12.11` via pip and runs both commands ahead of the jq preflight and pytest. Population is `git ls-files -z ''*.py'' ''*.pyi'' ''*.ipynb''` passed explicitly with `--no-force-exclude`, guarded by an empty-list arm; discovery-based invocation is defeated by an `exclude` in three config scopes, two of them per command. Bumping the pin is a deliberate PR because a new ruff release adds rules.'
|
||||
---
|
||||
|
||||
The global instructions tell every session to run `ruff check`, `ruff format --check` and `pyright`
|
||||
after touching Python. Before this, the repo enforced none of them and committed no config, so ruff
|
||||
fell back to whichever `~/.config/ruff/ruff.toml` the operator's machine happened to have — **a second
|
||||
machine lints this repo differently, or not at all.** That is the same shape as #643/#647/#648 (a
|
||||
shell gate whose behaviour was a function of an untested interpreter version) and #512 (a test that
|
||||
passed on a fast laptop and flaked on a starved CI VM): the verdict was a property of the environment
|
||||
rather than of the repo.
|
||||
|
||||
The committed config is the operator's global one apart from `per-file-ignores`, which is narrowed
|
||||
to `scripts/tests/**`. That is what the tree was de-facto written against, so adopting it cost a
|
||||
mechanical reformat rather than a rewrite: 74 findings against `706674272`, of which 57 were fixed in
|
||||
code (mostly by the format pass) and 17 carry a per-site `# noqa` with its reason inline. `RUF100` is
|
||||
selected so those suppressions stay honest — a `# noqa` that suppresses nothing is otherwise
|
||||
invisible, and three were live the moment the rule was switched on: one whose rule had stopped firing,
|
||||
one for a rule this config never enables, and one added mid-branch on a site the same branch had
|
||||
already fixed in code.
|
||||
|
||||
**One exemption is directory-wide, and it is the boring one.** `S101` for `scripts/tests/**`, because
|
||||
a test suite asserts. **`S105` is deliberately NOT directory-wide.** All eight of its hits among
|
||||
those 74 findings are stub credentials handed to the real hooks (`env["ETV_GITEA_TOKEN"] = "stub"`),
|
||||
with no true positive in the tree today (a ninth `# noqa: S105` predates this and sits on a
|
||||
commit-message marker in `decisions_validate.py`). A directory blanket would give up
|
||||
hardcoded-credential coverage over the largest Python surface in the repo, permanently, to suppress
|
||||
eight known lines — and this is the only Python lint the repo runs, so nothing else would catch a real
|
||||
token pasted into a fixture next year. Per-site `# noqa: S105` costs the same and keeps the rule live.
|
||||
|
||||
**The population comes from `git ls-files`, not from ruff's discovery, and that is the load-bearing
|
||||
part.** `ruff check .` reports on what it *discovers*, and an `exclude` defeats discovery in three
|
||||
different config scopes — including `[format]`, which is where an appended line lands by TOML rules (two of the
|
||||
three defeat each command). Measured with ruff 0.12.11 and `exclude = ["scripts/**"]`, against a
|
||||
tracked file holding an unused import, a hardcoded credential and a formatting error. The pattern
|
||||
matters: `exclude` is matched per FILE, so a bare `["scripts"]` works at the top level but matches
|
||||
nothing under `[lint]`/`[format]`. GREEN means the gate was silently off:
|
||||
|
||||
| `exclude` in | `ruff check .` | explicit `check` | `ruff format --check .` | explicit `format` |
|
||||
|---|---|---|---|---|
|
||||
| top level | GREEN | red | GREEN | red |
|
||||
| `[lint]` | GREEN | red | red | red |
|
||||
| `[format]` | red | red | GREEN | red |
|
||||
| top + `force-exclude` | GREEN | GREEN without `--no-force-exclude`, red with it | GREEN | same |
|
||||
|
||||
Only the top-level scope empties both discovery commands; `[lint]` empties `check`, `[format]` empties
|
||||
`format --check`, so in those two the job would still redden on the other step. `[format]` is where a
|
||||
line appended to `ruff.toml` lands, by TOML rules. The last row is the whole reason for the flag.
|
||||
|
||||
`include = []`, `extend-exclude` and a nested `scripts/ruff.toml` were tried too, and are equally
|
||||
inert against the explicit form. The empty-list arm is the anti-vacuity
|
||||
check — `ruff check .` over no files exits **0** with a stderr warning, so an emptied population is a
|
||||
green gate, not a red one. Enumerating from git also covers tracked-but-gitignored files, which
|
||||
discovery skips (`git add -f` under an ignored path is established practice here).
|
||||
|
||||
**The rule set is not covered, and that is a stated limit rather than an oversight.** `select = []`
|
||||
silences every selected rule, so the `ruff check` step goes green over any lint violation (a syntax
|
||||
error still reds) while still printing a reassuring file count. `ruff format --check` is unaffected, because formatting is not rule-selected. So the population
|
||||
arm makes an emptied *file* set loud, nothing makes an emptied *rule* set loud, and half the gate is
|
||||
killable by a config edit only a reviewer catches.
|
||||
|
||||
Both steps were witnessed red on the runner before merge, not argued to work — **on the body that
|
||||
shipped**: run 2173 job 9176 (`❌ Failure - Main Lint scripts (ruff check)` on an `F401`) and run 2170
|
||||
job 9163 (`❌ Failure - Main Lint scripts (ruff format --check)`), printing `Linting 34 tracked
|
||||
Python files` and `Format-checking 34 tracked Python files` — the population arm executing (34 = the
|
||||
33 tracked files plus the probe; the merged tree has 33). Each came from a temporary probe commit
|
||||
reverted before merge. Two probes are needed, not one: a check-dirty file stops the job
|
||||
before the format step ever runs. Earlier reds against the previous, discovery-based bodies were
|
||||
discarded rather than cited — a proof belongs to the code that ran, not to its predecessor.
|
||||
|
||||
**Lint runs early in the job, ahead of the jq preflight.** `Preflight jq version` is a hard `--expect`
|
||||
tripwire; a lint step behind it stops running for as long as the jq contract is broken, under a red
|
||||
that names jq. Ordering is the difference between a gate that is skipped and one that is not. The
|
||||
`git` half of `Preflight external tools` stays *ahead* of the lint steps, because they consume `git`:
|
||||
without it, a missing git arrives as an empty population and both steps report a population problem
|
||||
instead of the missing tool.
|
||||
|
||||
**`pyright` is deliberately NOT gated.** Its only findings here are `reportMissingImports` for
|
||||
`etv_client` in `scripts/scripted-schedules/entrypoint.py`, resolvable only inside that script's
|
||||
deploy environment. Gating it would put a node toolchain on the git-only `small` lane to find nothing.
|
||||
Revisit when this repo grows a typed Python surface — the reason is the cost/finding ratio today, not
|
||||
a judgement that type checking does not belong.
|
||||
|
||||
**The pin is the second half of the fix.** An unpinned `pip install ruff` re-introduces exactly the
|
||||
divergence the config closes, one layer up: the verdict becomes a function of *when* the job ran. Same
|
||||
argument as the `jq` pin in the same job (`ci.jq-version-contract`), and the same consequence — a bump
|
||||
is a PR someone reads. `pytest` and `pyyaml` in the same job stay unpinned, and the
|
||||
asymmetry is the point rather than an oversight: a pytest release does not add assertions to your
|
||||
suite, a ruff release adds rules to your lint.
|
||||
@@ -5,7 +5,7 @@ status: active
|
||||
since: '2026-07-26'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it.'
|
||||
rule: 'The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest pyyaml`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`; since #780 it also runs a pinned ruff over a `git ls-files` population first), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it.'
|
||||
signals: 'scripts/tests never ran in CI, pytest not in any workflow, python test suite local-only, decorative test, decisions-guard runs the code not the tests, script-tests job, small lane pytest, negative control CI goes red · paths: `.gitea/workflows/pr-checks.yml`, `scripts/tests/`, `docs/ci-cd.md` · issues: #631, #610, #621, #622, #542'
|
||||
mechanics: '`.gitea/workflows/pr-checks.yml` -> `script-tests`; `docs/ci-cd.md` -> "`script-tests` job"'
|
||||
---
|
||||
@@ -29,7 +29,7 @@ it lives in, so **a job under a standing ignore-rule can host no real gate.**
|
||||
|
||||
This does not conflict with `ci.ui-e2e-harness` ("never their own job"). That record folds UI-E2E
|
||||
into `functional-e2e` because the specs need an app the job has *already booted* — sharing expensive
|
||||
setup. Here there is no shared setup to reuse (a checkout plus `pip install pytest pyyaml`), and the
|
||||
setup. Here there is no shared setup to reuse (a checkout plus `pip install pytest pyyaml` and a pinned ruff), and the
|
||||
sibling job carries an ignore-rule. Same question, opposite answers, for stated reasons.
|
||||
|
||||
**Unconditional, not path-filtered.** The suite's real input set spans more than `scripts/`:
|
||||
@@ -104,3 +104,9 @@ from a proxy ("fewer than we asked for", "jq didn't complain").
|
||||
It is not yet a *required* status check — `main` requires only `Build & test (.NET)`,
|
||||
`EF migration integrity` and `review-verdict/h10`. It reddens the run; promoting it to required is a
|
||||
branch-protection change left deliberately separate.
|
||||
|
||||
Since #780 the job also lints Python before pytest, so its display name is
|
||||
`Script lint and tests (ruff + pytest)`. It does **not** invoke `ruff check .` — the invocation and
|
||||
the reasons for its exact shape are `ci.python-lint-ruff-config-committed`. Why the lint lives here
|
||||
rather than in a job of its own: it needs the same `setup-python`, it costs seconds, and a second job would double the dispatch overhead
|
||||
this file exists to keep small.
|
||||
|
||||
@@ -445,6 +445,44 @@ machine lints differently, or not at all. One of the two must move — either th
|
||||
committed ruff config and enforces it, or the global instruction stops claiming this repo enforces
|
||||
something it does not.
|
||||
|
||||
**Resolved 2026-08-21 (#780): the repo moved.** `ruff.toml` is committed at the root and the
|
||||
`script-tests` job runs `ruff check` + `ruff format --check` under a pinned `ruff==0.12.11` over an
|
||||
explicit population from `git ls-files` — not `ruff check .`, which an `exclude` in the right config
|
||||
scope silently empties into a green run. The pre-fix state, reproducible rather than
|
||||
asserted — against `706674272`, the base this landed on, with the committed config dropped in:
|
||||
|
||||
```
|
||||
mkdir -p ~/scratch/m780 && git archive 706674272 | tar -x -C ~/scratch/m780
|
||||
git show 01f7a89e8:ruff.toml > ~/scratch/m780/ruff.toml # a sha: the file is not on main pre-merge
|
||||
cd ~/scratch/m780 && ruff check . ; ruff format --check . # ruff 0.12.11
|
||||
# -> Found 74 errors. / 20 files would be reformatted, 13 files already formatted
|
||||
```
|
||||
|
||||
(`;` not `&&` — `ruff check` exits 1, which would swallow the second command. Not `/tmp`: macOS purges
|
||||
it. Redirect the config into place *before* running anything: an empty `ruff.toml` is valid, so a
|
||||
failed `git show` leaves ruff silently using its own defaults and printing a different number.)
|
||||
|
||||
The row above measured **47** eight days earlier against the operator's global config; the tree grew
|
||||
and the configs differ, so the two numbers are not comparable and neither supersedes the other. Two
|
||||
of the 74 are `RUF100` on suppressions that were already in the tree before this change — they exist
|
||||
in this count only because the committed config enables that rule.
|
||||
|
||||
Of the 74, **57 were fixed in code** (most of them by the `ruff format` pass itself, which splits the
|
||||
40 semicolon statements) and **17 carry a per-site `# noqa` with its reason inline**: 8 `S105` on stub
|
||||
credentials handed to the real hooks by `scripts/tests`, 9 `E501` on one-line JSON and shell fixtures.
|
||||
The `S105`s are deliberately per-site rather than a directory exemption, so a real credential pasted
|
||||
into a fixture later still reddens the gate. Only `S101` is exempted directory-wide for `scripts/tests/**`,
|
||||
because a test suite asserts.
|
||||
|
||||
`RUF100` is selected, which is what keeps that split honest: a `# noqa` that suppresses nothing still
|
||||
reads as a suppression, and it is invisible without this rule. Three were live when it was switched
|
||||
on: one on a site that had already been fixed in code, plus the two counted above — one whose rule had
|
||||
stopped firing, one for a rule this config never enables.
|
||||
|
||||
`pyright` stayed ungated: its only findings are the `etv_client` imports in the row above, and gating
|
||||
it would put a node toolchain on the git-only `small` lane to find nothing. Rationale, the exemption
|
||||
list and the measured exclude matrix: `ci.python-lint-ruff-config-committed`.
|
||||
|
||||
### 5.3 Configured vs actually invoked
|
||||
|
||||
Measured over the session transcript corpus (811 files under
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# Ruff configuration for this repo's Python surface (all of it lives under `scripts/`).
|
||||
#
|
||||
# WHY THIS FILE EXISTS (ersatztv#780). Without a committed config, ruff falls back to whatever
|
||||
# `~/.config/ruff/ruff.toml` the operator's machine happens to have — so a second machine lints this
|
||||
# repo differently, or not at all. That is the environment-divergence class #643/#647/#648 (a shell
|
||||
# gate whose behaviour was a function of an untested interpreter version) and #512 (a test that
|
||||
# passed on a fast laptop and flaked on a starved CI VM). The settings below are pinned HERE so the
|
||||
# lint verdict is a property of the repo, not of the machine.
|
||||
#
|
||||
# It is enforced by the `script-tests` job (`Script lint and tests (ruff + pytest)`) in
|
||||
# .gitea/workflows/pr-checks.yml. A config nobody runs is the same divergence one step later.
|
||||
#
|
||||
# That job does NOT invoke `ruff check .`: it passes an explicit population from `git ls-files` with
|
||||
# `--no-force-exclude`. An `exclude` added to this file silently empties a discovery-based run into a
|
||||
# GREEN one — a top-level `exclude` empties both commands, one under `[lint]` empties `check`, one
|
||||
# under `[format]` (where an appended line lands, by TOML rules) empties `format --check`. Adding
|
||||
# `exclude` here will therefore not do what you expect, which is the point. The measured matrix is in
|
||||
# `ci.python-lint-ruff-config-committed`.
|
||||
#
|
||||
# `pyright` is deliberately NOT gated: its only findings here are `reportMissingImports` for
|
||||
# `etv_client` in scripts/scripted-schedules/entrypoint.py, which resolves only inside that script's
|
||||
# deploy environment, and gating it would put a node toolchain on the git-only `small` lane for zero
|
||||
# real findings. Revisit if this repo grows a typed Python surface.
|
||||
|
||||
target-version = "py311"
|
||||
line-length = 120
|
||||
|
||||
[lint]
|
||||
select = [
|
||||
"E", # pycodestyle errors
|
||||
"W", # pycodestyle warnings
|
||||
"F", # pyflakes
|
||||
"I", # isort
|
||||
"B", # flake8-bugbear
|
||||
"UP", # pyupgrade
|
||||
"SIM", # flake8-simplify
|
||||
"S", # flake8-bandit (security)
|
||||
# RUF100 is load-bearing, not tidiness: every `# noqa` below is an assertion that a real finding
|
||||
# is being suppressed for a stated reason, and without this a suppression that suppresses nothing
|
||||
# stays in the file reading as one. #780 did exactly that mid-branch — a `# noqa: UP031` on a site
|
||||
# the same branch had already fixed in code — and found two more already in the tree: one whose
|
||||
# rule had stopped firing, one for a rule this config never enables.
|
||||
"RUF100",
|
||||
]
|
||||
ignore = [
|
||||
"S603", # subprocess call - check for execution of untrusted input (too noisy for scripts)
|
||||
"S607", # starting a process with a partial executable path
|
||||
]
|
||||
|
||||
[lint.per-file-ignores]
|
||||
# scripts/tests asserts, so S101 would fire on every test. S105 is deliberately NOT exempted here:
|
||||
# the eight sites that trip it (`env["ETV_GITEA_TOKEN"] = "stub"`) carry a per-site `# noqa: S105`
|
||||
# instead, so a real credential pasted into a fixture next year still reddens the gate. A directory
|
||||
# blanket would have given up hardcoded-credential coverage over the largest Python surface in the
|
||||
# repo, permanently, to suppress eight known lines.
|
||||
"scripts/tests/**" = ["S101"]
|
||||
|
||||
[format]
|
||||
quote-style = "double"
|
||||
@@ -19,7 +19,7 @@ from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
import scripts.decisions_lib as dl # noqa: E402 (run with PYTHONPATH=. or as module)
|
||||
import scripts.decisions_lib as dl # (run with PYTHONPATH=. or as module)
|
||||
|
||||
SKIP_HEADINGS = dl.SKIP_HEADINGS # single source of truth
|
||||
# `signals` is required alongside the lifecycle fields: the `**Signals:**` line (plus `key:`) is what
|
||||
@@ -469,12 +469,7 @@ def _is_stripped_index(path: Path, archive_dir: Path, recs: list) -> bool:
|
||||
file there is exempt only if it actually LOOKS like a stripped index: exactly one keyless
|
||||
record whose heading is one of the known generated ones (`dl.SKIP_HEADINGS`).
|
||||
"""
|
||||
return (
|
||||
path.parent == archive_dir
|
||||
and len(recs) == 1
|
||||
and not recs[0].key
|
||||
and recs[0].heading in dl.SKIP_HEADINGS
|
||||
)
|
||||
return path.parent == archive_dir and len(recs) == 1 and not recs[0].key and recs[0].heading in dl.SKIP_HEADINGS
|
||||
|
||||
|
||||
def record_wing_faults(records_dir: Path | None = None, archive_dir: Path | None = None) -> list[str]:
|
||||
|
||||
@@ -62,9 +62,7 @@ def render(spec: dict) -> str:
|
||||
"`scripts/update-openapi.sh`.*"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append(
|
||||
f"{len(spec.get('paths', {}))} endpoints, {len(operations)} operations."
|
||||
)
|
||||
lines.append(f"{len(spec.get('paths', {}))} endpoints, {len(operations)} operations.")
|
||||
lines.append("")
|
||||
|
||||
for tag in tags:
|
||||
@@ -73,9 +71,7 @@ def render(spec: dict) -> str:
|
||||
lines.append("| Method | Path | Operation | Summary |")
|
||||
lines.append("|---|---|---|---|")
|
||||
for op in sorted(grouped[tag], key=lambda o: (o["path"], o["method"])):
|
||||
lines.append(
|
||||
f"| {op['method']} | `{op['path']}` | {op['operationId']} | {op['summary']} |"
|
||||
)
|
||||
lines.append(f"| {op['method']} | `{op['path']}` | {op['operationId']} | {op['summary']} |")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines).rstrip("\n") + "\n"
|
||||
|
||||
+33
-15
@@ -37,6 +37,7 @@ explicitly NOT bounded; that is the accepted limit stated above.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
@@ -47,6 +48,7 @@ import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
|
||||
def fail(msg: str, code: int) -> int:
|
||||
print(f"FAIL: {msg}")
|
||||
return code
|
||||
@@ -60,15 +62,19 @@ def main() -> int:
|
||||
i = 0
|
||||
while i < len(argv):
|
||||
if argv[i] == "--expect-server" and i + 1 < len(argv):
|
||||
expect_server = argv[i + 1]; i += 2
|
||||
expect_server = argv[i + 1]
|
||||
i += 2
|
||||
elif argv[i] == "--expect-tool" and i + 1 < len(argv):
|
||||
expect_tools.append(argv[i + 1]); i += 2
|
||||
expect_tools.append(argv[i + 1])
|
||||
i += 2
|
||||
else:
|
||||
positional.append(argv[i]); i += 1
|
||||
positional.append(argv[i])
|
||||
i += 1
|
||||
|
||||
if len(positional) < 2:
|
||||
return fail("usage: mcp_smoke.py <.mcp.json> <server> [timeout] "
|
||||
"[--expect-server NAME] [--expect-tool NAME]...", 2)
|
||||
return fail(
|
||||
"usage: mcp_smoke.py <.mcp.json> <server> [timeout] [--expect-server NAME] [--expect-tool NAME]...", 2
|
||||
)
|
||||
cfg_path, server = positional[0], positional[1]
|
||||
if len(positional) > 2:
|
||||
try:
|
||||
@@ -136,8 +142,12 @@ def main() -> int:
|
||||
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
[resolved, *args], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL, env=env, cwd=workdir,
|
||||
[resolved, *args],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
env=env,
|
||||
cwd=workdir,
|
||||
start_new_session=True, # own process group, so children die with us
|
||||
)
|
||||
except OSError as exc:
|
||||
@@ -222,10 +232,8 @@ def main() -> int:
|
||||
os.killpg(pgid, sig)
|
||||
except OSError:
|
||||
break # no group members left
|
||||
try:
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
proc.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
time.sleep(0.2)
|
||||
else:
|
||||
for sig in (signal.SIGTERM, signal.SIGKILL):
|
||||
@@ -245,9 +253,20 @@ def main() -> int:
|
||||
id_tools = secrets.randbelow(2**31 - 1000) + 1000
|
||||
while id_tools == id_init:
|
||||
id_tools = secrets.randbelow(2**31 - 1000) + 1000
|
||||
init = expect(id_init, {"jsonrpc": "2.0", "id": id_init, "method": "initialize", "params": {
|
||||
"protocolVersion": "2024-11-05", "capabilities": {},
|
||||
"clientInfo": {"name": "mcp-smoke", "version": "0"}}}, deadline)
|
||||
init = expect(
|
||||
id_init,
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": id_init,
|
||||
"method": "initialize",
|
||||
"params": {
|
||||
"protocolVersion": "2024-11-05",
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "mcp-smoke", "version": "0"},
|
||||
},
|
||||
},
|
||||
deadline,
|
||||
)
|
||||
if init is None:
|
||||
return fail(f"no 'initialize' response within {budget}s (server did not start)", 9)
|
||||
if "error" in init:
|
||||
@@ -263,8 +282,7 @@ def main() -> int:
|
||||
return fail(f"wrong server: expected '{expect_server}', got '{actual}'", 13)
|
||||
|
||||
send({"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}})
|
||||
listed = expect(id_tools, {"jsonrpc": "2.0", "id": id_tools,
|
||||
"method": "tools/list", "params": {}}, deadline)
|
||||
listed = expect(id_tools, {"jsonrpc": "2.0", "id": id_tools, "method": "tools/list", "params": {}}, deadline)
|
||||
if listed is None:
|
||||
return fail(f"no 'tools/list' response within {budget}s", 10)
|
||||
lresult = listed.get("result")
|
||||
|
||||
@@ -3,18 +3,18 @@
|
||||
import argparse
|
||||
import importlib
|
||||
import sys
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
import etv_client
|
||||
from etv_client.api import ScriptedScheduleApi
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Run an ETV scripted schedule")
|
||||
parser.add_argument('host', help="The ETV host (e.g., http://localhost:8409)")
|
||||
parser.add_argument('build_id', type=UUID, help="The build ID for the playout")
|
||||
parser.add_argument('mode', choices=['reset', 'continue'], help="The playout build mode")
|
||||
parser.add_argument('script_name', help="The name of the script module to use (e.g., one)")
|
||||
parser.add_argument("host", help="The ETV host (e.g., http://localhost:8409)")
|
||||
parser.add_argument("build_id", type=UUID, help="The build ID for the playout")
|
||||
parser.add_argument("mode", choices=["reset", "continue"], help="The playout build mode")
|
||||
parser.add_argument("script_name", help="The name of the script module to use (e.g., one)")
|
||||
|
||||
known_args, unknown_args = parser.parse_known_args()
|
||||
|
||||
@@ -28,9 +28,9 @@ def main():
|
||||
|
||||
with etv_client.ApiClient(configuration) as api_client:
|
||||
try:
|
||||
define_content = getattr(script_module, 'define_content')
|
||||
reset_playout = getattr(script_module, 'reset_playout')
|
||||
build_playout = getattr(script_module, 'build_playout')
|
||||
define_content = script_module.define_content
|
||||
reset_playout = script_module.reset_playout
|
||||
build_playout = script_module.build_playout
|
||||
|
||||
api_instance = ScriptedScheduleApi(api_client)
|
||||
|
||||
@@ -48,6 +48,6 @@ def main():
|
||||
except AttributeError as e:
|
||||
print(f"Error: the '{known_args.script_name}' script is missing a required function. {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
|
||||
@@ -24,6 +24,4 @@ import pytest
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def isolate_hook_fire_log(tmp_path_factory, monkeypatch):
|
||||
monkeypatch.setenv(
|
||||
"ETV_HOOK_FIRE_LOG_DIR", str(tmp_path_factory.mktemp("hook-fire-log"))
|
||||
)
|
||||
monkeypatch.setenv("ETV_HOOK_FIRE_LOG_DIR", str(tmp_path_factory.mktemp("hook-fire-log")))
|
||||
|
||||
@@ -36,9 +36,17 @@ BOM = b"\xef\xbb\xbf"
|
||||
|
||||
def _git(cwd: Path, *args: str) -> None:
|
||||
subprocess.run(
|
||||
["git", *args], cwd=str(cwd), check=True, capture_output=True,
|
||||
env={**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e",
|
||||
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e"},
|
||||
["git", *args],
|
||||
cwd=str(cwd),
|
||||
check=True,
|
||||
capture_output=True,
|
||||
env={
|
||||
**os.environ,
|
||||
"GIT_AUTHOR_NAME": "t",
|
||||
"GIT_AUTHOR_EMAIL": "t@e",
|
||||
"GIT_COMMITTER_NAME": "t",
|
||||
"GIT_COMMITTER_EMAIL": "t@e",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -54,12 +62,18 @@ def _repo_with(tmp_path: Path, name: str, content: bytes) -> Path:
|
||||
|
||||
|
||||
def _run(hook: Path, repo: Path, env: dict) -> tuple[int, bytes]:
|
||||
payload = json.dumps({
|
||||
"session_id": "s", "hook_event_name": "PreToolUse", "tool_name": "Bash",
|
||||
"cwd": str(repo), "tool_input": {"command": "git commit -m x"},
|
||||
})
|
||||
p = subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
|
||||
cwd=str(repo), env=env, timeout=60)
|
||||
payload = json.dumps(
|
||||
{
|
||||
"session_id": "s",
|
||||
"hook_event_name": "PreToolUse",
|
||||
"tool_name": "Bash",
|
||||
"cwd": str(repo),
|
||||
"tool_input": {"command": "git commit -m x"},
|
||||
}
|
||||
)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)], input=payload.encode(), capture_output=True, cwd=str(repo), env=env, timeout=60
|
||||
)
|
||||
return p.returncode, p.stdout
|
||||
|
||||
|
||||
@@ -105,8 +119,9 @@ def _path_without_xxd(tmp_path: Path) -> dict:
|
||||
def test_the_fixture_really_stages_a_BOM(tmp_path):
|
||||
repo = _repo_with(tmp_path, "Bad.cs", BOM + b"class A {}\n")
|
||||
assert (repo / "Bad.cs").read_bytes()[:3] == BOM
|
||||
staged = subprocess.run(["git", "diff", "--name-only", "--cached"], cwd=str(repo),
|
||||
capture_output=True, text=True).stdout.split()
|
||||
staged = subprocess.run(
|
||||
["git", "diff", "--name-only", "--cached"], cwd=str(repo), capture_output=True, text=True
|
||||
).stdout.split()
|
||||
assert staged == ["Bad.cs"], f"nothing was staged, so the guard would have nothing to read: {staged}"
|
||||
|
||||
|
||||
@@ -179,6 +194,4 @@ def test_DISARMING_the_BOM_comparison_stops_detection(tmp_path):
|
||||
|
||||
rc, out = _run(mutated, repo, dict(os.environ))
|
||||
assert rc == 0
|
||||
assert out == b"", (
|
||||
f"disarming the BOM comparison did not stop detection, so it is not load-bearing: {out!r}"
|
||||
)
|
||||
assert out == b"", f"disarming the BOM comparison did not stop detection, so it is not load-bearing: {out!r}"
|
||||
|
||||
@@ -115,13 +115,13 @@ def _guard_buckets(job: str):
|
||||
argv = _guard(job)["run"].split()
|
||||
assert "--always" in argv and "--gated" in argv, argv
|
||||
a, g = argv.index("--always"), argv.index("--gated")
|
||||
return argv[a + 1:g], argv[g + 1:]
|
||||
return argv[a + 1 : g], argv[g + 1 :]
|
||||
|
||||
|
||||
# Mirrors the `if:` every gated step in these jobs carries. Compared as a normalised string rather
|
||||
# than by parsing the expression: what matters is that a step's gating and the guard's bucketing are
|
||||
# the SAME condition, and any rewrite of one that is not mirrored in the other should be loud.
|
||||
SKIP_GATE = ("steps.detect.outputs.docs_only!='true'&&steps.revalidate.outputs.skip!='true'")
|
||||
SKIP_GATE = "steps.detect.outputs.docs_only!='true'&&steps.revalidate.outputs.skip!='true'"
|
||||
|
||||
|
||||
def _is_gated(step) -> bool:
|
||||
@@ -211,17 +211,14 @@ def test_every_consequential_run_step_marks_itself_as_its_FIRST_act(job):
|
||||
# prefix as a preceding command and reddens every correctly-written step.
|
||||
lines = s["run"].splitlines()
|
||||
at = next(i for i, ln in enumerate(lines) if _MARK.search(ln))
|
||||
preceding = [
|
||||
ln.strip() for ln in lines[:at]
|
||||
if ln.strip() and not ln.strip().startswith("#")
|
||||
]
|
||||
preceding = [ln.strip() for ln in lines[:at] if ln.strip() and not ln.strip().startswith("#")]
|
||||
if [ln for ln in preceding if not ln.startswith("set -")]:
|
||||
late.append((s.get("name", "?"), preceding))
|
||||
assert not missing, (
|
||||
f"these run: steps of the REQUIRED job '{job}' do not record that they executed: {missing}. "
|
||||
"A step the runner drops concludes success, so without a marker its non-execution takes the "
|
||||
"whole required context green having done no work (ersatztv#756). Add "
|
||||
'`\"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh\" mark <key>` as the step\'s first line and '
|
||||
'`"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark <key>` as the step\'s first line and '
|
||||
"the key to the guard step's --always/--gated list."
|
||||
)
|
||||
assert not late, (
|
||||
@@ -357,8 +354,10 @@ def test_the_guards_OWN_body_cannot_be_dropped_by_the_mechanism_it_guards_agains
|
||||
# a red here blocks every merge through the combined status, so brittleness is a real cost and
|
||||
# not a free strictness win. Whitespace inside the delimiters is normalised for the same reason.
|
||||
env = {k: re.sub(r"\s+", "", str(v)) for k, v in (guard.get("env") or {}).items()}
|
||||
for name, want in (("ETV_DOCS_ONLY", "${{steps.detect.outputs.docs_only}}"),
|
||||
("ETV_REVALIDATE_SKIP", "${{steps.revalidate.outputs.skip}}")):
|
||||
for name, want in (
|
||||
("ETV_DOCS_ONLY", "${{steps.detect.outputs.docs_only}}"),
|
||||
("ETV_REVALIDATE_SKIP", "${{steps.revalidate.outputs.skip}}"),
|
||||
):
|
||||
assert env.get(name) == want, (
|
||||
f"the '{job}' guard's env: has {name}={guard.get('env', {}).get(name)!r}, expected the "
|
||||
f"output the gated steps' own `if:` reads ({want}). A typo here is SILENT rather than "
|
||||
@@ -427,12 +426,10 @@ def _env(tmp_path, **extra):
|
||||
|
||||
|
||||
def _run(script: str, env):
|
||||
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env,
|
||||
capture_output=True, text=True)
|
||||
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env, capture_output=True, text=True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("gate", GATE_VALUES_IN_THE_WILD,
|
||||
ids=["gate-false", "gate-empty", "gate-unset"])
|
||||
@pytest.mark.parametrize("gate", GATE_VALUES_IN_THE_WILD, ids=["gate-false", "gate-empty", "gate-unset"])
|
||||
@pytest.mark.parametrize("job", MARKED_JOBS)
|
||||
def test_the_guard_PASSES_when_every_step_marked_itself(job, gate, tmp_path):
|
||||
"""The positive control. Without it, a guard that always failed would satisfy every case below.
|
||||
@@ -458,7 +455,8 @@ def test_the_guard_PASSES_when_every_step_marked_itself(job, gate, tmp_path):
|
||||
# inverted provenance test would otherwise ship silently, and the operator reading this line to
|
||||
# settle the promotion question would read it wrong.
|
||||
assert f"Marker identity: job={job} run=424242 attempt=7 (from the runner)" in r.stdout, (
|
||||
f"the guard misreported its marker identity: {r.stdout!r}")
|
||||
f"the guard misreported its marker identity: {r.stdout!r}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("job", MARKED_JOBS)
|
||||
@@ -534,8 +532,7 @@ def test_an_EMPTY_gate_value_requires_the_gated_steps(job, tmp_path):
|
||||
what is required.
|
||||
"""
|
||||
guard = _guard(job)["run"]
|
||||
r = _run("\n".join(["set -e", guard]),
|
||||
_env(tmp_path, GITHUB_JOB=job, ETV_DOCS_ONLY="", ETV_REVALIDATE_SKIP=""))
|
||||
r = _run("\n".join(["set -e", guard]), _env(tmp_path, GITHUB_JOB=job, ETV_DOCS_ONLY="", ETV_REVALIDATE_SKIP=""))
|
||||
assert r.returncode != 0
|
||||
assert _guard_buckets(job)[1][-1] in (r.stdout + r.stderr), (
|
||||
f"empty gate values were read as a skip, so the gated steps went unchecked: {r.stdout}"
|
||||
@@ -571,8 +568,7 @@ def test_a_STALE_marker_from_another_run_cannot_satisfy_the_guard(tmp_path):
|
||||
# ...nor may the OTHER job in the same run inherit them.
|
||||
sibling = _env(tmp_path, GITHUB_JOB="migrations", GITHUB_RUN_ID="111", GITHUB_RUN_ATTEMPT="1")
|
||||
assert _run(_guard("migrations")["run"], sibling).returncode != 0, (
|
||||
"the two required jobs share one marker file, so one job's markers answer for the other's "
|
||||
"dropped steps"
|
||||
"the two required jobs share one marker file, so one job's markers answer for the other's dropped steps"
|
||||
)
|
||||
|
||||
|
||||
@@ -611,8 +607,7 @@ def test_a_key_is_matched_WHOLE_not_as_a_substring(tmp_path):
|
||||
assert _run(f"{SCRIPT} mark web-build && {SCRIPT} mark web-test", env).returncode == 0
|
||||
r = _run(f"{SCRIPT} assert --always build", env)
|
||||
assert r.returncode != 0, (
|
||||
"the key 'build' was satisfied by a marker for 'web-build' — a dropped `dotnet build` would "
|
||||
"pass unnoticed"
|
||||
"the key 'build' was satisfied by a marker for 'web-build' — a dropped `dotnet build` would pass unnoticed"
|
||||
)
|
||||
|
||||
|
||||
@@ -638,9 +633,11 @@ def test_a_degraded_run_IDENTITY_refuses_rather_than_sharing_a_marker_path(tmp_p
|
||||
f"{r.stdout}{r.stderr}"
|
||||
)
|
||||
assert "cannot identify this run" in (r.stdout + r.stderr), (
|
||||
f"refused, but without naming the cause: {r.stdout!r} {r.stderr!r}")
|
||||
f"refused, but without naming the cause: {r.stdout!r} {r.stderr!r}"
|
||||
)
|
||||
assert not list(tmp_path.iterdir()), (
|
||||
"a degraded-identity `mark` still created a marker file somewhere under RUNNER_TEMP")
|
||||
"a degraded-identity `mark` still created a marker file somewhere under RUNNER_TEMP"
|
||||
)
|
||||
|
||||
|
||||
def test_the_marker_identity_is_REPORTED_on_stdout_every_run(tmp_path):
|
||||
@@ -661,12 +658,15 @@ def test_the_marker_identity_is_REPORTED_on_stdout_every_run(tmp_path):
|
||||
documented contract (the record's `mechanics:`), and a future reader is told to trust it.
|
||||
"""
|
||||
marks = [_mark_line(s) for s, _ in _marked("test")]
|
||||
r = _run("\n".join(["set -e", *marks, _guard("test")["run"]]),
|
||||
_env(tmp_path, GITHUB_RUN_ID="1916", GITHUB_RUN_ATTEMPT="4"))
|
||||
r = _run(
|
||||
"\n".join(["set -e", *marks, _guard("test")["run"]]),
|
||||
_env(tmp_path, GITHUB_RUN_ID="1916", GITHUB_RUN_ATTEMPT="4"),
|
||||
)
|
||||
assert r.returncode == 0, r.stdout + r.stderr
|
||||
assert "Marker identity: job=test run=1916 attempt=4 (from the runner)" in r.stdout, (
|
||||
"the guard did not report the identity its marker path was actually keyed on, so a reader "
|
||||
f"cannot audit the keying from a run log: {r.stdout!r}")
|
||||
f"cannot audit the keying from a run log: {r.stdout!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_skip_gate_that_empties_the_expected_set_REFUSES(tmp_path):
|
||||
@@ -681,15 +681,16 @@ def test_a_skip_gate_that_empties_the_expected_set_REFUSES(tmp_path):
|
||||
that it proved everything while proving nothing" is the failure this whole file exists to remove.
|
||||
"""
|
||||
r = _run(f"{SCRIPT} assert --always --gated foo", _env(tmp_path, ETV_DOCS_ONLY="true"))
|
||||
assert r.returncode != 0, (
|
||||
f"the guard passed with an empty post-gate expectation set: {r.stdout!r}")
|
||||
assert r.returncode != 0, f"the guard passed with an empty post-gate expectation set: {r.stdout!r}"
|
||||
assert "no expected keys" in (r.stdout + r.stderr).lower() or "NO expected keys" in r.stderr
|
||||
|
||||
|
||||
@pytest.mark.parametrize("revalidate", ["true", "false", "", None],
|
||||
ids=lambda v: f"reval-{v if v is not None else 'unset'}")
|
||||
@pytest.mark.parametrize("docs_only", ["true", "false", "", None],
|
||||
ids=lambda v: f"docs-{v if v is not None else 'unset'}")
|
||||
@pytest.mark.parametrize(
|
||||
"revalidate", ["true", "false", "", None], ids=lambda v: f"reval-{v if v is not None else 'unset'}"
|
||||
)
|
||||
@pytest.mark.parametrize(
|
||||
"docs_only", ["true", "false", "", None], ids=lambda v: f"docs-{v if v is not None else 'unset'}"
|
||||
)
|
||||
def test_the_skip_gate_over_the_WHOLE_value_matrix(docs_only, revalidate, tmp_path):
|
||||
"""Every combination of the two gate values, not just the diagonal — cold review's last finding.
|
||||
|
||||
@@ -713,11 +714,14 @@ def test_the_skip_gate_over_the_WHOLE_value_matrix(docs_only, revalidate, tmp_pa
|
||||
job = "test"
|
||||
always, gated = _guard_buckets(job)
|
||||
marks = [_mark_line(s) for s, k in _marked(job) if k in always]
|
||||
r = _run("\n".join(["set -e", *marks, _guard(job)["run"]]),
|
||||
_env(tmp_path, ETV_DOCS_ONLY=docs_only, ETV_REVALIDATE_SKIP=revalidate))
|
||||
r = _run(
|
||||
"\n".join(["set -e", *marks, _guard(job)["run"]]),
|
||||
_env(tmp_path, ETV_DOCS_ONLY=docs_only, ETV_REVALIDATE_SKIP=revalidate),
|
||||
)
|
||||
should_skip = docs_only == "true" or revalidate == "true"
|
||||
assert (r.returncode == 0) is should_skip, (
|
||||
f"with docs_only={docs_only!r} and revalidate={revalidate!r} the guard "
|
||||
f"{'passed' if r.returncode == 0 else 'failed'}, expected it to "
|
||||
f"{'skip the gated keys' if should_skip else 'require them'}. The gate must treat a value as "
|
||||
"a skip if and only if it is exactly `true` in EITHER variable.\n" + r.stdout + r.stderr)
|
||||
"a skip if and only if it is exactly `true` in EITHER variable.\n" + r.stdout + r.stderr
|
||||
)
|
||||
|
||||
@@ -198,8 +198,7 @@ def test_the_registry_partitions_every_job_in_the_workflow():
|
||||
)
|
||||
assert not (TOOLCHAIN_JOBS & BARE_RUNNER_JOBS), "a job cannot be in both lists"
|
||||
assert all_jobs == TOOLCHAIN_JOBS | BARE_RUNNER_JOBS, (
|
||||
f"the registry names jobs that do not exist: "
|
||||
f"{sorted((TOOLCHAIN_JOBS | BARE_RUNNER_JOBS) - all_jobs)}"
|
||||
f"the registry names jobs that do not exist: {sorted((TOOLCHAIN_JOBS | BARE_RUNNER_JOBS) - all_jobs)}"
|
||||
)
|
||||
|
||||
|
||||
@@ -242,8 +241,7 @@ def test_a_single_job_losing_its_pin_is_DETECTED(doc):
|
||||
empty list would satisfy the live assertion above and prove nothing — which is how #621 and
|
||||
#685 both shipped."""
|
||||
assert pin_population_faults(doc), (
|
||||
"the population check accepted a workflow in which a container job no longer runs the "
|
||||
"pinned toolchain image"
|
||||
"the population check accepted a workflow in which a container job no longer runs the pinned toolchain image"
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -328,8 +328,7 @@ def _run_the_real_scan_body(repo: Path, tmp_path: Path):
|
||||
(repo / _NONCE_FILE).write_text(nonce)
|
||||
env[_FENCE] = nonce
|
||||
(tmp_path / "runner").mkdir(exist_ok=True)
|
||||
return subprocess.run(["bash", "-c", body], cwd=repo, env=env,
|
||||
capture_output=True, text=True)
|
||||
return subprocess.run(["bash", "-c", body], cwd=repo, env=env, capture_output=True, text=True)
|
||||
|
||||
|
||||
@_nested
|
||||
@@ -358,7 +357,7 @@ def test_the_scan_step_REALLY_FAILS_on_a_poisoned_workflow(tmp_path):
|
||||
text = wf.read_text()
|
||||
anchor = ' IMG="${IMAGE}:${SMOKE_SHORT_SHA}"'
|
||||
assert anchor in text, "anchor for the poison is gone — rewrite this control"
|
||||
wf.write_text(text.replace(anchor, ' # ${{ steps.meta.outputs.short }}\n' + anchor, 1))
|
||||
wf.write_text(text.replace(anchor, " # ${{ steps.meta.outputs.short }}\n" + anchor, 1))
|
||||
|
||||
res = _run_the_real_scan_body(repo, tmp_path)
|
||||
assert res.returncode != 0, (
|
||||
@@ -403,8 +402,10 @@ def test_the_PROOF_SCRIPT_itself_refuses_when_the_ban_is_deselected(tmp_path):
|
||||
)
|
||||
res = subprocess.run(
|
||||
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
|
||||
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True, text=True,
|
||||
cwd=repo,
|
||||
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res.returncode != 0, (
|
||||
"ci-prove-ban-detects.sh vouched for the gate while the ban test was deselected at the "
|
||||
@@ -446,19 +447,16 @@ def test_the_PROOF_SCRIPT_refuses_when_the_WRONG_test_fails(tmp_path):
|
||||
# landing on the "not enforcing" branch instead of the one under test. (First draft of this test
|
||||
# did exactly that and was red for the wrong reason.)
|
||||
ban = repo / BAN_TEST_FILE
|
||||
ban.write_text(
|
||||
ban.read_text()
|
||||
+ "\n\ndef test_an_unrelated_failure_for_this_probe():\n assert False\n"
|
||||
)
|
||||
ban.write_text(ban.read_text() + "\n\ndef test_an_unrelated_failure_for_this_probe():\n assert False\n")
|
||||
res = subprocess.run(
|
||||
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
|
||||
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True, text=True,
|
||||
cwd=repo,
|
||||
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
combined = res.stdout + res.stderr
|
||||
assert res.returncode != 0, (
|
||||
f"the script read an unrelated test's failure as proof.\n{combined}"
|
||||
)
|
||||
assert res.returncode != 0, f"the script read an unrelated test's failure as proof.\n{combined}"
|
||||
# THE SPECIFIC branch. This scenario is built to land on "wrong test failed" (exit 1, no `[build]`
|
||||
# failure); accepting "could not prove anything" too would let it drift onto the exit-5 branch and
|
||||
# silently cover a branch it was not written for, while still looking green.
|
||||
@@ -474,15 +472,17 @@ def test_the_PROOF_SCRIPT_passes_on_a_clean_tree(tmp_path):
|
||||
repo = _repo_copy(tmp_path)
|
||||
res = subprocess.run(
|
||||
["bash", str(repo / "scripts" / "ci-prove-ban-detects.sh")],
|
||||
cwd=repo, env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True, text=True,
|
||||
cwd=repo,
|
||||
env={**os.environ, "GITHUB_WORKSPACE": str(repo)},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert res.returncode == 0, (
|
||||
f"ci-prove-ban-detects.sh failed on a clean tree.\nstdout:\n{res.stdout}\nstderr:\n{res.stderr}"
|
||||
)
|
||||
assert (repo / ".gitea" / "workflows" / "docker-build.yml").read_text() == (
|
||||
WORKFLOW.read_text()
|
||||
), "the script did not restore the workflow file it poisoned"
|
||||
assert (repo / ".gitea" / "workflows" / "docker-build.yml").read_text() == (WORKFLOW.read_text()), (
|
||||
"the script did not restore the workflow file it poisoned"
|
||||
)
|
||||
|
||||
|
||||
def test_the_pytest_invocation_cannot_DESELECT_or_swallow_its_result():
|
||||
@@ -494,7 +494,7 @@ def test_the_pytest_invocation_cannot_DESELECT_or_swallow_its_result():
|
||||
# Only the tokens AFTER `pytest` are pytest's own arguments. Checking the whole line would flag
|
||||
# the `-m` in `python3 -m pytest`, which is how the interpreter is invoked — a false positive
|
||||
# that would make this test red on the correct command.
|
||||
args = tokens[tokens.index("pytest") + 1:]
|
||||
args = tokens[tokens.index("pytest") + 1 :]
|
||||
banned = {"-k", "-m", "--deselect", "--ignore", "--collect-only", "--co"}
|
||||
assert not (banned & set(args)), f"pytest invocation may deselect tests: {line!r}"
|
||||
for op in ("||", "&&", ";", "|"):
|
||||
@@ -527,8 +527,7 @@ def test_no_step_in_the_scan_job_is_advisory():
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"step_name",
|
||||
[s.get("name", "?") for s in yaml.safe_load(WORKFLOW.read_text())["jobs"][JOB]["steps"]
|
||||
if s.get("run")],
|
||||
[s.get("name", "?") for s in yaml.safe_load(WORKFLOW.read_text())["jobs"][JOB]["steps"] if s.get("run")],
|
||||
)
|
||||
def test_every_run_body_in_the_scan_job_is_delimiter_free(step_name):
|
||||
"""The guard must not be vulnerable to the defect it guards against.
|
||||
@@ -561,7 +560,7 @@ def test_the_guard_expectations_match_the_markers_exactly():
|
||||
restated here, so adding a step without a marker is a red."""
|
||||
argv = _guard()["run"].split()
|
||||
assert "--always" in argv, argv
|
||||
always = argv[argv.index("--always") + 1:]
|
||||
always = argv[argv.index("--always") + 1 :]
|
||||
assert "--gated" not in argv, "every step in this job is unconditional; there is nothing to gate"
|
||||
assert sorted(always) == sorted(k for _, k in _marked())
|
||||
|
||||
@@ -601,8 +600,7 @@ def _env(tmp_path, **extra):
|
||||
|
||||
|
||||
def _run(script: str, env):
|
||||
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env,
|
||||
capture_output=True, text=True)
|
||||
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env, capture_output=True, text=True)
|
||||
|
||||
|
||||
def test_the_guard_PASSES_when_every_step_ran(tmp_path):
|
||||
|
||||
@@ -744,7 +744,9 @@ def test_wing_faults_block_scalar_record_fails_loudly(tmp_path):
|
||||
this corpus's very long `rule:` values — makes the whole record silently invisible."""
|
||||
records, archive = _wing(tmp_path)
|
||||
bad = records / "ci" / "blockscalar.md"
|
||||
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\n a long rule wrapped\n over two lines\n"))
|
||||
bad.write_text(
|
||||
_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\n a long rule wrapped\n over two lines\n")
|
||||
)
|
||||
|
||||
# Precondition: this really is the silent-vanish case, not some other parse error.
|
||||
assert dl.parse_file(bad) == [], "expected the reader to drop the record entirely"
|
||||
@@ -805,8 +807,10 @@ def test_wing_faults_exempts_stripped_legacy_archive_files(tmp_path):
|
||||
|
||||
def test_validate_surfaces_wing_faults_as_errors(tmp_path):
|
||||
"""Faults must arrive as validator ERRORS (exit 1), not warnings."""
|
||||
errs = _v([_rec(key="ci.a", source=Path("docs/decisions/records/ci/a.md"), heading="A")],
|
||||
wing_faults=["docs/decisions/records/ci/x.md: parsed to 0 records, expected exactly 1"])
|
||||
errs = _v(
|
||||
[_rec(key="ci.a", source=Path("docs/decisions/records/ci/a.md"), heading="A")],
|
||||
wing_faults=["docs/decisions/records/ci/x.md: parsed to 0 records, expected exactly 1"],
|
||||
)
|
||||
assert any("x.md" in e for e in errs), errs
|
||||
|
||||
|
||||
@@ -826,7 +830,7 @@ def test_wing_faults_sees_a_DEEPER_nested_archive_record(tmp_path):
|
||||
one level down'."""
|
||||
records, archive = _wing(tmp_path)
|
||||
(records / "ci" / "good.md").write_text(_GOOD)
|
||||
(archive / "api.md").write_text("# api\n\n## Records formerly in this file\n") # still exempt
|
||||
(archive / "api.md").write_text("# api\n\n## Records formerly in this file\n") # still exempt
|
||||
deep = archive / "ci" / "sub"
|
||||
deep.mkdir(parents=True)
|
||||
(deep / "broken.md").write_text("# not a record\n")
|
||||
@@ -887,8 +891,7 @@ def test_junk_frontmatter_key_from_a_split_value_is_faulted(tmp_path):
|
||||
PyYAML rejects this input, so the hand reader is more permissive than the writer."""
|
||||
records, archive = _wing(tmp_path)
|
||||
bad = records / "ci" / "corrupt.md"
|
||||
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n",
|
||||
"rule: >-\nthe real rule: with a colon\n"))
|
||||
bad.write_text(_GOOD.replace("rule: 'a rule on one quoted line'\n", "rule: >-\nthe real rule: with a colon\n"))
|
||||
recs = dl.parse_file(bad)
|
||||
assert len(recs) == 1 and recs[0].key, "precondition: this parses to one KEYED record"
|
||||
assert recs[0].rule == ">-", f"precondition: the real value was truncated, got {recs[0].rule!r}"
|
||||
@@ -903,7 +906,7 @@ def test_an_empty_or_missing_record_wing_is_LOUD(tmp_path):
|
||||
missing = dv.record_wing_faults(tmp_path / "nope" / "records", tmp_path / "nope" / "archive")
|
||||
assert missing and "missing or contains no" in missing[0], missing
|
||||
|
||||
records, archive = _wing(tmp_path) # exists but holds no *.md
|
||||
records, archive = _wing(tmp_path) # exists but holds no *.md
|
||||
empty = dv.record_wing_faults(records, archive)
|
||||
assert empty and "missing or contains no" in empty[0], empty
|
||||
|
||||
@@ -1437,7 +1440,6 @@ def test_no_budget_flag_means_no_retirement_warning(capsys):
|
||||
assert "is RETIRED and was IGNORED" not in capsys.readouterr().err
|
||||
|
||||
|
||||
|
||||
def test_main_reports_ceiling_drift_as_a_NOTICE_and_still_exits_0(capsys):
|
||||
"""The fine claim's live wiring (#688): the drift notice must fire, and must NOT turn the run
|
||||
red — the entire point of the v5 split.
|
||||
@@ -1511,8 +1513,7 @@ def test_main_actually_REPORTS_the_ceiling_and_the_trend(capsys):
|
||||
err = capsys.readouterr().err
|
||||
assert "prose lines across" in err, "the aggregate trend notice must always print"
|
||||
|
||||
over = [r.key for r in dl.all_active_records()
|
||||
if r.key and dv.record_prose_lines(r) > dv.RECORD_CEILING_DEFAULT]
|
||||
over = [r.key for r in dl.all_active_records() if r.key and dv.record_prose_lines(r) > dv.RECORD_CEILING_DEFAULT]
|
||||
warned = "exceed the" in err and "prose ceiling" in err
|
||||
assert warned is bool(over), f"ceiling warning printed={warned} but {len(over)} record(s) are over it"
|
||||
if over:
|
||||
|
||||
@@ -72,16 +72,17 @@ def derived_guard_files() -> set[str]:
|
||||
not a guard, and globbing would drag in every helper and make the inventory a chore that gets
|
||||
rubber-stamped.
|
||||
"""
|
||||
found = {
|
||||
str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")
|
||||
} | {
|
||||
str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()
|
||||
} | {
|
||||
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
|
||||
# invoked and none is individually named anywhere. Globbing is the only derivation that
|
||||
# matches how they actually run.
|
||||
str(p.relative_to(REPO_ROOT)) for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
|
||||
}
|
||||
found = (
|
||||
{str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh")}
|
||||
| {str(p.relative_to(REPO_ROOT)) for p in HUSKY_DIR.iterdir() if p.is_file()}
|
||||
| {
|
||||
# `pr-checks.yml` runs `pytest scripts/tests` as a directory, so every file in it is
|
||||
# invoked and none is individually named anywhere. Globbing is the only derivation that
|
||||
# matches how they actually run.
|
||||
str(p.relative_to(REPO_ROOT))
|
||||
for p in (REPO_ROOT / "scripts" / "tests").glob("test_*.py")
|
||||
}
|
||||
)
|
||||
|
||||
callers = list(WORKFLOWS_DIR.glob("*.yml")) + list(HOOKS_DIR.glob("*.sh"))
|
||||
callers += [p for p in HUSKY_DIR.iterdir() if p.is_file()]
|
||||
@@ -113,13 +114,8 @@ def wired_hook_files() -> set[str]:
|
||||
text = (REPO_ROOT / ".claude" / "settings.json").read_text()
|
||||
for husky in HUSKY_DIR.iterdir():
|
||||
if husky.is_file():
|
||||
text += "\n".join(
|
||||
line for line in husky.read_text().splitlines()
|
||||
if not line.lstrip().startswith("#")
|
||||
)
|
||||
return {
|
||||
str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text
|
||||
}
|
||||
text += "\n".join(line for line in husky.read_text().splitlines() if not line.lstrip().startswith("#"))
|
||||
return {str(p.relative_to(REPO_ROOT)) for p in HOOKS_DIR.glob("*.sh") if p.name in text}
|
||||
|
||||
|
||||
def inventory_rows() -> list[tuple[str, str, str, str]]:
|
||||
@@ -235,9 +231,7 @@ def test_proof_rows_do_not_themselves_claim_a_proof():
|
||||
for guard, kind, proof, ref in inventory_rows():
|
||||
if kind == "PROOF":
|
||||
assert proof == "NONE", f"{guard} is PROOF but claims Proof {proof}"
|
||||
assert guard.startswith("scripts/tests/"), (
|
||||
f"{guard} is marked PROOF but does not live in scripts/tests/"
|
||||
)
|
||||
assert guard.startswith("scripts/tests/"), f"{guard} is marked PROOF but does not live in scripts/tests/"
|
||||
assert ref in ("—", "-", ""), f"{guard}: PROOF rows carry no proof ref"
|
||||
|
||||
|
||||
@@ -283,8 +277,12 @@ def test_the_summary_counts_match_the_table():
|
||||
)
|
||||
claimed = tuple(int(g) for g in m.groups())
|
||||
actual = (
|
||||
kinds["GUARD"], kinds["TOOLING"], kinds["PROOF"],
|
||||
grades["MUTATION"], grades["BEHAVIOUR-ONLY"], grades["NONE"],
|
||||
kinds["GUARD"],
|
||||
kinds["TOOLING"],
|
||||
kinds["PROOF"],
|
||||
grades["MUTATION"],
|
||||
grades["BEHAVIOUR-ONLY"],
|
||||
grades["NONE"],
|
||||
)
|
||||
assert claimed == actual, (
|
||||
f"the summary claims (guards, tooling, proofs, mutation, behaviour-only, none) = {claimed} "
|
||||
|
||||
+322
-204
@@ -27,9 +27,9 @@ that is fine alone and lethal in context.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import pty
|
||||
import re
|
||||
import subprocess
|
||||
import pty
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -103,9 +103,7 @@ def instrumentation_faults(text: str, name: str) -> list[str]:
|
||||
if not mode:
|
||||
faults.append(f"{name}: etv_hook_fire_begin names no stdout mode")
|
||||
elif mode.group(1) != want:
|
||||
faults.append(
|
||||
f"{name}: begins in {mode.group(1)!r} mode but its wiring implies {want!r}"
|
||||
)
|
||||
faults.append(f"{name}: begins in {mode.group(1)!r} mode but its wiring implies {want!r}")
|
||||
|
||||
# Order is the whole point: 8 of the hooks slurp stdin with `input=$(cat)`, and a begin
|
||||
# placed after that read would find the pipe already drained — recording a fire with no
|
||||
@@ -133,8 +131,9 @@ def strip_instrumentation(text: str) -> str:
|
||||
out.pop()
|
||||
continue
|
||||
if skipping:
|
||||
if line.startswith(("#", "ETV_HOOK_FIRE_LIB=", "[ -r ", "type etv_hook_fire_begin",
|
||||
"etv_hook_fire_begin ")):
|
||||
if line.startswith(
|
||||
("#", "ETV_HOOK_FIRE_LIB=", "[ -r ", "type etv_hook_fire_begin", "etv_hook_fire_begin ")
|
||||
):
|
||||
continue
|
||||
skipping = False
|
||||
out.append(line)
|
||||
@@ -173,8 +172,15 @@ def test_the_stripper_removes_EXACTLY_the_preamble_and_nothing_else():
|
||||
"""
|
||||
import difflib
|
||||
|
||||
allowed = ("# ersatztv#776", "# git hook:", "# Claude hook:", "ETV_HOOK_FIRE_LIB=",
|
||||
"[ -r \"$ETV_HOOK_FIRE_LIB\"", "type etv_hook_fire_begin", "etv_hook_fire_begin ")
|
||||
allowed = (
|
||||
"# ersatztv#776",
|
||||
"# git hook:",
|
||||
"# Claude hook:",
|
||||
"ETV_HOOK_FIRE_LIB=",
|
||||
'[ -r "$ETV_HOOK_FIRE_LIB"',
|
||||
"type etv_hook_fire_begin",
|
||||
"etv_hook_fire_begin ",
|
||||
)
|
||||
for hook in hook_scripts():
|
||||
cur = hook.read_text().splitlines(keepends=True)
|
||||
stripped = strip_instrumentation("".join(cur)).splitlines(keepends=True)
|
||||
@@ -189,7 +195,6 @@ def test_the_stripper_removes_EXACTLY_the_preamble_and_nothing_else():
|
||||
)
|
||||
|
||||
|
||||
|
||||
def test_the_stripper_actually_strips():
|
||||
"""The A/B control and the mutation are the same function. If it were a no-op, the differential
|
||||
test would compare each hook against itself and pass on a wrapper that breaks everything."""
|
||||
@@ -211,8 +216,9 @@ def test_every_hook_reports_that_it_fired():
|
||||
for hook in hook_scripts():
|
||||
faults += instrumentation_faults(hook.read_text(), hook.stem)
|
||||
assert not faults, (
|
||||
"these hooks do not report their own execution:\n " + "\n ".join(faults) +
|
||||
"\n\nAdd the three-line preamble after the `set -` line and BEFORE any stdin read. A hook "
|
||||
"these hooks do not report their own execution:\n "
|
||||
+ "\n ".join(faults)
|
||||
+ "\n\nAdd the three-line preamble after the `set -` line and BEFORE any stdin read. A hook "
|
||||
"that does not report is one whose firing we can only infer, which is ersatztv#776."
|
||||
)
|
||||
|
||||
@@ -235,7 +241,7 @@ def test_a_hook_that_LOSES_its_instrumentation_is_DETECTED():
|
||||
|
||||
def test_begin_placed_AFTER_the_stdin_read_is_DETECTED():
|
||||
"""The subtler mutation: present but too late. Ordering is the property that makes it work."""
|
||||
late = 'set -euo pipefail\ninput=$(cat)\nETV_HOOK_FIRE_LIB="x"\n[ -r "$ETV_HOOK_FIRE_LIB" ] && . "$ETV_HOOK_FIRE_LIB" || true\netv_hook_fire_begin demo "" capture || true\n'
|
||||
late = 'set -euo pipefail\ninput=$(cat)\nETV_HOOK_FIRE_LIB="x"\n[ -r "$ETV_HOOK_FIRE_LIB" ] && . "$ETV_HOOK_FIRE_LIB" || true\netv_hook_fire_begin demo "" capture || true\n' # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
faults = instrumentation_faults(late, "demo")
|
||||
assert any("reads stdin before" in f for f in faults), (
|
||||
"a begin call placed after `input=$(cat)` was accepted. It would record a fire with no "
|
||||
@@ -257,15 +263,15 @@ def test_begin_placed_AFTER_the_stdin_read_is_DETECTED():
|
||||
# issue `deny`. An A/B over silent allows proves transparency on the one path where there is nothing
|
||||
# to be transparent about.
|
||||
PAYLOADS = {
|
||||
"bash-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ETV_UPDATE_GOLDENS=1 dotnet test"}}',
|
||||
"bash-allow": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ls -la"}}',
|
||||
"git-commit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git commit -m x"}}',
|
||||
"bash-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ETV_UPDATE_GOLDENS=1 dotnet test"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"bash-allow": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"ls -la"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"git-commit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git commit -m x"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"nav-deny": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__x__navigate","cwd":"%(cwd)s","tool_input":{"url":"http://h/iptv/channels.m3u"}}',
|
||||
"agent-no-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p"}}',
|
||||
"agent-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p","model":"sonnet"}}',
|
||||
"merge": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__gitea__pull_request_write","cwd":"%(cwd)s","tool_input":{"method":"merge","owner":"timothy","repo":"ersatztv","pull_number":1}}',
|
||||
"worktree-add": '{"session_id":"s","hook_event_name":"PostToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git worktree add /tmp/nope-%(nonce)s HEAD"}}',
|
||||
"ui-edit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Edit","cwd":"%(cwd)s","tool_input":{"file_path":"web/src/screens/Channels.tsx"}}',
|
||||
"agent-no-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"agent-model": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Agent","cwd":"%(cwd)s","tool_input":{"description":"d","prompt":"p","model":"sonnet"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"merge": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"mcp__gitea__pull_request_write","cwd":"%(cwd)s","tool_input":{"method":"merge","owner":"timothy","repo":"ersatztv","pull_number":1}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"worktree-add": '{"session_id":"s","hook_event_name":"PostToolUse","tool_name":"Bash","cwd":"%(cwd)s","tool_input":{"command":"git worktree add /tmp/nope-%(nonce)s HEAD"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"ui-edit": '{"session_id":"s","hook_event_name":"PreToolUse","tool_name":"Edit","cwd":"%(cwd)s","tool_input":{"file_path":"web/src/screens/Channels.tsx"}}', # noqa: E501 - kept on one line so the fixture is greppable against the payload the hook receives
|
||||
"empty": "",
|
||||
"garbage": "not json at all",
|
||||
}
|
||||
@@ -300,17 +306,23 @@ def sandbox(tmp_path_factory):
|
||||
|
||||
def _run(script: Path, payload: str, env: dict, cwd: Path, arg: str | None = None):
|
||||
cmd = ["bash", str(script)] + ([arg] if arg else [])
|
||||
p = subprocess.run(
|
||||
cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=env, timeout=60
|
||||
)
|
||||
p = subprocess.run(cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=env, timeout=60)
|
||||
return p.returncode, p.stdout
|
||||
|
||||
|
||||
def _git(cwd: Path, *args: str) -> None:
|
||||
subprocess.run(
|
||||
["git", *args], cwd=str(cwd), check=True, capture_output=True,
|
||||
env={**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@e",
|
||||
"GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@e"},
|
||||
["git", *args],
|
||||
cwd=str(cwd),
|
||||
check=True,
|
||||
capture_output=True,
|
||||
env={
|
||||
**os.environ,
|
||||
"GIT_AUTHOR_NAME": "t",
|
||||
"GIT_AUTHOR_EMAIL": "t@e",
|
||||
"GIT_COMMITTER_NAME": "t",
|
||||
"GIT_COMMITTER_EMAIL": "t@e",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -328,18 +340,23 @@ def positives(sandbox, tmp_path_factory):
|
||||
|
||||
def pay(**kw) -> str:
|
||||
import json as _json
|
||||
return _json.dumps({"session_id": "me", "hook_event_name": "PreToolUse",
|
||||
"tool_name": "Bash", **kw})
|
||||
|
||||
return _json.dumps({"session_id": "me", "hook_event_name": "PreToolUse", "tool_name": "Bash", **kw})
|
||||
|
||||
# --- worktree-guard: a marker naming ANOTHER session, on a `git commit` -------------------
|
||||
wt = w / "sibling"
|
||||
wt.mkdir()
|
||||
_git(wt, "init", "-q", ".")
|
||||
(wt / ".claude-worktree-owner").write_text("SOME-OTHER-SESSION\n")
|
||||
cases["pretooluse-worktree-guard"] = [(
|
||||
"foreign-marker-deny",
|
||||
pay(cwd=str(wt), tool_input={"command": "git commit -m x"}), None, base_env, wt,
|
||||
)]
|
||||
cases["pretooluse-worktree-guard"] = [
|
||||
(
|
||||
"foreign-marker-deny",
|
||||
pay(cwd=str(wt), tool_input={"command": "git commit -m x"}),
|
||||
None,
|
||||
base_env,
|
||||
wt,
|
||||
)
|
||||
]
|
||||
|
||||
# --- bom-guard: repo path must match `*ersatztv*`, with a staged BOM-carrying .cs ----------
|
||||
br = w / "ersatztv-scratch"
|
||||
@@ -347,10 +364,15 @@ def positives(sandbox, tmp_path_factory):
|
||||
_git(br, "init", "-q", ".")
|
||||
(br / "Bad.cs").write_bytes(b"\xef\xbb\xbfclass A {}\n")
|
||||
_git(br, "add", "Bad.cs")
|
||||
cases["pretooluse-bom-guard"] = [(
|
||||
"staged-bom-deny",
|
||||
pay(cwd=str(br), tool_input={"command": "git commit -m x"}), None, base_env, br,
|
||||
)]
|
||||
cases["pretooluse-bom-guard"] = [
|
||||
(
|
||||
"staged-bom-deny",
|
||||
pay(cwd=str(br), tool_input={"command": "git commit -m x"}),
|
||||
None,
|
||||
base_env,
|
||||
br,
|
||||
)
|
||||
]
|
||||
|
||||
# --- agent-ram: both thresholds, via a stubbed `memory_pressure` ---------------------------
|
||||
ram_cases = []
|
||||
@@ -362,11 +384,15 @@ def positives(sandbox, tmp_path_factory):
|
||||
stub.chmod(0o755)
|
||||
env = dict(base_env)
|
||||
env["PATH"] = f"{bindir}:{os.environ['PATH']}"
|
||||
ram_cases.append((
|
||||
f"free-{pct}pct-{label}",
|
||||
pay(tool_name="Agent", cwd=str(w), tool_input={"prompt": "p", "model": "sonnet"}),
|
||||
None, env, w,
|
||||
))
|
||||
ram_cases.append(
|
||||
(
|
||||
f"free-{pct}pct-{label}",
|
||||
pay(tool_name="Agent", cwd=str(w), tool_input={"prompt": "p", "model": "sonnet"}),
|
||||
None,
|
||||
env,
|
||||
w,
|
||||
)
|
||||
)
|
||||
cases["pretooluse-agent-ram"] = ram_cases
|
||||
|
||||
# --- decisions-guard: a validator that prints and blocks, and one that prints and passes ---
|
||||
@@ -409,12 +435,16 @@ def positives(sandbox, tmp_path_factory):
|
||||
_git(ahead, "add", "-A")
|
||||
_git(ahead, "commit", "-qm", "advance main")
|
||||
_git(ahead, "push", "-q", "origin", "main")
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(behind),
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
cases["prepush-rebase-check"] = [(
|
||||
"behind-origin-main-blocks",
|
||||
f"refs/heads/feat {head} refs/heads/feat {'0' * 40}\n", None, base_env, behind,
|
||||
)]
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(behind), capture_output=True, text=True).stdout.strip()
|
||||
cases["prepush-rebase-check"] = [
|
||||
(
|
||||
"behind-origin-main-blocks",
|
||||
f"refs/heads/feat {head} refs/heads/feat {'0' * 40}\n",
|
||||
None,
|
||||
base_env,
|
||||
behind,
|
||||
)
|
||||
]
|
||||
|
||||
# clean-worktree-check: a file both MODIFIED in the tree and present in the pushed set.
|
||||
dirty = clone("dirty")
|
||||
@@ -422,13 +452,16 @@ def positives(sandbox, tmp_path_factory):
|
||||
_git(dirty, "add", "-A")
|
||||
_git(dirty, "commit", "-qm", "change file")
|
||||
(dirty / "file.txt").write_text("uncommitted change\n")
|
||||
dhead = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dirty),
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
cases["prepush-clean-worktree-check"] = [(
|
||||
"dirty-file-in-pushed-set-blocks",
|
||||
f"refs/heads/main {dhead} refs/heads/main {'0' * 40}\n", None, base_env, dirty,
|
||||
)]
|
||||
|
||||
dhead = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dirty), capture_output=True, text=True).stdout.strip()
|
||||
cases["prepush-clean-worktree-check"] = [
|
||||
(
|
||||
"dirty-file-in-pushed-set-blocks",
|
||||
f"refs/heads/main {dhead} refs/heads/main {'0' * 40}\n",
|
||||
None,
|
||||
base_env,
|
||||
dirty,
|
||||
)
|
||||
]
|
||||
|
||||
# --- prepush-donewhen: a push to main closing an issue with an unticked Done-when box -------
|
||||
#
|
||||
@@ -443,7 +476,7 @@ def positives(sandbox, tmp_path_factory):
|
||||
body = _json.dumps({"body": "## Done-when\n\n- [ ] not finished\n- [x] finished\n"}).encode()
|
||||
|
||||
class _Stub(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self): # noqa: N802
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
@@ -457,20 +490,23 @@ def positives(sandbox, tmp_path_factory):
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
|
||||
dw = clone("donewhen")
|
||||
(dw / "src.cs").write_text("class A {}\n") # NOT docs-only, so the exemption does not apply
|
||||
(dw / "src.cs").write_text("class A {}\n") # NOT docs-only, so the exemption does not apply
|
||||
_git(dw, "add", "-A")
|
||||
_git(dw, "commit", "-qm", "feat: thing\n\nfixes #999")
|
||||
dw_head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dw),
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
dw_prev = subprocess.run(["git", "rev-parse", "HEAD~1"], cwd=str(dw),
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
dw_head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=str(dw), capture_output=True, text=True).stdout.strip()
|
||||
dw_prev = subprocess.run(["git", "rev-parse", "HEAD~1"], cwd=str(dw), capture_output=True, text=True).stdout.strip()
|
||||
dw_env = dict(base_env)
|
||||
dw_env["ETV_GITEA_URL"] = f"http://127.0.0.1:{srv.server_address[1]}"
|
||||
dw_env["ETV_GITEA_BASICAUTH"] = "stub:stub"
|
||||
cases["prepush-donewhen"] = [(
|
||||
"unticked-donewhen-blocks-push-to-main",
|
||||
f"refs/heads/main {dw_head} refs/heads/main {dw_prev}\n", None, dw_env, dw,
|
||||
)]
|
||||
cases["prepush-donewhen"] = [
|
||||
(
|
||||
"unticked-donewhen-blocks-push-to-main",
|
||||
f"refs/heads/main {dw_head} refs/heads/main {dw_prev}\n",
|
||||
None,
|
||||
dw_env,
|
||||
dw,
|
||||
)
|
||||
]
|
||||
|
||||
return cases
|
||||
|
||||
@@ -482,10 +518,15 @@ def _ab_cases(hook: Path, sandbox, positives) -> list[tuple]:
|
||||
args = [None, "start", "finish"] if hook.stem == "design-sync-reminder" else [None]
|
||||
for name, template in PAYLOADS.items():
|
||||
for arg in args:
|
||||
cases.append((
|
||||
f"matrix:{name}:{arg}", template % {"cwd": str(root), "nonce": name},
|
||||
arg, env, root,
|
||||
))
|
||||
cases.append(
|
||||
(
|
||||
f"matrix:{name}:{arg}",
|
||||
template % {"cwd": str(root), "nonce": name},
|
||||
arg,
|
||||
env,
|
||||
root,
|
||||
)
|
||||
)
|
||||
for label, payload, arg, penv, cwd in positives.get(hook.stem, []):
|
||||
cases.append((f"positive:{label}", payload, arg, penv, cwd))
|
||||
return cases
|
||||
@@ -513,8 +554,7 @@ def _ab_run(hook: Path, sandbox, case, tag_root: str):
|
||||
|
||||
def one(script: Path, t: str):
|
||||
cmd = ["bash", str(script)] + ([arg] if arg else [])
|
||||
p = subprocess.run(cmd, input=payload.encode(), capture_output=True,
|
||||
cwd=str(cwd), env=side(t), timeout=90)
|
||||
p = subprocess.run(cmd, input=payload.encode(), capture_output=True, cwd=str(cwd), env=side(t), timeout=90)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
|
||||
return one(control, "control"), one(hook, "instrumented")
|
||||
@@ -613,11 +653,16 @@ def test_the_worktree_markers_SIDE_EFFECT_is_unchanged(sandbox, tmp_path):
|
||||
script = tmp_path / "control-marker.sh"
|
||||
script.write_text(strip_instrumentation(hook.read_text()))
|
||||
import json as _json
|
||||
payload = _json.dumps({
|
||||
"session_id": "SESSION-XYZ", "hook_event_name": "PostToolUse", "tool_name": "Bash",
|
||||
"cwd": str(repo),
|
||||
"tool_input": {"command": f"git worktree add {target} HEAD"},
|
||||
})
|
||||
|
||||
payload = _json.dumps(
|
||||
{
|
||||
"session_id": "SESSION-XYZ",
|
||||
"hook_event_name": "PostToolUse",
|
||||
"tool_name": "Bash",
|
||||
"cwd": str(repo),
|
||||
"tool_input": {"command": f"git worktree add {target} HEAD"},
|
||||
}
|
||||
)
|
||||
_git(repo, "worktree", "add", "-q", str(target), "HEAD")
|
||||
_run(script, payload, env, repo)
|
||||
marker = target / ".claude-worktree-owner"
|
||||
@@ -641,15 +686,13 @@ def test_stdout_is_replayed_BYTE_EXACT(sandbox):
|
||||
hook.write_text(
|
||||
"#!/usr/bin/env bash\nset -euo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
"etv_hook_fire_begin trailing \"\" capture || true\n"
|
||||
'etv_hook_fire_begin trailing "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}\n\n"' + "\n"
|
||||
)
|
||||
rc, out = _run(hook, '{"tool_name":"Bash"}', env, root)
|
||||
assert rc == 0
|
||||
assert out == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}\n\n', (
|
||||
f"trailing bytes were altered: {out!r}"
|
||||
)
|
||||
assert out == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}\n\n', f"trailing bytes were altered: {out!r}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("code", [0, 1, 2, 3])
|
||||
@@ -704,8 +747,12 @@ def test_a_TTY_stdin_is_not_slurped(sandbox):
|
||||
master, slave = pty.openpty()
|
||||
try:
|
||||
p = subprocess.Popen(
|
||||
["bash", str(hook)], stdin=slave, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL, cwd=str(root), env=env,
|
||||
["bash", str(hook)],
|
||||
stdin=slave,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
cwd=str(root),
|
||||
env=env,
|
||||
)
|
||||
os.close(slave)
|
||||
try:
|
||||
@@ -751,11 +798,11 @@ def test_logging_failure_does_not_break_the_hook(sandbox):
|
||||
@pytest.mark.parametrize(
|
||||
"emit,mode,expected",
|
||||
[
|
||||
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}', "capture", "deny"),
|
||||
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"ask\"}}', "capture", "ask"),
|
||||
(r'{\"hookSpecificOutput\":{\"permissionDecision\":\"allow\"}}', "capture", "allow"),
|
||||
(r'{\"hookSpecificOutput\":{\"additionalContext\":\"hi\"}}', "capture", "context"),
|
||||
(r'{\"decision\":\"block\",\"reason\":\"r\"}', "capture", "block"),
|
||||
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}", "capture", "deny"),
|
||||
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"ask\"}}", "capture", "ask"),
|
||||
(r"{\"hookSpecificOutput\":{\"permissionDecision\":\"allow\"}}", "capture", "allow"),
|
||||
(r"{\"hookSpecificOutput\":{\"additionalContext\":\"hi\"}}", "capture", "context"),
|
||||
(r"{\"decision\":\"block\",\"reason\":\"r\"}", "capture", "block"),
|
||||
("", "capture", "no-op"),
|
||||
("", "stream", "pass"),
|
||||
],
|
||||
@@ -822,7 +869,10 @@ def test_the_report_names_hooks_that_NEVER_fired(sandbox):
|
||||
)
|
||||
p = subprocess.run(
|
||||
["bash", str(SINK), "report", "--all", "--dir", str(logdir)],
|
||||
capture_output=True, cwd=str(REPO_ROOT), env=env, timeout=60,
|
||||
capture_output=True,
|
||||
cwd=str(REPO_ROOT),
|
||||
env=env,
|
||||
timeout=60,
|
||||
)
|
||||
out = p.stdout.decode()
|
||||
assert p.returncode == 0, p.stderr.decode()
|
||||
@@ -859,7 +909,10 @@ def test_the_report_REFUSES_an_empty_population(tmp_path, sandbox):
|
||||
runenv.pop("CLAUDE_PROJECT_DIR", None)
|
||||
p = subprocess.run(
|
||||
["bash", str(copied), "report", "--all", "--dir", str(tmp_path / "log")],
|
||||
capture_output=True, cwd=str(tmp_path), env=runenv, timeout=60,
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env=runenv,
|
||||
timeout=60,
|
||||
)
|
||||
assert p.returncode == 2, (
|
||||
"the report exited 0 over an empty hook population. It must refuse rather than print a "
|
||||
@@ -895,12 +948,11 @@ def test_stderr_is_NOT_silenced(sandbox):
|
||||
'printf "IMPORTANT DIAGNOSTIC\\n" >&2\n'
|
||||
"exit 1\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root), env=env, timeout=60)
|
||||
assert p.returncode == 1
|
||||
assert b"IMPORTANT DIAGNOSTIC" in p.stderr, (
|
||||
f"the hook's stderr was swallowed by the instrumentation: {p.stderr!r}"
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True, cwd=str(root), env=env, timeout=60
|
||||
)
|
||||
assert p.returncode == 1
|
||||
assert b"IMPORTANT DIAGNOSTIC" in p.stderr, f"the hook's stderr was swallowed by the instrumentation: {p.stderr!r}"
|
||||
|
||||
|
||||
def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
|
||||
@@ -923,8 +975,14 @@ def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
|
||||
)
|
||||
tmp = root / "vanish-tmp"
|
||||
tmp.mkdir(exist_ok=True)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
|
||||
cwd=str(root), env={**env, "TMPDIR": str(tmp)}, timeout=60)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "TMPDIR": str(tmp)},
|
||||
timeout=60,
|
||||
)
|
||||
assert p.returncode == 0
|
||||
assert b'"permissionDecision":"deny"' in p.stdout, (
|
||||
"the guard's deny was DISCARDED when its stdout scratch file vanished mid-run. The hook "
|
||||
@@ -932,24 +990,22 @@ def test_output_SURVIVES_a_vanished_stdout_tempfile(sandbox):
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"emit,code,expected",
|
||||
[
|
||||
# A NON-CANONICAL value is recorded as non-canonical, not laundered into a valid decision.
|
||||
# This case went both ways before settling: first filed as unclassified prose, then
|
||||
# lowercased into a clean `deny` (manufacturing a decision the harness may never honour).
|
||||
(r'{\"permissionDecision\":\"Deny\"}', 0, "unrecognized"),
|
||||
(r"{\"permissionDecision\":\"Deny\"}", 0, "unrecognized"),
|
||||
# A non-zero status neither ERASES a printed decision nor annotates it. The status has its
|
||||
# own field; the decision field states the decision and nothing else.
|
||||
(r'{\"permissionDecision\":\"deny\"}', 1, "deny"),
|
||||
(r"{\"permissionDecision\":\"deny\"}", 1, "deny"),
|
||||
# A failure with nothing classifiable IS an error.
|
||||
("", 1, "error"),
|
||||
# Exit 2 is the harness's block channel and DOMINATES the printed JSON. Recording the
|
||||
# printed `allow` would report a permit for a call that was actually refused — the one
|
||||
# direction a log of security decisions must never be wrong in.
|
||||
(r'{\"permissionDecision\":\"allow\"}', 2, "deny-exit2"),
|
||||
(r"{\"permissionDecision\":\"allow\"}", 2, "deny-exit2"),
|
||||
],
|
||||
)
|
||||
def test_the_classifier_does_not_MISREPORT(sandbox, emit, code, expected):
|
||||
@@ -961,12 +1017,16 @@ def test_the_classifier_does_not_MISREPORT(sandbox, emit, code, expected):
|
||||
"#!/usr/bin/env bash\nset -uo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin cls "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
+ (f'printf "{emit}"\n' if emit else "")
|
||||
+ f"exit {code}\n"
|
||||
"input=$(cat)\n" + (f'printf "{emit}"\n' if emit else "") + f"exit {code}\n"
|
||||
)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env=runenv,
|
||||
timeout=60,
|
||||
)
|
||||
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root), env=runenv, timeout=60)
|
||||
rec = [ln for ln in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in ln]
|
||||
assert f'"decision":"{expected}"' in rec[0], f"expected {expected}, got {rec[0]}"
|
||||
|
||||
@@ -983,9 +1043,14 @@ def test_the_word_additionalContext_in_PROSE_is_not_a_decision(sandbox):
|
||||
"input=$(cat)\n"
|
||||
'printf "note: this hook does not use additionalContext at all\\n"\n'
|
||||
)
|
||||
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
rec = [ln for ln in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in ln]
|
||||
assert '"decision":"output"' in rec[0], f"prose was classified as a decision: {rec[0]}"
|
||||
|
||||
@@ -1001,14 +1066,17 @@ def test_the_session_id_cannot_ESCAPE_the_log_directory(sandbox, tmp_path):
|
||||
logdir = tmp_path / "logs"
|
||||
hook = tmp_path / "esc.sh"
|
||||
hook.write_text(
|
||||
"#!/usr/bin/env bash\nset -uo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin esc "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin esc "" capture || true\ninput=$(cat)\n'
|
||||
)
|
||||
evil = '{"session_id":"../../escaped","tool_name":"Bash"}'
|
||||
subprocess.run(["bash", str(hook)], input=evil.encode(), capture_output=True,
|
||||
cwd=str(tmp_path), env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=evil.encode(),
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
written = list(logdir.glob("*.jsonl"))
|
||||
assert written, "nothing was logged at all"
|
||||
for f in written:
|
||||
@@ -1040,8 +1108,14 @@ def test_the_suite_does_not_write_to_the_PRODUCTION_log(sandbox):
|
||||
'etv_hook_fire_begin prodcheck "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
)
|
||||
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root), env=env, timeout=60)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env=env,
|
||||
timeout=60,
|
||||
)
|
||||
after = sorted(p.stat().st_mtime_ns for p in default.glob("*.jsonl")) if default.exists() else []
|
||||
assert before == after, "a test run modified the production hook-fire log"
|
||||
|
||||
@@ -1075,16 +1149,17 @@ def test_DELETING_the_replay_makes_the_differential_go_RED(sandbox, positives, t
|
||||
"input=$(cat)\n"
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
|
||||
cwd=str(root), env={**env, "TMPDIR": str(tmp_path)}, timeout=60)
|
||||
assert p.stdout == b"", (
|
||||
"the mutation did not actually disarm the replay, so this proof establishes nothing: "
|
||||
f"stdout={p.stdout!r}"
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "TMPDIR": str(tmp_path)},
|
||||
timeout=60,
|
||||
)
|
||||
assert p.stdout == b"", (
|
||||
f"the mutation did not actually disarm the replay, so this proof establishes nothing: stdout={p.stdout!r}"
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
def test_no_exec_in_the_sink_carries_a_STDERR_REDIRECT():
|
||||
@@ -1123,17 +1198,20 @@ def test_the_log_and_the_REPLAY_agree_when_the_hook_uses_fd_4(sandbox):
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin fd4 "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
'exec 4</dev/null\n' # the hook takes fd 4 for itself
|
||||
"exec 4</dev/null\n" # the hook takes fd 4 for itself
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
assert b'"permissionDecision":"deny"' in p.stdout, f"replay lost the decision: {p.stdout!r}"
|
||||
exits = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
|
||||
assert '"decision":"deny"' in exits[0], (
|
||||
f"the harness saw a deny but the log recorded something else: {exits[0]}"
|
||||
)
|
||||
assert '"decision":"deny"' in exits[0], f"the harness saw a deny but the log recorded something else: {exits[0]}"
|
||||
|
||||
|
||||
def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
|
||||
@@ -1160,9 +1238,14 @@ def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
|
||||
"input=$(cat)\n"
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(tmp_path),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
assert p.returncode == 0
|
||||
assert p.stdout == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}'
|
||||
assert p.stderr == b"", (
|
||||
@@ -1171,10 +1254,6 @@ def test_an_UNWRITABLE_log_file_is_still_stderr_transparent(sandbox, tmp_path):
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
def test_NUL_bytes_in_hook_output_survive(sandbox):
|
||||
"""A shell variable cannot hold a NUL, so replaying through `$(...)` silently drops them.
|
||||
|
||||
@@ -1192,8 +1271,9 @@ def test_NUL_bytes_in_hook_output_survive(sandbox):
|
||||
"input=$(cat)\n"
|
||||
r"printf 'a\000b\n'" + "\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True,
|
||||
cwd=str(root), env=env, timeout=60)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)], input=b'{"tool_name":"Bash"}', capture_output=True, cwd=str(root), env=env, timeout=60
|
||||
)
|
||||
assert p.stdout == b"a\x00b\n", f"NUL-containing output was mangled: {p.stdout!r}"
|
||||
assert b"null byte" not in p.stderr, f"a warning leaked to the harness: {p.stderr!r}"
|
||||
|
||||
@@ -1202,8 +1282,8 @@ def test_NUL_bytes_in_hook_output_survive(sandbox):
|
||||
"emit,code,expected",
|
||||
[
|
||||
# Every non-canonical value, not just the ones a restricted character class admits.
|
||||
(r'{\"permissionDecision\":\"deny2\"}', 0, "unrecognized"),
|
||||
(r'{\"permissionDecision\":\"deny_now\"}', 0, "unrecognized"),
|
||||
(r"{\"permissionDecision\":\"deny2\"}", 0, "unrecognized"),
|
||||
(r"{\"permissionDecision\":\"deny_now\"}", 0, "unrecognized"),
|
||||
# Unclassified output plus a FAILING exit is an error, not `output`: the report histograms
|
||||
# the decision, so filing it as `output` hid the failure entirely.
|
||||
("diagnostic text", 1, "error"),
|
||||
@@ -1217,13 +1297,16 @@ def test_odd_values_and_failing_exits_are_not_LAUNDERED(sandbox, emit, code, exp
|
||||
"#!/usr/bin/env bash\nset -uo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin odd "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
+ (f'printf "{emit}"\n' if emit else "")
|
||||
+ f"exit {code}\n"
|
||||
"input=$(cat)\n" + (f'printf "{emit}"\n' if emit else "") + f"exit {code}\n"
|
||||
)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
rec = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
|
||||
assert f'"decision":"{expected}"' in rec[0], f"expected {expected}, got {rec[0]}"
|
||||
|
||||
@@ -1247,9 +1330,12 @@ def test_an_INHERITED_flushed_flag_does_not_disable_reporting(sandbox):
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"deny\"}}"' + "\n"
|
||||
)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}', capture_output=True,
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir), "ETV_HOOK_FIRE_FLUSHED": "1"}, timeout=60,
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir), "ETV_HOOK_FIRE_FLUSHED": "1"},
|
||||
timeout=60,
|
||||
)
|
||||
assert p.stdout == b'{"hookSpecificOutput":{"permissionDecision":"deny"}}', (
|
||||
f"an inherited FLUSHED=1 left stdout redirected and swallowed the decision: {p.stdout!r}"
|
||||
@@ -1284,9 +1370,7 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
|
||||
)
|
||||
instrumented = root / f"sigab-{signame}.sh"
|
||||
instrumented.write_text(
|
||||
"#!/usr/bin/env bash\nset -uo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
f'etv_hook_fire_begin sigab "" stream || true\n' + body
|
||||
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin sigab "" stream || true\n' + body
|
||||
)
|
||||
control = root / f"sigab-control-{signame}.sh"
|
||||
control.write_text("#!/usr/bin/env bash\nset -uo pipefail\n" + body)
|
||||
@@ -1302,9 +1386,15 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
|
||||
# anything. Measured: pid-only gives control 4.0s and trapped 4.1s; group signalling
|
||||
# gives control 0.002s and trapped 0.054s. A supervisor kills the group, so this is
|
||||
# also the shape that actually occurs.
|
||||
p = subprocess.Popen(["bash", str(script)], stdin=fh, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, cwd=str(root), env=env,
|
||||
start_new_session=True)
|
||||
p = subprocess.Popen(
|
||||
["bash", str(script)],
|
||||
stdin=fh,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
cwd=str(root),
|
||||
env=env,
|
||||
start_new_session=True,
|
||||
)
|
||||
time.sleep(1.0)
|
||||
t0 = time.monotonic()
|
||||
os.killpg(os.getpgid(p.pid), sig)
|
||||
@@ -1312,7 +1402,8 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
|
||||
results[tag] = (p.returncode, time.monotonic() - t0, out, err)
|
||||
|
||||
(rc_a, dt_a, out_a, err_a), (rc_b, dt_b, out_b, err_b) = (
|
||||
results["control"], results["instrumented"],
|
||||
results["control"],
|
||||
results["instrumented"],
|
||||
)
|
||||
assert out_a == out_b, f"SIG{signame}: stdout differs under signal: {out_a!r} vs {out_b!r}"
|
||||
|
||||
@@ -1329,13 +1420,11 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
|
||||
# require one of bash's four signal words followed by `:` or whitespace, so an ordinary
|
||||
# diagnostic still fails the comparison.
|
||||
return b"\n".join(
|
||||
ln for ln in e.split(b"\n")
|
||||
if not re.search(rb"(^|\s)(Terminated|Hangup|Interrupt|Killed)(:|\s|$)", ln)
|
||||
ln for ln in e.split(b"\n") if not re.search(rb"(^|\s)(Terminated|Hangup|Interrupt|Killed)(:|\s|$)", ln)
|
||||
)
|
||||
|
||||
assert strip_jobnotice(err_a) == strip_jobnotice(err_b), (
|
||||
f"SIG{signame}: stderr differs under signal beyond bash's job-control notice: "
|
||||
f"{err_a!r} vs {err_b!r}"
|
||||
f"SIG{signame}: stderr differs under signal beyond bash's job-control notice: {err_a!r} vs {err_b!r}"
|
||||
)
|
||||
assert rc_a == rc_b, (
|
||||
f"SIG{signame}: exit status differs, control={rc_a} instrumented={rc_b}. git and the "
|
||||
@@ -1348,14 +1437,15 @@ def test_a_SIGNALLED_hook_behaves_EXACTLY_as_an_uninstrumented_one(sandbox, sign
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
@pytest.mark.parametrize("locale_env", [
|
||||
{"LANG": "en_US.UTF-8"},
|
||||
{"LC_CTYPE": "en_US.UTF-8"},
|
||||
{"LC_CTYPE": "UTF-8"}, # macOS Terminal's default, and the case that broke the fix
|
||||
{"LC_ALL": "en_US.UTF-8"},
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"locale_env",
|
||||
[
|
||||
{"LANG": "en_US.UTF-8"},
|
||||
{"LC_CTYPE": "en_US.UTF-8"},
|
||||
{"LC_CTYPE": "UTF-8"}, # macOS Terminal's default, and the case that broke the fix
|
||||
{"LC_ALL": "en_US.UTF-8"},
|
||||
],
|
||||
)
|
||||
def test_an_INVALID_UTF8_byte_in_a_decision_is_still_classified(tmp_path, locale_env):
|
||||
"""`local LC_ALL=C` does not export, so the child `sed`/`tr` never saw it — the fix was INERT.
|
||||
|
||||
@@ -1387,10 +1477,15 @@ def test_an_INVALID_UTF8_byte_in_a_decision_is_still_classified(tmp_path, locale
|
||||
'printf \'{"hookSpecificOutput":{"permissionDecision":"deny",'
|
||||
'"permissionDecisionReason":"caf\\xe9"}}\'\n'
|
||||
)
|
||||
env = {"PATH": os.environ["PATH"], "HOME": os.environ["HOME"],
|
||||
"ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(tmp_path), env=env, timeout=60)
|
||||
env = {"PATH": os.environ["PATH"], "HOME": os.environ["HOME"], "ETV_HOOK_FIRE_LOG_DIR": str(logdir), **locale_env}
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env=env,
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
assert p.stderr == b"", f"{locale_env}: a locale diagnostic leaked to the harness: {p.stderr!r}"
|
||||
record = (logdir / "s1.jsonl").read_text()
|
||||
@@ -1419,16 +1514,24 @@ def test_a_NESTED_identity_field_does_not_outrank_the_TOP_LEVEL_one(tmp_path):
|
||||
"input=$(cat)\n"
|
||||
)
|
||||
import json as _json
|
||||
payload = _json.dumps({
|
||||
"session_id": "TOP-SESSION",
|
||||
"hook_event_name": "PreToolUse",
|
||||
"tool_name": "TopTool",
|
||||
"tool_input": {"filler": "x" * 1000},
|
||||
"tool_response": {"session_id": "NESTED-SESSION", "tool_name": "NestedTool"},
|
||||
})
|
||||
subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
|
||||
cwd=str(tmp_path), env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60)
|
||||
|
||||
payload = _json.dumps(
|
||||
{
|
||||
"session_id": "TOP-SESSION",
|
||||
"hook_event_name": "PreToolUse",
|
||||
"tool_name": "TopTool",
|
||||
"tool_input": {"filler": "x" * 1000},
|
||||
"tool_response": {"session_id": "NESTED-SESSION", "tool_name": "NestedTool"},
|
||||
}
|
||||
)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=payload.encode(),
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
assert (logdir / "TOP-SESSION.jsonl").exists(), (
|
||||
f"records filed under the wrong session: {[f.name for f in logdir.glob('*.jsonl')]}"
|
||||
@@ -1451,9 +1554,14 @@ def test_a_PRESENT_but_empty_decision_is_not_laundered(sandbox):
|
||||
"input=$(cat)\n"
|
||||
r'printf "{\"hookSpecificOutput\":{\"permissionDecision\":\"\"}}"' + "\n"
|
||||
)
|
||||
subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)}, timeout=60)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env={**env, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=60,
|
||||
)
|
||||
rec = [r for r in (logdir / "s1.jsonl").read_text().splitlines() if '"phase":"exit"' in r]
|
||||
assert '"decision":"unrecognized"' in rec[0], f"an empty decision was laundered: {rec[0]}"
|
||||
|
||||
@@ -1473,21 +1581,25 @@ def test_identity_fields_BEYOND_the_fast_path_cap_are_still_found(tmp_path):
|
||||
|
||||
hook = tmp_path / "big.sh"
|
||||
hook.write_text(
|
||||
"#!/usr/bin/env bash\nset -uo pipefail\n"
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin big "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
f'#!/usr/bin/env bash\nset -uo pipefail\n. "{SINK}"\netv_hook_fire_begin big "" capture || true\ninput=$(cat)\n'
|
||||
)
|
||||
logdir = tmp_path / "biglog"
|
||||
payload = _json.dumps({
|
||||
"tool_input": {"filler": "x" * 400_000},
|
||||
"session_id": "TOP-SESSION",
|
||||
"hook_event_name": "PreToolUse",
|
||||
"tool_name": "TopTool",
|
||||
})
|
||||
subprocess.run(["bash", str(hook)], input=payload.encode(), capture_output=True,
|
||||
cwd=str(tmp_path), env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=90)
|
||||
payload = _json.dumps(
|
||||
{
|
||||
"tool_input": {"filler": "x" * 400_000},
|
||||
"session_id": "TOP-SESSION",
|
||||
"hook_event_name": "PreToolUse",
|
||||
"tool_name": "TopTool",
|
||||
}
|
||||
)
|
||||
subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=payload.encode(),
|
||||
capture_output=True,
|
||||
cwd=str(tmp_path),
|
||||
env={**os.environ, "ETV_HOOK_FIRE_LOG_DIR": str(logdir)},
|
||||
timeout=90,
|
||||
)
|
||||
|
||||
assert (logdir / "TOP-SESSION.jsonl").exists(), (
|
||||
"identity beyond the fast-path cap was not found, so the fire filed under "
|
||||
@@ -1510,11 +1622,17 @@ def test_the_field_helper_does_not_LEAK_into_the_hooks_namespace(sandbox):
|
||||
f'. "{SINK}"\n'
|
||||
'etv_hook_fire_begin ns "" capture || true\n'
|
||||
"input=$(cat)\n"
|
||||
'if declare -F _etv_field >/dev/null 2>&1; then echo LEAKED_etv_field; fi\n'
|
||||
'if declare -F etv_hook_fire__field >/dev/null 2>&1; then echo LEAKED_namespaced; fi\n'
|
||||
"if declare -F _etv_field >/dev/null 2>&1; then echo LEAKED_etv_field; fi\n"
|
||||
"if declare -F etv_hook_fire__field >/dev/null 2>&1; then echo LEAKED_namespaced; fi\n"
|
||||
"echo CLEAN\n"
|
||||
)
|
||||
p = subprocess.run(["bash", str(hook)], input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True, cwd=str(root), env=env, timeout=60)
|
||||
p = subprocess.run(
|
||||
["bash", str(hook)],
|
||||
input=b'{"session_id":"s1","tool_name":"Bash"}',
|
||||
capture_output=True,
|
||||
cwd=str(root),
|
||||
env=env,
|
||||
timeout=60,
|
||||
)
|
||||
assert b"LEAKED" not in p.stdout, f"a helper leaked into the hook's namespace: {p.stdout!r}"
|
||||
assert b"CLEAN" in p.stdout
|
||||
|
||||
@@ -51,12 +51,12 @@ def preflight(tmp_path):
|
||||
so every test passed.
|
||||
"""
|
||||
shim = bindir / "jq"
|
||||
body = "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then\n"
|
||||
body = '#!/bin/sh\nif [ "$1" = "--version" ]; then\n'
|
||||
if version_line:
|
||||
body += ' printf "%%s\\n" %s\n' % _shq(version_line)
|
||||
body += f' printf "%s\\n" {_shq(version_line)}\n'
|
||||
if stderr:
|
||||
body += ' printf "%%s\\n" %s >&2\n' % _shq(stderr)
|
||||
body += " exit %d\nfi\nexit 0\n" % exit_code
|
||||
body += f' printf "%s\\n" {_shq(stderr)} >&2\n'
|
||||
body += f" exit {exit_code}\nfi\nexit 0\n"
|
||||
shim.write_text(body)
|
||||
shim.chmod(0o755)
|
||||
|
||||
@@ -73,8 +73,7 @@ def preflight(tmp_path):
|
||||
# but jq itself (`command -v` is a builtin, and bash is invoked by absolute path), so
|
||||
# there is nothing to keep.
|
||||
env["PATH"] = str(bindir)
|
||||
return subprocess.run([BASH, str(SCRIPT), *args],
|
||||
env=env, capture_output=True, text=True)
|
||||
return subprocess.run([BASH, str(SCRIPT), *args], env=env, capture_output=True, text=True)
|
||||
|
||||
def run_bytes(self, *args):
|
||||
"""Same, but WITHOUT text mode.
|
||||
@@ -87,8 +86,7 @@ def preflight(tmp_path):
|
||||
"""
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = str(bindir)
|
||||
return subprocess.run([BASH, str(SCRIPT), *args],
|
||||
env=env, capture_output=True)
|
||||
return subprocess.run([BASH, str(SCRIPT), *args], env=env, capture_output=True)
|
||||
|
||||
return Handle()
|
||||
|
||||
@@ -172,11 +170,15 @@ def test_expect_without_a_value_is_a_usage_error_WITH_output(preflight):
|
||||
# the script exited 0 having asserted NOTHING. That is this script's own stated failure mode,
|
||||
# reproduced inside itself, which is why these cases are pinned rather than left to inspection.
|
||||
|
||||
@pytest.mark.parametrize("version_line", [
|
||||
"jq version 1.6", # some distro wrappers print this form
|
||||
"JQ-1.6",
|
||||
"jq-1.6-dirty",
|
||||
])
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"version_line",
|
||||
[
|
||||
"jq version 1.6", # some distro wrappers print this form
|
||||
"JQ-1.6",
|
||||
"jq-1.6-dirty",
|
||||
],
|
||||
)
|
||||
def test_unusual_but_parseable_version_forms_are_accepted(preflight, version_line):
|
||||
preflight.with_jq(version_line)
|
||||
r = preflight.run()
|
||||
@@ -190,7 +192,8 @@ def test_unparseable_version_fails_CLOSED_rather_than_asserting_nothing(prefligh
|
||||
r = preflight.run()
|
||||
assert r.returncode == 1, (
|
||||
f"{version_line!r} exited {r.returncode}: an unparsed version must never reach — or "
|
||||
"silently skip — the floor assertion")
|
||||
"silently skip — the floor assertion"
|
||||
)
|
||||
assert "could not parse" in r.stderr
|
||||
|
||||
|
||||
@@ -202,20 +205,21 @@ def test_a_jq_that_cannot_START_fails_closed(preflight):
|
||||
discarding the exit status, so that message became the parse input, `2.34` matched, and the floor
|
||||
was certified green on a jq that cannot run at all.
|
||||
"""
|
||||
preflight.with_jq(
|
||||
"", stderr="jq: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found",
|
||||
exit_code=127)
|
||||
preflight.with_jq("", stderr="jq: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found", exit_code=127)
|
||||
r = preflight.run()
|
||||
assert r.returncode == 1
|
||||
assert "cannot run" in r.stderr
|
||||
assert "parsed 2.34" not in r.stdout, "stderr must never be parsed as a version"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version_line", [
|
||||
"warning: something 3.14", # a noise line carrying a plausible number
|
||||
"2026.07.26 jq-1.6", # a date prefix, which outranks the real version if unanchored
|
||||
"jq-master-v0.0.0-1.6",
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"version_line",
|
||||
[
|
||||
"warning: something 3.14", # a noise line carrying a plausible number
|
||||
"2026.07.26 jq-1.6", # a date prefix, which outranks the real version if unanchored
|
||||
"jq-master-v0.0.0-1.6",
|
||||
],
|
||||
)
|
||||
def test_a_number_that_is_not_the_VERSION_is_not_accepted_as_one(preflight, version_line):
|
||||
"""Matching the first `<digits>.<digits>` ANYWHERE let a prefix win over the real version.
|
||||
`2026.07.26 jq-1.6` parsed as 2026.07 and sailed over the floor. The pattern is anchored to the
|
||||
@@ -226,10 +230,13 @@ def test_a_number_that_is_not_the_VERSION_is_not_accepted_as_one(preflight, vers
|
||||
assert "could not parse" in r.stderr
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version_line", [
|
||||
"jq-99999999999999999999999.0",
|
||||
"jq-1.99999999999999999999999",
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"version_line",
|
||||
[
|
||||
"jq-99999999999999999999999.0",
|
||||
"jq-1.99999999999999999999999",
|
||||
],
|
||||
)
|
||||
def test_an_OUT_OF_RANGE_digit_run_fails_closed(preflight, version_line):
|
||||
"""The round-1 fail-open mechanism, resurrected via an over-long number.
|
||||
|
||||
@@ -247,25 +254,28 @@ def test_an_OUT_OF_RANGE_digit_run_fails_closed(preflight, version_line):
|
||||
assert r.returncode == 1, f"{version_line!r} exited 0 — the floor was not asserted"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version_line", [
|
||||
# Killed by the SEPARATOR restriction (a blank separator must be followed by `version`).
|
||||
"jq\n2.34: cannot load shared library",
|
||||
"jq\n\n\n99.9",
|
||||
"jq -- 2.34 (real jq-1.6)",
|
||||
"jq\t\t9.9",
|
||||
# Killed ONLY by the first-line slice + `[[:blank:]]`. These carry the literal word `version`,
|
||||
# so the separator restriction is satisfied and cannot save us — the newline must be excluded
|
||||
# from the separator class AND the parse confined to line one.
|
||||
#
|
||||
# Without these, a round-5 mutation check found that reverting BOTH of those changes together
|
||||
# (`[[:blank:]]`→`[[:space:]]` and parsing `$raw` instead of `$first`) left the whole suite
|
||||
# GREEN: the four cases above are all killed by the separator alone, so they attributed the fix
|
||||
# to the wrong layer. A test that passes for the wrong reason is how the previous three rounds
|
||||
# each shipped a defect.
|
||||
"jq\nversion\n9.9",
|
||||
"jq\nversion 9.9",
|
||||
"jq \n version \n 9.9",
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"version_line",
|
||||
[
|
||||
# Killed by the SEPARATOR restriction (a blank separator must be followed by `version`).
|
||||
"jq\n2.34: cannot load shared library",
|
||||
"jq\n\n\n99.9",
|
||||
"jq -- 2.34 (real jq-1.6)",
|
||||
"jq\t\t9.9",
|
||||
# Killed ONLY by the first-line slice + `[[:blank:]]`. These carry the literal word `version`,
|
||||
# so the separator restriction is satisfied and cannot save us — the newline must be excluded
|
||||
# from the separator class AND the parse confined to line one.
|
||||
#
|
||||
# Without these, a round-5 mutation check found that reverting BOTH of those changes together
|
||||
# (`[[:blank:]]`→`[[:space:]]` and parsing `$raw` instead of `$first`) left the whole suite
|
||||
# GREEN: the four cases above are all killed by the separator alone, so they attributed the fix
|
||||
# to the wrong layer. A test that passes for the wrong reason is how the previous three rounds
|
||||
# each shipped a defect.
|
||||
"jq\nversion\n9.9",
|
||||
"jq\nversion 9.9",
|
||||
"jq \n version \n 9.9",
|
||||
],
|
||||
)
|
||||
def test_a_number_AFTER_the_jq_token_is_not_reachable_across_filler(preflight, version_line):
|
||||
"""Two independent layers keep a stray number from being read as the version, and both are
|
||||
pinned here: the separator must be one of the forms real jq emits (`jq-1.6` / `jq version 1.6`),
|
||||
@@ -306,14 +316,17 @@ def test_the_observability_line_stays_on_ONE_line(preflight):
|
||||
assert "trailing noise" not in r.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize("version_line,expected", [
|
||||
("jq-1.6 (Debian 1.6-2.1)", "1.6"), # distro packaging suffix
|
||||
("jq-1.10", "1.10"), # two-digit minor: must compare numerically, not lexically
|
||||
("jq-1.7.1", "1.7"),
|
||||
("jq-1.6.0", "1.6"),
|
||||
("jq-v1.6", "1.6"),
|
||||
("JQ-1.6", "1.6"),
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"version_line,expected",
|
||||
[
|
||||
("jq-1.6 (Debian 1.6-2.1)", "1.6"), # distro packaging suffix
|
||||
("jq-1.10", "1.10"), # two-digit minor: must compare numerically, not lexically
|
||||
("jq-1.7.1", "1.7"),
|
||||
("jq-1.6.0", "1.6"),
|
||||
("jq-v1.6", "1.6"),
|
||||
("JQ-1.6", "1.6"),
|
||||
],
|
||||
)
|
||||
def test_legitimate_forms_still_parse_to_the_right_version(preflight, version_line, expected):
|
||||
preflight.with_jq(version_line)
|
||||
r = preflight.run()
|
||||
@@ -323,6 +336,7 @@ def test_legitimate_forms_still_parse_to_the_right_version(preflight, version_li
|
||||
|
||||
# --- Wiring guards: the preflight is worthless if a caller silently stops running it ------------
|
||||
|
||||
|
||||
def test_script_tests_pins_the_jq_version():
|
||||
"""The pin is the tripwire, so its presence is asserted rather than merely commented.
|
||||
|
||||
@@ -333,8 +347,7 @@ def test_script_tests_pins_the_jq_version():
|
||||
is `main`, which does not yet contain `scripts/jq-preflight.sh`.
|
||||
"""
|
||||
pr_checks = (WORKFLOWS / "pr-checks.yml").read_text()
|
||||
assert "jq-preflight.sh --expect" in pr_checks, \
|
||||
"script-tests must pin the jq version — that pin is the tripwire"
|
||||
assert "jq-preflight.sh --expect" in pr_checks, "script-tests must pin the jq version — that pin is the tripwire"
|
||||
|
||||
|
||||
def test_review_verdict_never_pins_a_jq_version():
|
||||
@@ -344,6 +357,7 @@ def test_review_verdict_never_pins_a_jq_version():
|
||||
follow-up PR adds the floor-only call — rather than being a comment someone can miss.
|
||||
"""
|
||||
review_verdict = (WORKFLOWS / "review-verdict.yml").read_text()
|
||||
assert "jq-preflight.sh --expect" not in review_verdict, \
|
||||
("review-verdict.yml must NOT pin a jq version: it writes the required review-verdict/h10 "
|
||||
"status, so a pin would deadlock every merge on a jq bump (ersatztv#648)")
|
||||
assert "jq-preflight.sh --expect" not in review_verdict, (
|
||||
"review-verdict.yml must NOT pin a jq version: it writes the required review-verdict/h10 "
|
||||
"status, so a pin would deadlock every merge on a jq bump (ersatztv#648)"
|
||||
)
|
||||
|
||||
@@ -33,7 +33,7 @@ SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
|
||||
|
||||
# The PR is deliberately NOT docs-only: the docs-only exemption short-circuits the whole gate, so a
|
||||
# docs PR would never reach the base check and the tests would pass without exercising it.
|
||||
CURL_SHIM = r'''#!/usr/bin/env python3
|
||||
CURL_SHIM = r"""#!/usr/bin/env python3
|
||||
import json, os, sys, pathlib, urllib.parse
|
||||
|
||||
state = pathlib.Path(os.environ["STUB_DIR"])
|
||||
@@ -75,14 +75,18 @@ if "/pulls/" in url:
|
||||
sys.exit(0)
|
||||
|
||||
print("{}")
|
||||
'''
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def hook(tmp_path):
|
||||
bindir = tmp_path / "bin"; bindir.mkdir()
|
||||
curl = bindir / "curl"; curl.write_text(CURL_SHIM); curl.chmod(0o755)
|
||||
state = tmp_path / "state"; state.mkdir()
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
curl = bindir / "curl"
|
||||
curl.write_text(CURL_SHIM)
|
||||
curl.chmod(0o755)
|
||||
state = tmp_path / "state"
|
||||
state.mkdir()
|
||||
(state / "live_base").write_text("main")
|
||||
(state / "verdict_desc").write_text("Review-verdict: MERGEABLE @ a9e3e23 (base: main)")
|
||||
|
||||
@@ -90,7 +94,7 @@ def hook(tmp_path):
|
||||
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["STUB_SHA"] = SHA
|
||||
env["ETV_GITEA_TOKEN"] = "stub"
|
||||
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
env["ETV_GITEA_URL"] = "http://gitea.example"
|
||||
env.pop("ETV_GITEA_BASICAUTH", None)
|
||||
|
||||
@@ -103,10 +107,8 @@ def hook(tmp_path):
|
||||
(state / "verdict_desc").write_text(desc)
|
||||
|
||||
def decision(self):
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy",
|
||||
"repo": "ersatztv", "pull_number": 42}}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
|
||||
env=env, capture_output=True, text=True)
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
|
||||
assert r.returncode == 0, r.stderr
|
||||
if not r.stdout.strip():
|
||||
return None
|
||||
@@ -124,21 +126,24 @@ def test_a_retargeted_base_denies_a_verdict_formed_against_the_old_one(hook):
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, "a verdict formed against a different base was allowed to stand"
|
||||
assert "release/26.4" in reason and "main" in reason, (
|
||||
"the deny must name both bases; a reader cannot act on 'the base changed'")
|
||||
"the deny must name both bases; a reader cannot act on 'the base changed'"
|
||||
)
|
||||
|
||||
|
||||
def test_positive_control_an_unchanged_base_does_not_trigger_the_base_deny(hook):
|
||||
"""Without this, the test above could pass because the hook denies on every path — which it
|
||||
very nearly does, since this PR is non-docs and the rest of the gate is unstubbed."""
|
||||
reason = hook.reason()
|
||||
assert "ersatztv#632" not in reason, (
|
||||
"the base check fired on a PR whose base never moved")
|
||||
assert "ersatztv#632" not in reason, "the base check fired on a PR whose base never moved"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("desc", [
|
||||
"Review-verdict: MERGEABLE @ a9e3e23", # posted before #632
|
||||
"NONE", # no verdict status on this head at all
|
||||
])
|
||||
@pytest.mark.parametrize(
|
||||
"desc",
|
||||
[
|
||||
"Review-verdict: MERGEABLE @ a9e3e23", # posted before #632
|
||||
"NONE", # no verdict status on this head at all
|
||||
],
|
||||
)
|
||||
def test_a_verdict_with_no_recorded_base_gets_no_opinion(hook, desc):
|
||||
"""Graceful adoption. Denying here would block every in-flight PR the day this lands, and the
|
||||
window closes on its own: verdicts are per-head and short-lived, so every verdict posted after
|
||||
@@ -151,7 +156,8 @@ def test_a_verdict_with_no_recorded_base_gets_no_opinion(hook, desc):
|
||||
hook.set_live_base("release/26.4")
|
||||
hook.set_verdict_description(desc)
|
||||
assert "base" not in hook.reason(), (
|
||||
"a pre-#632 verdict drew a base-related decision for a field it could not have carried")
|
||||
"a pre-#632 verdict drew a base-related decision for a field it could not have carried"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["SCALAR-ROW", "NONSTRING-DESC"])
|
||||
@@ -173,7 +179,7 @@ def test_a_malformed_status_MEMBER_asks_too(hook, failure):
|
||||
|
||||
@pytest.mark.parametrize("failure", ["TRANSPORT-ERROR", "GARBAGE"])
|
||||
def test_an_UNREADABLE_status_response_asks_rather_than_skipping_the_check(hook, failure):
|
||||
""""Could not check" is a third outcome, not a quiet synonym for "no base recorded".
|
||||
""" "Could not check" is a third outcome, not a quiet synonym for "no base recorded".
|
||||
|
||||
The first draft collapsed the two: an unreadable status response produced an empty
|
||||
`recorded_base`, took the graceful-adoption path, and skipped validation in silence — after
|
||||
@@ -202,5 +208,4 @@ def test_the_comparator_is_the_base_REF_not_its_tip_sha():
|
||||
A base branch that merely ADVANCES must be silent here; rebasing onto it moves the head sha,
|
||||
which the per-sha binding already covers."""
|
||||
assert ".base.ref" in HOOK.read_text(), "the hook must compare the base BRANCH, not its tip sha"
|
||||
assert ".base.sha" not in HOOK.read_text(), (
|
||||
"comparing base.sha deadlocks every open PR whenever main advances")
|
||||
assert ".base.sha" not in HOOK.read_text(), "comparing base.sha deadlocks every open PR whenever main advances"
|
||||
|
||||
@@ -33,7 +33,7 @@ HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
||||
SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
|
||||
|
||||
# Serves paged `pulls/N/files`, plus the minimal PR object the hook reads first.
|
||||
CURL_SHIM = r'''#!/usr/bin/env python3
|
||||
CURL_SHIM = r"""#!/usr/bin/env python3
|
||||
import json, os, sys, pathlib, urllib.parse
|
||||
|
||||
state = pathlib.Path(os.environ["STUB_DIR"])
|
||||
@@ -88,7 +88,7 @@ if "/pulls/" in url:
|
||||
sys.exit(0)
|
||||
|
||||
print("{}")
|
||||
'''
|
||||
"""
|
||||
|
||||
|
||||
def _rows(paths):
|
||||
@@ -97,15 +97,19 @@ def _rows(paths):
|
||||
|
||||
@pytest.fixture
|
||||
def hook(tmp_path):
|
||||
bindir = tmp_path / "bin"; bindir.mkdir()
|
||||
shim = bindir / "curl"; shim.write_text(CURL_SHIM); shim.chmod(0o755)
|
||||
state = tmp_path / "state"; state.mkdir()
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
shim = bindir / "curl"
|
||||
shim.write_text(CURL_SHIM)
|
||||
shim.chmod(0o755)
|
||||
state = tmp_path / "state"
|
||||
state.mkdir()
|
||||
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["STUB_SHA"] = SHA
|
||||
env["ETV_GITEA_TOKEN"] = "stub"
|
||||
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
env["ETV_GITEA_URL"] = "http://gitea.example"
|
||||
env.pop("ETV_GITEA_BASICAUTH", None)
|
||||
|
||||
@@ -114,10 +118,10 @@ def hook(tmp_path):
|
||||
(state / "pages.json").write_text(json.dumps(list(pages)))
|
||||
|
||||
def run(self):
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy",
|
||||
"repo": "ersatztv", "pull_number": 42}}
|
||||
return subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
|
||||
env=env, capture_output=True, text=True)
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
|
||||
return subprocess.run(
|
||||
["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True
|
||||
)
|
||||
|
||||
def exempted(self):
|
||||
"""Exempt == passthrough == exit 0 with no decision JSON."""
|
||||
@@ -140,16 +144,14 @@ def test_code_pr_is_not_exempt(hook):
|
||||
|
||||
def test_protected_path_on_a_LATER_page_is_still_seen(hook):
|
||||
"""The #619 shape: 50 docs files on page 1, code hiding on page 2."""
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
|
||||
_rows(["scripts/decisions_lib.py"]))
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["scripts/decisions_lib.py"]))
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
def test_full_first_page_alone_does_not_end_enumeration(hook):
|
||||
"""A full 50-row page must trigger a second fetch, not terminate the loop."""
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
|
||||
_rows(["docs/tail.md"]))
|
||||
assert hook.exempted() is True # genuinely all docs, but only provable by reading page 2
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["docs/tail.md"]))
|
||||
assert hook.exempted() is True # genuinely all docs, but only provable by reading page 2
|
||||
|
||||
|
||||
def test_rename_of_code_into_docs_is_not_exempt(hook):
|
||||
@@ -161,15 +163,15 @@ def test_rename_of_code_into_docs_is_not_exempt(hook):
|
||||
exempt. That is the hook's contract and differs from `review-verdict.yml`'s stricter
|
||||
PROTECTED list, which must never auto-post a green status for those paths.
|
||||
"""
|
||||
hook.set_pages([{"filename": "docs/innocuous-note.md", "status": "renamed",
|
||||
"previous_filename": "ErsatzTV/Program.cs"}])
|
||||
hook.set_pages(
|
||||
[{"filename": "docs/innocuous-note.md", "status": "renamed", "previous_filename": "ErsatzTV/Program.cs"}]
|
||||
)
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
def test_rename_between_two_exempt_paths_stays_exempt(hook):
|
||||
"""Guards the above: reading previous_filename must not over-trigger on legitimate moves."""
|
||||
hook.set_pages([{"filename": "docs/b.md", "status": "renamed",
|
||||
"previous_filename": "docs/a.md"}])
|
||||
hook.set_pages([{"filename": "docs/b.md", "status": "renamed", "previous_filename": "docs/a.md"}])
|
||||
assert hook.exempted() is True
|
||||
|
||||
|
||||
@@ -232,14 +234,15 @@ def test_malformed_rename_row_withholds_the_exemption(hook):
|
||||
Real Gitea always populates it (verified by constructing a rename), so this is the
|
||||
malformed-2xx class the guard claims to fail closed on; the claim should match the behaviour.
|
||||
"""
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
|
||||
[{"filename": "docs/moved.md", "status": "renamed"}])
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), [{"filename": "docs/moved.md", "status": "renamed"}])
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
def test_rename_row_with_empty_previous_filename_withholds_the_exemption(hook):
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]),
|
||||
[{"filename": "docs/moved.md", "status": "renamed", "previous_filename": ""}])
|
||||
hook.set_pages(
|
||||
_rows([f"docs/f{i}.md" for i in range(50)]),
|
||||
[{"filename": "docs/moved.md", "status": "renamed", "previous_filename": ""}],
|
||||
)
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
@@ -254,9 +257,13 @@ def test_ordinary_row_without_previous_filename_is_still_valid(hook):
|
||||
`test_a_rename_disguised_by_an_unknown_status_is_rejected[None]`. The statusless row this test
|
||||
used to carry was incidental to what it is actually pinning.
|
||||
"""
|
||||
hook.set_pages([{"filename": "docs/a.md", "status": "modified"},
|
||||
{"filename": "docs/b.md", "status": "added"},
|
||||
{"filename": "docs/c.md", "status": "changed"}])
|
||||
hook.set_pages(
|
||||
[
|
||||
{"filename": "docs/a.md", "status": "modified"},
|
||||
{"filename": "docs/b.md", "status": "added"},
|
||||
{"filename": "docs/c.md", "status": "changed"},
|
||||
]
|
||||
)
|
||||
assert hook.exempted() is True
|
||||
|
||||
|
||||
@@ -312,16 +319,20 @@ sys.exit(127)
|
||||
@pytest.fixture
|
||||
def hook_jq16(tmp_path):
|
||||
"""Same harness as `hook`, plus a jq shim emulating jq 1.6's empty-input exit status."""
|
||||
bindir = tmp_path / "bin"; bindir.mkdir()
|
||||
(bindir / "curl").write_text(CURL_SHIM); (bindir / "curl").chmod(0o755)
|
||||
(bindir / "jq").write_text(_JQ16_SHIM); (bindir / "jq").chmod(0o755)
|
||||
state = tmp_path / "state"; state.mkdir()
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
(bindir / "curl").write_text(CURL_SHIM)
|
||||
(bindir / "curl").chmod(0o755)
|
||||
(bindir / "jq").write_text(_JQ16_SHIM)
|
||||
(bindir / "jq").chmod(0o755)
|
||||
state = tmp_path / "state"
|
||||
state.mkdir()
|
||||
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["STUB_SHA"] = SHA
|
||||
env["ETV_GITEA_TOKEN"] = "stub"
|
||||
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
env["ETV_GITEA_URL"] = "http://gitea.example"
|
||||
env.pop("ETV_GITEA_BASICAUTH", None)
|
||||
|
||||
@@ -330,10 +341,8 @@ def hook_jq16(tmp_path):
|
||||
(state / "pages.json").write_text(json.dumps(list(pages)))
|
||||
|
||||
def exempted(self):
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy",
|
||||
"repo": "ersatztv", "pull_number": 42}}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
|
||||
env=env, capture_output=True, text=True)
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
|
||||
assert r.returncode == 0, r.stderr
|
||||
return r.stdout.strip() == ""
|
||||
|
||||
@@ -394,8 +403,9 @@ def test_newline_in_previous_filename_is_also_rejected(hook):
|
||||
rejects on its own merits, so the test passed with the newline guard entirely removed — it
|
||||
asserted the outcome without ever exercising the mechanism. That is the same
|
||||
filter-hides-the-defect trap the guard itself is about."""
|
||||
hook.set_pages([{"filename": "docs/ok.md", "previous_filename": "safe.md\ndocs/Program.cs",
|
||||
"status": "renamed"}], [])
|
||||
hook.set_pages(
|
||||
[{"filename": "docs/ok.md", "previous_filename": "safe.md\ndocs/Program.cs", "status": "renamed"}], []
|
||||
)
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
@@ -406,8 +416,7 @@ def test_previous_filename_is_validated_on_NON_renamed_rows_too(hook, status):
|
||||
`chunk` emits `(.previous_filename // empty)` for EVERY row regardless of `.status`, but the
|
||||
field was validated only when `.status == "renamed"`. A row marked `modified` (or Gitea's
|
||||
distinct `copied`) carrying a newline in `previous_filename` was reproducibly exempted."""
|
||||
hook.set_pages([{"filename": "docs/ok.md", "status": status,
|
||||
"previous_filename": "safe.md\ndocs/Program.cs"}], [])
|
||||
hook.set_pages([{"filename": "docs/ok.md", "status": status, "previous_filename": "safe.md\ndocs/Program.cs"}], [])
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
@@ -421,21 +430,18 @@ def test_dotdot_path_component_is_rejected(hook):
|
||||
|
||||
def test_legitimate_rename_within_docs_still_exempts(hook):
|
||||
"""Positive control: the tightened row schema must not break a real docs-only rename."""
|
||||
hook.set_pages([{"filename": "docs/b.md", "status": "renamed",
|
||||
"previous_filename": "docs/a.md"}], [])
|
||||
hook.set_pages([{"filename": "docs/b.md", "status": "renamed", "previous_filename": "docs/a.md"}], [])
|
||||
assert hook.exempted() is True
|
||||
|
||||
|
||||
def test_short_NONTERMINAL_page_does_not_end_the_enumeration(hook):
|
||||
""""Fewer rows than we asked for" must not be read as "last page".
|
||||
""" "Fewer rows than we asked for" must not be read as "last page".
|
||||
|
||||
Gitea caps `limit` at the server-wide MAX_RESPONSE_ITEMS (default 50, configurable) and may
|
||||
return fewer rows than requested. A 30-row docs page followed by a page of code would otherwise
|
||||
complete the enumeration over a PARTIAL list — the same fail-open, reached with no transport
|
||||
error at all. Only a validated EMPTY page may terminate it."""
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]),
|
||||
_rows(["ErsatzTV/Program.cs"]),
|
||||
[])
|
||||
hook.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]), _rows(["ErsatzTV/Program.cs"]), [])
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
@@ -473,9 +479,14 @@ def test_gitea_real_status_values_are_accepted(hook):
|
||||
"""Positive control for the closed set. The real Gitea 1.25.4 value for an edit is `changed`,
|
||||
NOT `modified` — a closed allow-list built from the wrong vocabulary would reject every real
|
||||
docs-only PR, which is a far worse failure than the hole it closes."""
|
||||
hook.set_pages([{"filename": "docs/a.md", "status": "changed"},
|
||||
{"filename": "docs/b.md", "status": "added"},
|
||||
{"filename": "docs/c.md", "status": "deleted"}], [])
|
||||
hook.set_pages(
|
||||
[
|
||||
{"filename": "docs/a.md", "status": "changed"},
|
||||
{"filename": "docs/b.md", "status": "added"},
|
||||
{"filename": "docs/c.md", "status": "deleted"},
|
||||
],
|
||||
[],
|
||||
)
|
||||
assert hook.exempted() is True
|
||||
|
||||
|
||||
@@ -483,6 +494,7 @@ def test_gitea_real_status_values_are_accepted(hook):
|
||||
# The round-3 `..` finding was an anchor subversion, and mutating the anchors showed no test
|
||||
# covered them: dropping `^` from the docs/ alternative, or `$` from `.md`, both survived.
|
||||
|
||||
|
||||
def test_docs_must_be_a_PREFIX_not_a_substring(hook):
|
||||
"""Dropping `^` would exempt `ErsatzTV/docs/Program.cs`."""
|
||||
hook.set_pages([{"filename": "ErsatzTV/docs/Program.cs", "status": "changed"}], [])
|
||||
@@ -517,7 +529,6 @@ def test_object_valued_status_is_also_rejected(hook):
|
||||
assert hook.exempted() is False
|
||||
|
||||
|
||||
|
||||
# --- The `grep -q` / pipefail inversion, on the ADVISORY side (ersatztv#698) --------------------
|
||||
#
|
||||
# Round-2 cross-family review noted the enforced gate gained large-input regression tests while the
|
||||
@@ -530,6 +541,7 @@ def test_object_valued_status_is_also_rejected(hook):
|
||||
# negated docs-only test. ~171KB is needed to cross the threshold; every other test in this file uses a
|
||||
# handful of short paths, which is exactly why the class was invisible here.
|
||||
|
||||
|
||||
def _many_docs(n=1900):
|
||||
return [f"docs/{'d' * 40}-{i:040d}.md" for i in range(n)]
|
||||
|
||||
@@ -539,12 +551,12 @@ def test_a_LARGE_pr_containing_a_code_file_is_NOT_exempt(hook):
|
||||
bulk of ~171KB still to write."""
|
||||
hook.set_pages(_rows(["A.cs", *_many_docs()]))
|
||||
assert hook.exempted() is False, (
|
||||
"a large PR containing A.cs was granted the docs-only exemption — the predicate inverted")
|
||||
"a large PR containing A.cs was granted the docs-only exemption — the predicate inverted"
|
||||
)
|
||||
|
||||
|
||||
def test_positive_control_a_LARGE_genuinely_docs_only_pr_IS_still_exempt(hook):
|
||||
"""Guards the opposite failure: if large lists merely errored, the test above would pass while the
|
||||
hook prompted on every big docs PR. Without this, 'fixed' and 'broken' are indistinguishable."""
|
||||
hook.set_pages(_rows(_many_docs()))
|
||||
assert hook.exempted() is True, (
|
||||
"a large but genuinely docs-only PR lost its exemption")
|
||||
assert hook.exempted() is True, "a large but genuinely docs-only PR lost its exemption"
|
||||
|
||||
@@ -45,7 +45,7 @@ SHORT = SHA[:7]
|
||||
# The PR is deliberately NOT docs-only: the docs-only exemption short-circuits the whole gate before
|
||||
# the scheduled-merge branch is reached, so a docs PR would pass these tests without ever running
|
||||
# the code under test.
|
||||
CURL_SHIM = r'''#!/usr/bin/env python3
|
||||
CURL_SHIM = r"""#!/usr/bin/env python3
|
||||
import json, os, sys, pathlib, urllib.parse
|
||||
|
||||
state = pathlib.Path(os.environ["STUB_DIR"])
|
||||
@@ -242,14 +242,18 @@ if "/pulls/" in url:
|
||||
sys.exit(0)
|
||||
|
||||
print("{}")
|
||||
'''
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def hook(tmp_path):
|
||||
bindir = tmp_path / "bin"; bindir.mkdir()
|
||||
curl = bindir / "curl"; curl.write_text(CURL_SHIM); curl.chmod(0o755)
|
||||
state = tmp_path / "state"; state.mkdir()
|
||||
bindir = tmp_path / "bin"
|
||||
bindir.mkdir()
|
||||
curl = bindir / "curl"
|
||||
curl.write_text(CURL_SHIM)
|
||||
curl.chmod(0o755)
|
||||
state = tmp_path / "state"
|
||||
state.mkdir()
|
||||
(state / "bp").write_text("GUARDED")
|
||||
|
||||
env = dict(os.environ)
|
||||
@@ -257,7 +261,7 @@ def hook(tmp_path):
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["STUB_SHA"] = SHA
|
||||
env["STUB_SHORT"] = SHORT
|
||||
env["ETV_GITEA_TOKEN"] = "stub"
|
||||
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
env["ETV_GITEA_URL"] = "http://gitea.example"
|
||||
env["CLAUDE_PROJECT_DIR"] = str(REPO_ROOT)
|
||||
env.pop("ETV_GITEA_BASICAUTH", None)
|
||||
@@ -283,11 +287,16 @@ def hook(tmp_path):
|
||||
return f.read_text().splitlines() if f.exists() else []
|
||||
|
||||
def decision(self, scheduled=True):
|
||||
payload = {"tool_input": {"method": "merge", "owner": "timothy",
|
||||
"repo": "ersatztv", "pull_number": 42,
|
||||
"merge_when_checks_succeed": scheduled}}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload),
|
||||
env=env, capture_output=True, text=True)
|
||||
payload = {
|
||||
"tool_input": {
|
||||
"method": "merge",
|
||||
"owner": "timothy",
|
||||
"repo": "ersatztv",
|
||||
"pull_number": 42,
|
||||
"merge_when_checks_succeed": scheduled,
|
||||
}
|
||||
}
|
||||
r = subprocess.run(["bash", str(HOOK)], input=json.dumps(payload), env=env, capture_output=True, text=True)
|
||||
assert r.returncode == 0, r.stderr
|
||||
if not r.stdout.strip():
|
||||
return None
|
||||
@@ -307,9 +316,11 @@ def test_a_base_without_the_required_check_denies_a_SCHEDULED_merge(hook):
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
"a scheduled auto-merge was armed with no 'review-verdict/h10' required check on the base — "
|
||||
"that is ersatztv#622's hole reopened")
|
||||
"that is ersatztv#622's hole reopened"
|
||||
)
|
||||
assert "review-verdict/h10" in reason, (
|
||||
"the deny must name the missing context; a reader cannot act on 'branch protection is wrong'")
|
||||
"the deny must name the missing context; a reader cannot act on 'branch protection is wrong'"
|
||||
)
|
||||
|
||||
|
||||
def test_status_checks_disabled_wholesale_also_denies(hook):
|
||||
@@ -318,8 +329,7 @@ def test_status_checks_disabled_wholesale_also_denies(hook):
|
||||
check-the-label-not-the-capability shape (#697/#698) this repo has paid for twice."""
|
||||
hook.set_branch_protection("STATUS-CHECK-OFF")
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
"status checks were disabled entirely and the listed context was read as protection anyway")
|
||||
assert "deny" in reason, "status checks were disabled entirely and the listed context was read as protection anyway"
|
||||
|
||||
|
||||
def test_positive_control_a_guarded_base_REACHES_the_check_and_still_auto_grants(hook):
|
||||
@@ -334,7 +344,8 @@ def test_positive_control_a_guarded_base_REACHES_the_check_and_still_auto_grants
|
||||
decision = hook.decision()
|
||||
assert hook.branch_protection_urls(), (
|
||||
"the guarded case never reached the branch-protection endpoint, so the other tests are not "
|
||||
"exercising the code they claim to")
|
||||
"exercising the code they claim to"
|
||||
)
|
||||
assert decision is not None, "the hook passed through instead of auto-granting"
|
||||
verdict = decision["hookSpecificOutput"]["permissionDecision"]
|
||||
assert verdict == "allow", f"a fully-satisfied gate did not auto-grant (got {verdict!r})"
|
||||
@@ -354,9 +365,11 @@ def test_a_context_name_that_merely_CONTAINS_the_required_one_does_not_satisfy_i
|
||||
hook.set_branch_protection(shape)
|
||||
reason = hook.reason()
|
||||
assert "allow" not in reason or "deny" in reason or "ask" in reason, (
|
||||
f"payload shape {shape} auto-granted a scheduled merge")
|
||||
f"payload shape {shape} auto-granted a scheduled merge"
|
||||
)
|
||||
assert "satisfied" not in reason, (
|
||||
f"a context name that merely contains 'review-verdict/h10' ({shape}) was accepted as it")
|
||||
f"a context name that merely contains 'review-verdict/h10' ({shape}) was accepted as it"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", ["EMPTY-CONTEXTS", "NULL-CONTEXTS"])
|
||||
@@ -366,8 +379,7 @@ def test_an_empty_or_null_contexts_list_DENIES_rather_than_asking(hook, shape):
|
||||
unknown-shape ask alongside genuinely unreadable payloads."""
|
||||
hook.set_branch_protection(shape)
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
f"{shape} was treated as unreadable rather than as a confirmed absent required check")
|
||||
assert "deny" in reason, f"{shape} was treated as unreadable rather than as a confirmed absent required check"
|
||||
|
||||
|
||||
def test_a_FALSE_contexts_value_asks_rather_than_denying(hook):
|
||||
@@ -400,7 +412,8 @@ def test_a_malformed_contexts_MEMBER_asks_rather_than_denying_with_the_wrong_rea
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, "a malformed contexts member produced a decision instead of a question"
|
||||
assert "NOT a required status check" not in reason, (
|
||||
"an unreadable payload was reported as a confirmed missing required check")
|
||||
"an unreadable payload was reported as a confirmed missing required check"
|
||||
)
|
||||
|
||||
|
||||
def test_a_base_with_NO_branch_protection_at_all_denies_rather_than_asking(hook):
|
||||
@@ -417,11 +430,10 @@ def test_a_base_with_NO_branch_protection_at_all_denies_rather_than_asking(hook)
|
||||
"""
|
||||
hook.set_branch_protection("EMPTY-LIST")
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
"a base with no branch protection at all did not deny a scheduled auto-merge")
|
||||
assert "deny" in reason, "a base with no branch protection at all did not deny a scheduled auto-merge"
|
||||
assert "none matches" in reason, (
|
||||
"the deny must distinguish 'the list was read and nothing governs this base' from "
|
||||
"'could not read'")
|
||||
"the deny must distinguish 'the list was read and nothing governs this base' from 'could not read'"
|
||||
)
|
||||
|
||||
|
||||
def test_a_403_asks_because_it_says_only_that_we_could_not_look(hook):
|
||||
@@ -431,13 +443,12 @@ def test_a_403_asks_because_it_says_only_that_we_could_not_look(hook):
|
||||
hook.set_branch_protection("FORBIDDEN")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, "a 403 was treated as evidence about the protection"
|
||||
assert "none matches" not in reason, (
|
||||
"a 403 was reported as a confirmed absence of branch protection")
|
||||
assert "none matches" not in reason, "a 403 was reported as a confirmed absence of branch protection"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("failure", ["TRANSPORT-ERROR", "GARBAGE", "EMPTY", "LIST-404"])
|
||||
def test_an_UNREADABLE_branch_protection_asks_rather_than_denying_or_passing(hook, failure):
|
||||
""""Could not check" is a third outcome, not a synonym for either neighbour.
|
||||
""" "Could not check" is a third outcome, not a synonym for either neighbour.
|
||||
|
||||
Denying would strand every scheduled merge on a Gitea hiccup or on credentials without the
|
||||
repo-admin scope this endpoint needs. Passing would be worse: it would restore the exact
|
||||
@@ -446,10 +457,8 @@ def test_an_UNREADABLE_branch_protection_asks_rather_than_denying_or_passing(hoo
|
||||
"""
|
||||
hook.set_branch_protection(failure)
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
f"an unreadable branch-protection response ({failure}) did not fall through to a human")
|
||||
assert "branch-protection rules" in reason, (
|
||||
"the ask must name what could not be checked")
|
||||
assert "ask" in reason, f"an unreadable branch-protection response ({failure}) did not fall through to a human"
|
||||
assert "branch-protection rules" in reason, "the ask must name what could not be checked"
|
||||
|
||||
|
||||
def test_an_IMMEDIATE_merge_is_not_subjected_to_this_check(hook):
|
||||
@@ -465,7 +474,8 @@ def test_an_IMMEDIATE_merge_is_not_subjected_to_this_check(hook):
|
||||
hook.set_branch_protection("MISSING-CONTEXT")
|
||||
reason = hook.reason(scheduled=False)
|
||||
assert "NOT a required status check" not in reason, (
|
||||
"the required-check deny fired on an immediate merge, which has no post-scheduling window")
|
||||
"the required-check deny fired on an immediate merge, which has no post-scheduling window"
|
||||
)
|
||||
|
||||
|
||||
def test_a_NON_BOOLEAN_enable_status_check_asks(hook):
|
||||
@@ -504,8 +514,8 @@ def test_a_SCALAR_status_row_asks_instead_of_killing_the_hook(hook):
|
||||
hook.set_branch_protection("SCALAR-STATUS-ROW")
|
||||
decision = hook.decision()
|
||||
assert decision is not None, (
|
||||
"the hook emitted no decision at all for a malformed statuses payload — it neither granted, "
|
||||
"denied nor asked")
|
||||
"the hook emitted no decision at all for a malformed statuses payload — it neither granted, denied nor asked"
|
||||
)
|
||||
assert decision["hookSpecificOutput"]["permissionDecision"] == "ask"
|
||||
|
||||
|
||||
@@ -521,8 +531,7 @@ def test_a_PERSISTENT_retarget_denies_on_the_IMMEDIATE_path_too(hook):
|
||||
hook.drop_recorded_base()
|
||||
hook.set_retarget("scratch")
|
||||
reason = hook.reason(scheduled=False)
|
||||
assert "deny" in reason, (
|
||||
"an immediate merge was auto-granted after the PR was retargeted mid-evaluation")
|
||||
assert "deny" in reason, "an immediate merge was auto-granted after the PR was retargeted mid-evaluation"
|
||||
assert "scratch" in reason and "main" in reason
|
||||
|
||||
|
||||
@@ -540,10 +549,10 @@ def test_a_PERSISTENT_retarget_after_the_first_read_denies(hook):
|
||||
hook.drop_recorded_base()
|
||||
hook.set_retarget("scratch")
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
"the PR was retargeted mid-evaluation and the gate still granted on the original base")
|
||||
assert "deny" in reason, "the PR was retargeted mid-evaluation and the gate still granted on the original base"
|
||||
assert "scratch" in reason and "main" in reason, (
|
||||
"the deny must name both branches; a reader cannot act on 'the base changed'")
|
||||
"the deny must name both branches; a reader cannot act on 'the base changed'"
|
||||
)
|
||||
|
||||
|
||||
def test_a_NON_STRING_status_reaches_the_scheduled_validator(hook):
|
||||
@@ -573,13 +582,12 @@ def test_a_base_that_a_GLOB_rule_could_govern_ASKS(hook):
|
||||
"""
|
||||
hook.set_branch_protection("GLOB-RULE")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
"a base a glob rule could govern was decided rather than referred to a human")
|
||||
assert "ask" in reason, "a base a glob rule could govern was decided rather than referred to a human"
|
||||
assert "could govern it" in reason, (
|
||||
"the ask came from the generic could-not-read arm, not the undecidable-glob arm — those "
|
||||
"are different outcomes and a crashed classifier must not pass as a correct classification")
|
||||
assert "none matches" not in reason, (
|
||||
"a base a glob rule could govern was reported as having no protection at all")
|
||||
"are different outcomes and a crashed classifier must not pass as a correct classification"
|
||||
)
|
||||
assert "none matches" not in reason, "a base a glob rule could govern was reported as having no protection at all"
|
||||
|
||||
|
||||
def test_an_unreadable_rule_LIST_asks_rather_than_denying(hook):
|
||||
@@ -606,8 +614,7 @@ def test_the_protection_lookup_puts_NO_ref_in_the_url(hook):
|
||||
urls = hook.branch_protection_urls()
|
||||
assert urls, "the branch-protection endpoint was never requested"
|
||||
for u in urls:
|
||||
assert u.rstrip("/").endswith("/branch_protections"), (
|
||||
f"a ref reached the branch-protection URL: {u}")
|
||||
assert u.rstrip("/").endswith("/branch_protections"), f"a ref reached the branch-protection URL: {u}"
|
||||
|
||||
|
||||
def test_a_rule_name_with_REGEX_METACHARACTERS_does_not_match_a_different_base(hook):
|
||||
@@ -624,11 +631,11 @@ def test_a_rule_name_with_REGEX_METACHARACTERS_does_not_match_a_different_base(h
|
||||
"""
|
||||
hook.set_branch_protection("REGEX-META-RULE")
|
||||
reason = hook.reason()
|
||||
assert "deny" in reason, (
|
||||
"a rule named 'mai.' was regex-matched against base 'main' and read as protection")
|
||||
assert "deny" in reason, "a rule named 'mai.' was regex-matched against base 'main' and read as protection"
|
||||
assert "none matches" in reason, (
|
||||
"'mai.' contains no GLOB metacharacter, so it is decidable: it simply does not govern "
|
||||
"'main', and the base is genuinely unprotected")
|
||||
"'main', and the base is genuinely unprotected"
|
||||
)
|
||||
|
||||
|
||||
def test_a_GLOB_rule_whose_literal_part_has_a_metacharacter_still_MATCHES(hook):
|
||||
@@ -645,17 +652,19 @@ def test_a_GLOB_rule_whose_literal_part_has_a_metacharacter_still_MATCHES(hook):
|
||||
rather than reporting the base as unprotected.
|
||||
"""
|
||||
hook.set_base("release/26.4")
|
||||
hook.drop_recorded_base() # keep the #632 comparison out of this test's way
|
||||
hook.drop_recorded_base() # keep the #632 comparison out of this test's way
|
||||
hook.set_branch_protection("GLOB-WITH-DOT-RULE")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
"the glob rule 'release/26.*' could govern base 'release/26.4', which is undecidable here "
|
||||
"and must ask")
|
||||
"the glob rule 'release/26.*' could govern base 'release/26.4', which is undecidable here and must ask"
|
||||
)
|
||||
assert "could govern it" in reason, (
|
||||
"the ask must come from the undecidable-glob arm, not from the classifier failing")
|
||||
"the ask must come from the undecidable-glob arm, not from the classifier failing"
|
||||
)
|
||||
assert "none matches" not in reason, (
|
||||
"a base a glob rule could govern was reported as entirely unprotected — the over-escaping "
|
||||
"failure this test exists to catch")
|
||||
"failure this test exists to catch"
|
||||
)
|
||||
|
||||
|
||||
def test_a_rule_name_containing_a_CHAR_CLASS_bracket_does_not_crash_the_matcher(hook):
|
||||
@@ -667,12 +676,13 @@ def test_a_rule_name_containing_a_CHAR_CLASS_bracket_does_not_crash_the_matcher(
|
||||
assert reason, "the hook emitted no decision at all"
|
||||
assert "ask" not in reason, (
|
||||
"the exact-name rule was decidable and must have been honoured; an ask here means the "
|
||||
"classifier failed rather than classified")
|
||||
"classifier failed rather than classified"
|
||||
)
|
||||
assert "none matches" not in reason, (
|
||||
"one rule with a bracket in its name discarded the whole list, including the exact-name "
|
||||
"rule that actually protects this base")
|
||||
assert "deny" not in reason, (
|
||||
"an exact-name rule requiring review-verdict/h10 was present and was not honoured")
|
||||
"rule that actually protects this base"
|
||||
)
|
||||
assert "deny" not in reason, "an exact-name rule requiring review-verdict/h10 was present and was not honoured"
|
||||
|
||||
|
||||
def test_a_plain_rule_name_is_matched_CASE_INSENSITIVELY(hook):
|
||||
@@ -683,9 +693,11 @@ def test_a_plain_rule_name_is_matched_CASE_INSENSITIVELY(hook):
|
||||
reason = hook.reason()
|
||||
assert "ask" not in reason, (
|
||||
"the case-folded exact rule was decidable and must have been honoured; an ask means the "
|
||||
"classifier failed rather than classified")
|
||||
"classifier failed rather than classified"
|
||||
)
|
||||
assert "none matches" not in reason, (
|
||||
"a rule named 'MAIN' governs base 'main' in Gitea but was missed by a case-sensitive compare")
|
||||
"a rule named 'MAIN' governs base 'main' in Gitea but was missed by a case-sensitive compare"
|
||||
)
|
||||
assert "deny" not in reason
|
||||
|
||||
|
||||
@@ -707,12 +719,13 @@ def test_a_BACKSLASH_ESCAPED_metacharacter_in_a_rule_name_is_undecidable_not_abs
|
||||
hook.set_branch_protection("ESCAPED-META-RULE")
|
||||
reason = hook.reason()
|
||||
assert "none matches" not in reason, (
|
||||
"a rule whose escaped brace governs this base was reported as unable to govern it")
|
||||
assert "ask" in reason, (
|
||||
"an escaped-metacharacter rule is undecidable here and must ask")
|
||||
"a rule whose escaped brace governs this base was reported as unable to govern it"
|
||||
)
|
||||
assert "ask" in reason, "an escaped-metacharacter rule is undecidable here and must ask"
|
||||
assert "could govern it" in reason, (
|
||||
"the ask must come from the undecidable-glob arm, not from the classifier failing — this "
|
||||
"test hits the same arm as its two siblings and needs the same pin")
|
||||
"test hits the same arm as its two siblings and needs the same pin"
|
||||
)
|
||||
|
||||
|
||||
def test_the_precedence_check_runs_even_when_an_exactly_named_rule_EXISTS(hook):
|
||||
@@ -732,13 +745,13 @@ def test_the_precedence_check_runs_even_when_an_exactly_named_rule_EXISTS(hook):
|
||||
"""
|
||||
hook.set_branch_protection("EXACT-PLUS-GLOB-RULE")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
"an exactly-named rule was trusted without asking which rule Gitea would actually apply")
|
||||
assert "ask" in reason, "an exactly-named rule was trusted without asking which rule Gitea would actually apply"
|
||||
assert "could govern it" in reason
|
||||
# And the by-name endpoint must not be consulted at all — its existence is what split the paths.
|
||||
for u in hook.branch_protection_urls():
|
||||
assert u.rstrip("/").endswith("/branch_protections"), (
|
||||
f"the by-name lookup is back, and with it the unguarded path: {u}")
|
||||
f"the by-name lookup is back, and with it the unguarded path: {u}"
|
||||
)
|
||||
|
||||
|
||||
def test_a_GLOB_rule_that_could_outrank_an_exact_one_wins_and_ASKS(hook):
|
||||
@@ -758,9 +771,11 @@ def test_a_GLOB_rule_that_could_outrank_an_exact_one_wins_and_ASKS(hook):
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
"an exact rule was trusted while a glob rule could outrank it — the gate granted on a base "
|
||||
"whose enforced rule it never identified")
|
||||
"whose enforced rule it never identified"
|
||||
)
|
||||
assert "could govern it" in reason, (
|
||||
"the ask must come from the undecidable-glob arm, not from the classifier failing")
|
||||
"the ask must come from the undecidable-glob arm, not from the classifier failing"
|
||||
)
|
||||
|
||||
|
||||
def test_two_rules_differing_only_in_CASE_are_undecidable(hook):
|
||||
@@ -769,8 +784,7 @@ def test_two_rules_differing_only_in_CASE_are_undecidable(hook):
|
||||
whose enforced rule was never identified — the same defect as the arm order, one level down."""
|
||||
hook.set_branch_protection("TWO-CASE-VARIANT-RULES")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, (
|
||||
"two fold-equal rules disagree about review-verdict/h10 and one was picked by list order")
|
||||
assert "ask" in reason, "two fold-equal rules disagree about review-verdict/h10 and one was picked by list order"
|
||||
assert "could govern it" in reason
|
||||
|
||||
|
||||
@@ -784,7 +798,8 @@ def test_a_NON_ASCII_rule_or_base_is_undecidable_rather_than_fold_compared(hook)
|
||||
hook.set_branch_protection("NONASCII-RULE")
|
||||
reason = hook.reason()
|
||||
assert "none matches" not in reason, (
|
||||
"a rule that folds equal to this base under EqualFold was reported as unable to govern it")
|
||||
"a rule that folds equal to this base under EqualFold was reported as unable to govern it"
|
||||
)
|
||||
assert "ask" in reason
|
||||
|
||||
|
||||
@@ -803,8 +818,7 @@ def test_an_HTTP_404_on_the_LIST_read_asks_and_does_not_claim_the_list_was_read(
|
||||
hook.set_branch_protection("LIST-404")
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, "an unreadable repo was treated as evidence about the base"
|
||||
assert "none matches" not in reason, (
|
||||
"a 404 read claimed the full rule list had been read and matched nothing")
|
||||
assert "none matches" not in reason, "a 404 read claimed the full rule list had been read and matched nothing"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", ["UNPARSEABLE-RULES", "GARBAGE", "EMPTY"])
|
||||
@@ -825,6 +839,5 @@ def test_a_200_the_classifier_cannot_PARSE_asks_without_blaming_the_transport(ho
|
||||
hook.set_branch_protection(shape)
|
||||
reason = hook.reason()
|
||||
assert "ask" in reason, "an unparseable rule list produced a decision instead of a question"
|
||||
assert "could not parse" in reason, (
|
||||
"the ask blamed the transport for a 200 the classifier simply could not read")
|
||||
assert "could not parse" in reason, "the ask blamed the transport for a 200 the classifier simply could not read"
|
||||
assert "unreachable" not in reason
|
||||
|
||||
@@ -104,7 +104,7 @@ def gitea(tmp_path):
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
||||
env["STUB_DIR"] = str(state)
|
||||
env["ETV_GITEA_TOKEN"] = "stub-token"
|
||||
env["ETV_GITEA_TOKEN"] = "stub-token" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
||||
env["ETV_GITEA_URL"] = "http://gitea.example"
|
||||
env["ETV_GITEA_REPO"] = "timothy/ersatztv"
|
||||
env.pop("ETV_GITEA_BASICAUTH", None)
|
||||
@@ -127,7 +127,9 @@ def gitea(tmp_path):
|
||||
def run(self, *args):
|
||||
return subprocess.run(
|
||||
["bash", str(SCRIPT), *args],
|
||||
env=env, capture_output=True, text=True,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
def posts(self):
|
||||
@@ -229,6 +231,7 @@ def test_unreachable_pr_is_an_error_not_a_silent_success(gitea):
|
||||
|
||||
# --- Cross-checks against the hook's own condition-(c) parser -----------------------------------
|
||||
|
||||
|
||||
def _classify(body: str, head: str) -> str:
|
||||
"""Run the REAL H10 classifier over a comment body — no Python mirror of the grammar.
|
||||
|
||||
@@ -239,9 +242,7 @@ def _classify(body: str, head: str) -> str:
|
||||
passing here while the shell drifted.
|
||||
"""
|
||||
payload = json.dumps([{"body": body}])
|
||||
p = subprocess.run(
|
||||
["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True
|
||||
)
|
||||
p = subprocess.run(["bash", str(CLASSIFIER), "--head", head], input=payload, capture_output=True, text=True)
|
||||
assert p.returncode == 0, f"classifier errored: {p.stderr}"
|
||||
return p.stdout.strip()
|
||||
|
||||
@@ -280,6 +281,7 @@ def test_note_cannot_forge_a_second_verdict_line(gitea):
|
||||
# mirror case: the head sha and the status both hold still while the effective DIFF changes, so the
|
||||
# verdict keeps reading green for a review nobody performed against that base.
|
||||
|
||||
|
||||
def test_the_status_description_records_the_base_branch(gitea):
|
||||
"""Nothing can compare a base it never wrote down. This field is what the hook reads back."""
|
||||
assert gitea.run("42", "MERGEABLE").returncode == 0
|
||||
@@ -333,7 +335,8 @@ def test_a_failed_HEAD_RECHECK_writes_no_status(gitea):
|
||||
result = gitea.run("42", "MERGEABLE")
|
||||
assert result.returncode != 0
|
||||
assert gitea.statuses() == [], (
|
||||
"a status was written even though the head/base re-read failed — nothing was confirmed")
|
||||
"a status was written even though the head/base re-read failed — nothing was confirmed"
|
||||
)
|
||||
|
||||
|
||||
# --- write-side polarity, the half the #774 rescue initially missed ------------------------------
|
||||
@@ -404,8 +407,8 @@ def test_a_reread_that_LOSES_a_field_refuses_instead_of_posting(head_seq, base_s
|
||||
gitea.set_base_sequence(*base_seq)
|
||||
result = gitea.run("42", "MERGEABLE")
|
||||
assert result.returncode != 0, (
|
||||
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed "
|
||||
"nothing about it")
|
||||
f"a re-read missing its {field} was accepted; the script posted a verdict having confirmed nothing about it"
|
||||
)
|
||||
assert not gitea.statuses(), (
|
||||
f"a status was written despite the re-read carrying no {field} — this is the fail-open "
|
||||
"the -n conjunct created")
|
||||
f"a status was written despite the re-read carrying no {field} — this is the fail-open the -n conjunct created"
|
||||
)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -35,14 +35,18 @@ PROVE_FIX = Path(os.environ.get("PROVE_FIX_PATH") or (REPO_ROOT / "scripts" / "p
|
||||
def _git(repo: Path, *args: str) -> str:
|
||||
return subprocess.run(
|
||||
["git", "-C", str(repo), *args],
|
||||
check=True, capture_output=True, text=True,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
|
||||
|
||||
def _run(repo: Path, *args: str) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
["bash", str(PROVE_FIX), "--repo", str(repo), *args],
|
||||
capture_output=True, text=True, cwd=str(repo),
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(repo),
|
||||
)
|
||||
|
||||
|
||||
@@ -61,9 +65,7 @@ def fixrepo(tmp_path: Path) -> Path:
|
||||
|
||||
# --- commit 1: the bug, plus a test that cannot see it
|
||||
(repo / "calc.py").write_text("def add(a, b):\n return a - b # bug\n")
|
||||
(repo / "scripts" / "tests" / "test_unrelated.py").write_text(
|
||||
"def test_unrelated():\n assert 1 + 1 == 2\n"
|
||||
)
|
||||
(repo / "scripts" / "tests" / "test_unrelated.py").write_text("def test_unrelated():\n assert 1 + 1 == 2\n")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "initial: buggy add, unrelated test")
|
||||
|
||||
@@ -73,8 +75,7 @@ def fixrepo(tmp_path: Path) -> Path:
|
||||
"from calc import add\n\n\ndef test_add():\n assert add(2, 3) == 5\n"
|
||||
)
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m",
|
||||
"fix: add() returned a difference\n\nProves: scripts/tests/test_add.py")
|
||||
_git(repo, "commit", "-q", "-m", "fix: add() returned a difference\n\nProves: scripts/tests/test_add.py")
|
||||
return repo
|
||||
|
||||
|
||||
@@ -143,7 +144,8 @@ def test_added_code_file_is_removed_not_checked_out(tmp_path: Path) -> None:
|
||||
_git(repo, "config", "user.email", "t@example.com")
|
||||
_git(repo, "config", "user.name", "T")
|
||||
(repo / "scripts" / "tests" / "keep.py").write_text("# placeholder\n")
|
||||
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "initial")
|
||||
|
||||
# the fix ADDS helper.py (it has no parent version) and a test that needs it
|
||||
(repo / "helper.py").write_text("def shout(s):\n return s.upper()\n")
|
||||
@@ -165,7 +167,8 @@ def test_root_commit_REFUSES(tmp_path: Path) -> None:
|
||||
_git(repo, "config", "user.email", "t@example.com")
|
||||
_git(repo, "config", "user.name", "T")
|
||||
(repo / "a.py").write_text("x = 1\n")
|
||||
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "root")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "root")
|
||||
r = _run(repo, "HEAD", "scripts/tests")
|
||||
assert r.returncode == 5
|
||||
assert "root commit" in r.stderr
|
||||
@@ -185,9 +188,10 @@ def test_runs_under_the_system_bash(fixrepo: Path) -> None:
|
||||
# developer Mac. The 3.2 hazard (mapfile yielding an empty array) is what motivated it.
|
||||
ver = subprocess.run([str(system_bash), "--version"], capture_output=True, text=True).stdout
|
||||
r = subprocess.run(
|
||||
[str(system_bash), str(PROVE_FIX), "--repo", str(fixrepo), "HEAD",
|
||||
"scripts/tests/test_add.py"],
|
||||
capture_output=True, text=True, cwd=str(fixrepo),
|
||||
[str(system_bash), str(PROVE_FIX), "--repo", str(fixrepo), "HEAD", "scripts/tests/test_add.py"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(fixrepo),
|
||||
)
|
||||
assert r.returncode == 0, (
|
||||
f"prove-fix.sh must work under the system bash ({ver.splitlines()[0] if ver else '?'}); "
|
||||
@@ -204,8 +208,7 @@ def test_control_failure_REFUSES(fixrepo: Path) -> None:
|
||||
)
|
||||
(fixrepo / "calc.py").write_text("def add(a, b):\n return a + b # unchanged\n")
|
||||
_git(fixrepo, "add", "-A")
|
||||
_git(fixrepo, "commit", "-q", "-m",
|
||||
"fix: with an already-failing test\n\nProves: scripts/tests/test_broken.py")
|
||||
_git(fixrepo, "commit", "-q", "-m", "fix: with an already-failing test\n\nProves: scripts/tests/test_broken.py")
|
||||
r = _run(fixrepo, "HEAD")
|
||||
assert r.returncode == 6, f"expected 6 (control failed), got {r.returncode}\n{r.stdout}\n{r.stderr}"
|
||||
assert "control FAILED" in r.stderr
|
||||
@@ -248,9 +251,19 @@ def test_MUTATION_disarming_the_UNPROVEN_clause_reddens_the_refusal_test(
|
||||
|
||||
env = {**os.environ, "PROVE_FIX_PATH": str(mutant), "PROVE_FIX_MUTATION_RUN": "1"}
|
||||
nested = subprocess.run(
|
||||
["python3", "-m", "pytest", f"{Path(__file__).name}::test_unrelated_test_is_UNPROVEN",
|
||||
"-q", "-p", "no:cacheprovider"],
|
||||
cwd=str(Path(__file__).parent), env=env, capture_output=True, text=True,
|
||||
[
|
||||
"python3",
|
||||
"-m",
|
||||
"pytest",
|
||||
f"{Path(__file__).name}::test_unrelated_test_is_UNPROVEN",
|
||||
"-q",
|
||||
"-p",
|
||||
"no:cacheprovider",
|
||||
],
|
||||
cwd=str(Path(__file__).parent),
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
out = nested.stdout + nested.stderr
|
||||
assert nested.returncode == 1, (
|
||||
@@ -282,17 +295,20 @@ def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
|
||||
_git(repo, "config", "user.name", "T")
|
||||
(repo / "mod.py").write_text("VALUE = 1\n")
|
||||
(repo / "scripts" / "tests" / "test_slow.py").write_text(
|
||||
"import time\nfrom mod import VALUE\n\n\n"
|
||||
"def test_slow():\n time.sleep(20)\n assert VALUE == 2\n"
|
||||
"import time\nfrom mod import VALUE\n\n\ndef test_slow():\n time.sleep(20)\n assert VALUE == 2\n"
|
||||
)
|
||||
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "initial")
|
||||
(repo / "mod.py").write_text("VALUE = 2\n")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "fix: bump\n\nProves: scripts/tests/test_slow.py")
|
||||
|
||||
proc = subprocess.Popen(
|
||||
["bash", str(PROVE_FIX), "--repo", str(repo), "HEAD"],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd=str(repo),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
cwd=str(repo),
|
||||
)
|
||||
time.sleep(4) # inside the control run, which sleeps 20s
|
||||
assert proc.poll() is None, "the run finished before it could be signalled; test is void"
|
||||
@@ -305,9 +321,7 @@ def test_SIGTERM_mid_run_never_reports_PROVEN(tmp_path: Path) -> None:
|
||||
# bugs landing on the same observable is not a witnessed fix. Cold review measured that
|
||||
# pair failing to separate old from new; rc==5 AND the handler's own message do separate
|
||||
# them.
|
||||
assert proc.returncode == 5, (
|
||||
f"a signalled run must exit 5 from on_signal, got {proc.returncode}\n{out}\n{err}"
|
||||
)
|
||||
assert proc.returncode == 5, f"a signalled run must exit 5 from on_signal, got {proc.returncode}\n{out}\n{err}"
|
||||
assert "interrupted by signal" in err, (
|
||||
f"expected the signal handler's own message, so this test cannot be satisfied by an "
|
||||
f"unrelated later failure:\n{err}"
|
||||
@@ -335,7 +349,8 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
|
||||
_git(repo, "config", "user.email", "t@example.com")
|
||||
_git(repo, "config", "user.name", "T")
|
||||
(repo / "scripts" / "tests" / "keep.py").write_text("# placeholder\n")
|
||||
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "initial")
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "initial")
|
||||
(repo / "added.py").write_text("def val():\n return 7\n")
|
||||
(repo / "scripts" / "tests" / "test_added.py").write_text(
|
||||
"from added import val\n\n\ndef test_val():\n assert val() == 7\n"
|
||||
@@ -343,7 +358,8 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
|
||||
_git(repo, "add", "-A")
|
||||
_git(repo, "commit", "-q", "-m", "feat: add val\n\nProves: scripts/tests/test_added.py")
|
||||
|
||||
shim = tmp_path / "bin"; shim.mkdir()
|
||||
shim = tmp_path / "bin"
|
||||
shim.mkdir()
|
||||
counter = tmp_path / "count"
|
||||
(shim / "git").write_text(
|
||||
"#!/bin/sh\n"
|
||||
@@ -358,7 +374,10 @@ def test_a_harness_failure_is_NOT_reported_as_PROVEN(tmp_path: Path) -> None:
|
||||
env = {**os.environ, "PATH": f"{shim}:{os.environ['PATH']}"}
|
||||
r = subprocess.run(
|
||||
["bash", str(PROVE_FIX), "--repo", str(repo), "HEAD"],
|
||||
capture_output=True, text=True, cwd=str(repo), env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(repo),
|
||||
env=env,
|
||||
)
|
||||
assert "PROVEN" not in r.stdout, (
|
||||
"a phase whose worktree does not exist cannot witness anything; pre-fix this "
|
||||
|
||||
@@ -109,7 +109,8 @@ def _tracked_files() -> list[str]:
|
||||
"""
|
||||
proc = subprocess.run(
|
||||
["git", "-C", str(REPO_ROOT), "ls-files", "-z"],
|
||||
capture_output=True, check=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
return [p for p in proc.stdout.decode().split("\0") if p]
|
||||
|
||||
@@ -150,7 +151,8 @@ def _inventory_section(text: str) -> str:
|
||||
raise AssertionError(
|
||||
f"{INVENTORY.name} has no {heading!r} heading. Row parsing is bounded by "
|
||||
"'## The inventory' and '## Limits'; renaming or reordering either one would "
|
||||
"silently change which rows are checked, so it fails here instead.")
|
||||
"silently change which rows are checked, so it fails here instead."
|
||||
)
|
||||
start = text.index("## The inventory")
|
||||
try:
|
||||
end = text.index("## Limits", start)
|
||||
@@ -161,7 +163,8 @@ def _inventory_section(text: str) -> str:
|
||||
# either returns an index >= start or raises. A guard that cannot execute proves nothing.
|
||||
raise AssertionError(
|
||||
f"{INVENTORY.name}: '## Limits' precedes '## The inventory', so the parsed window "
|
||||
"would be empty and every completeness assertion would pass vacuously.") from None
|
||||
"would be empty and every completeness assertion would pass vacuously."
|
||||
) from None
|
||||
return text[start:end]
|
||||
|
||||
|
||||
@@ -188,10 +191,11 @@ def test_anti_vacuity_the_derivation_and_the_table_both_found_something():
|
||||
sites = inventory_sites()
|
||||
assert len(population) >= 40, (
|
||||
f"derived only {len(population)} in-scope files — the globs are broken, not the repo "
|
||||
"(the scope held 59 files on 2026-08-16, and it only grows)")
|
||||
"(the scope held 59 files on 2026-08-16, and it only grows)"
|
||||
)
|
||||
assert len(sites) >= 40, (
|
||||
f"parsed only {len(sites)} rows out of the inventory — the row regex has drifted from the "
|
||||
"table format")
|
||||
f"parsed only {len(sites)} rows out of the inventory — the row regex has drifted from the table format"
|
||||
)
|
||||
|
||||
|
||||
def test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file():
|
||||
@@ -208,9 +212,9 @@ def test_every_in_scope_file_has_a_row_and_every_row_names_a_real_file():
|
||||
phantom = sorted(sites - population)
|
||||
assert not missing, (
|
||||
"in scope but absent from docs/remote-state-inventory.md (classify each as "
|
||||
f"PINNED / CAS / UNSAFE-KNOWN / N/A): {missing}")
|
||||
assert not phantom, (
|
||||
f"listed in docs/remote-state-inventory.md but no such in-scope file exists: {phantom}")
|
||||
f"PINNED / CAS / UNSAFE-KNOWN / N/A): {missing}"
|
||||
)
|
||||
assert not phantom, f"listed in docs/remote-state-inventory.md but no such in-scope file exists: {phantom}"
|
||||
|
||||
|
||||
def test_MUTATION_PROOF_a_dropped_row_and_a_phantom_row_are_both_detected():
|
||||
@@ -228,27 +232,28 @@ def test_MUTATION_PROOF_a_dropped_row_and_a_phantom_row_are_both_detected():
|
||||
population = derived_population()
|
||||
|
||||
victim = sorted(population)[0]
|
||||
dropped = "\n".join(
|
||||
line for line in text.splitlines() if not line.startswith(f"| `{victim}`"))
|
||||
dropped = "\n".join(line for line in text.splitlines() if not line.startswith(f"| `{victim}`"))
|
||||
assert victim not in inventory_sites(dropped), (
|
||||
f"the mutation did not actually remove {victim}; the proof below would be vacuous")
|
||||
f"the mutation did not actually remove {victim}; the proof below would be vacuous"
|
||||
)
|
||||
assert population - inventory_sites(dropped), (
|
||||
"a row was removed from the inventory and the comparison still reported complete coverage")
|
||||
"a row was removed from the inventory and the comparison still reported complete coverage"
|
||||
)
|
||||
|
||||
# Inserted INSIDE the inventory section, not appended to the file: rows are parsed only between
|
||||
# "## The inventory" and "## Limits", so appending at the end would test nothing.
|
||||
phantom = text.replace(
|
||||
"## Limits",
|
||||
"| `scripts/does-not-exist.sh` — invented | `PINNED` | n/a |\n\n## Limits", 1)
|
||||
phantom = text.replace("## Limits", "| `scripts/does-not-exist.sh` — invented | `PINNED` | n/a |\n\n## Limits", 1)
|
||||
assert inventory_sites(phantom) - population == {"scripts/does-not-exist.sh"}, (
|
||||
"a row naming a file that does not exist was not reported as phantom")
|
||||
"a row naming a file that does not exist was not reported as phantom"
|
||||
)
|
||||
|
||||
|
||||
def test_every_class_cell_comes_from_the_closed_vocabulary():
|
||||
bad = sorted({cls for _, cls in inventory_rows() if cls not in CLASSES})
|
||||
assert not bad, (
|
||||
f"unknown classification(s) {bad}; allowed: {sorted(CLASSES)}. A typo here would silently "
|
||||
"create a state nobody reviews.")
|
||||
"create a state nobody reviews."
|
||||
)
|
||||
|
||||
|
||||
def test_every_unsafe_row_states_why_the_residual_is_accepted():
|
||||
@@ -268,8 +273,7 @@ def test_every_unsafe_row_states_why_the_residual_is_accepted():
|
||||
site, note = cells[0], cells[-1]
|
||||
if len(note) < 120:
|
||||
thin.append(site[:60])
|
||||
assert not thin, (
|
||||
f"UNSAFE-KNOWN row(s) with no stated justification: {thin}")
|
||||
assert not thin, f"UNSAFE-KNOWN row(s) with no stated justification: {thin}"
|
||||
|
||||
|
||||
def test_the_population_never_includes_a_file_git_does_not_track(monkeypatch):
|
||||
@@ -296,14 +300,15 @@ def test_the_population_never_includes_a_file_git_does_not_track(monkeypatch):
|
||||
)
|
||||
# Patch the module object this test is running inside, whatever name it was imported under.
|
||||
import sys
|
||||
|
||||
mod = sys.modules[__name__]
|
||||
monkeypatch.setattr(mod, "_tracked_files", lambda: tracked)
|
||||
|
||||
assert (REPO_ROOT / victim).is_file(), (
|
||||
f"{victim} must still exist on disk for this proof to mean anything")
|
||||
assert (REPO_ROOT / victim).is_file(), f"{victim} must still exist on disk for this proof to mean anything"
|
||||
assert victim not in derived_population(), (
|
||||
f"{victim} is on disk and matches the scope, but git no longer tracks it — it must not enter "
|
||||
"the population, or untracked build output can redden this guard again")
|
||||
"the population, or untracked build output can redden this guard again"
|
||||
)
|
||||
|
||||
|
||||
def test_every_derived_member_is_tracked():
|
||||
|
||||
Reference in New Issue
Block a user