scripts/tests/ is never executed by CI — the Python test suite is local-only #631

Closed
opened 2026-07-25 21:52:42 +02:00 by timothy · 3 comments
Owner

The gap

scripts/tests/ holds 95 pytest tests covering the decision-corpus parser, validator, catalog builder, the #610 migration-equivalence harness and (as of #622) the review-verdict poster. None of them run in CI.

Verified by grep: no workflow and no Husky hook invokes pytest or references scripts/tests. decisions-guard in pr-checks.yml runs decisions_validate.py and build_decisions_catalog.py --check directly — it exercises the code but never the tests. The test job is dotnet test only.

Why it matters

These are exactly the tests guarding the machinery that gates merges and the decision corpus — the components where "reports success while doing nothing" has now bitten repeatedly (#603 stale-after, #609 marker, #621 silent record invisibility). A regression in them is currently caught only if someone happens to run pytest locally.

It also means a test added "for CI enforcement" is quietly decorative. I nearly shipped #622's suite believing it was enforced.

Shape of the fix

A step in the existing decisions-guard job (already has setup-python, already runs-on: small) is probably the cheapest home:

- name: Script tests
  run: PYTHONPATH=. python3 -m pytest scripts/tests -q

Worth checking first whether pytest is present on the small lane or needs installing, and whether the suite is fast enough to sit in a blocking gate (~4s locally).

Done-when

  • scripts/tests/ runs in CI on every PR that touches scripts/ (or unconditionally, if cheap)
  • Verified by measurement, not a green tick — deliberately break one test on a scratch branch and confirm CI goes red
  • docs/ci-cd.md documents where the suite runs
  • Adversarial review passed
## The gap `scripts/tests/` holds 95 pytest tests covering the decision-corpus parser, validator, catalog builder, the #610 migration-equivalence harness and (as of #622) the review-verdict poster. **None of them run in CI.** Verified by grep: no workflow and no Husky hook invokes `pytest` or references `scripts/tests`. `decisions-guard` in `pr-checks.yml` runs `decisions_validate.py` and `build_decisions_catalog.py --check` directly — it exercises the *code* but never the *tests*. The `test` job is `dotnet test` only. ## Why it matters These are exactly the tests guarding the machinery that gates merges and the decision corpus — the components where "reports success while doing nothing" has now bitten repeatedly (#603 `stale-after`, #609 marker, #621 silent record invisibility). A regression in them is currently caught only if someone happens to run pytest locally. It also means a test added "for CI enforcement" is quietly decorative. I nearly shipped #622's suite believing it was enforced. ## Shape of the fix A step in the existing `decisions-guard` job (already has `setup-python`, already `runs-on: small`) is probably the cheapest home: ```yaml - name: Script tests run: PYTHONPATH=. python3 -m pytest scripts/tests -q ``` Worth checking first whether pytest is present on the `small` lane or needs installing, and whether the suite is fast enough to sit in a blocking gate (~4s locally). ## Done-when - [x] `scripts/tests/` runs in CI on every PR that touches `scripts/` (or unconditionally, if cheap) - [x] Verified by *measurement*, not a green tick — deliberately break one test on a scratch branch and confirm CI goes red - [x] `docs/ci-cd.md` documents where the suite runs - [x] Adversarial review passed
timothy added the ci-cdpriority: medium labels 2026-07-25 21:52:42 +02:00
timothy added the in-progress label 2026-07-26 00:27:32 +02:00
Author
Owner

Negative control — measured, not asserted

Done-when box 2 asks for verification "by measurement, not a green tick — deliberately break one test on a scratch branch and confirm CI goes red." Done on throwaway PR #638 (branch probe/631-negative-control = PR #637's branch + one deliberately failing test, never merged):

PR #637 (clean) PR #638 (one broken test)
PR Gates / Script tests (pytest) success failure
PR Gates / decisions lifecycle success success
PR Gates / CI image pin success success
PR Gates / Docs update reminder success success

Identical branches apart from the broken test, so the job's red is attributable to the test and nothing else.

What running the suite in CI immediately caught

Two real reds before the green, both of which had been sitting undetected precisely because nothing ran these tests:

  1. ModuleNotFoundError: No module named 'yaml' — the suite is not stdlib-only; test_migration_equivalence.py transitively imports PyYAML via migrate_decisions_split.py. It passed locally only because this machine had PyYAML. Fixed by installing pytest pyyaml, with the complete third-party set established by an AST import scan rather than by reading the files that seemed relevant.

  2. A fail-open in the merge-consent gate itself. jq -e over empty input exits 4 on jq ≥ 1.7 but 0 on jq 1.6 — verified against both binaries, and the CI runner ships jq 1.6. The docs-only pagination guard leaned on that exit status to reject a transport failure, so on jq 1.6: page errors → gq returns empty → guard wrongly passes → n empty so [ "$n" -lt 50 ] errors into false → loop walks past the failed page → next page legitimately returns [] → files_complete=yes over a partial list → the docs-only exemption fires over unread pages that may be pure code.

    That is the exact defect the guard's own comment describes, reintroduced one layer down by a jq version difference. Fixed by rejecting an empty body explicitly instead of inferring it from jq's exit status.

The second one is this issue's argument in miniature: it lived in the gate that decides whether a PR skips the Done-when checks, it was already covered by a test, and that test cannot catch it on a developer Mac (jq 1.8) — only in CI, where the suite had never run. A new test now pins the property on any jq by shimming only the 1.6 behaviour; mutation-verified.

Standing rule it leaves behind: never infer "empty input" from a jq exit status; check the string.

Deviation from the proposed fix

The issue suggested a step inside decisions-guard as "probably the cheapest home". That is the one place it must not go — ci.decisions-lifecycle-flake instructs sessions that a lone decisions lifecycle red is a known flake and must not be investigated, so a genuine pytest regression there would surface as exactly the red everyone is told to wave through. It is its own script-tests job instead. Recorded as ci.script-tests-job.

PR: #637. Probe PR #638 is closed unmerged and its branch deleted.

## Negative control — measured, not asserted Done-when box 2 asks for verification *"by measurement, not a green tick — deliberately break one test on a scratch branch and confirm CI goes red."* Done on throwaway PR #638 (branch `probe/631-negative-control` = PR #637's branch + one deliberately failing test, never merged): | | PR #637 (clean) | PR #638 (one broken test) | |---|---|---| | `PR Gates / Script tests (pytest)` | **success** | **failure** | | `PR Gates / decisions lifecycle` | success | success | | `PR Gates / CI image pin` | success | success | | `PR Gates / Docs update reminder` | success | success | Identical branches apart from the broken test, so the job's red is attributable to the test and nothing else. ## What running the suite in CI immediately caught Two real reds before the green, both of which had been sitting undetected precisely *because* nothing ran these tests: 1. **`ModuleNotFoundError: No module named 'yaml'`** — the suite is not stdlib-only; `test_migration_equivalence.py` transitively imports PyYAML via `migrate_decisions_split.py`. It passed locally only because this machine had PyYAML. Fixed by installing `pytest pyyaml`, with the complete third-party set established by an AST import scan rather than by reading the files that seemed relevant. 2. **A fail-open in the merge-consent gate itself.** `jq -e` over empty input exits 4 on jq ≥ 1.7 but **0 on jq 1.6** — verified against both binaries, and the CI runner ships jq 1.6. The docs-only pagination guard leaned on that exit status to reject a transport failure, so on jq 1.6: page errors → `gq` returns empty → guard wrongly passes → `n` empty so `[ "$n" -lt 50 ]` errors into false → loop walks **past** the failed page → next page legitimately returns `[]` → `files_complete=yes` over a **partial** list → the docs-only exemption fires over unread pages that may be pure code. That is the exact defect the guard's own comment describes, reintroduced one layer down by a jq version difference. Fixed by rejecting an empty body explicitly instead of inferring it from jq's exit status. The second one is this issue's argument in miniature: it lived in the gate that decides whether a PR skips the Done-when checks, it was already covered by a test, and that test **cannot** catch it on a developer Mac (jq 1.8) — only in CI, where the suite had never run. A new test now pins the property on any jq by shimming only the 1.6 behaviour; mutation-verified. Standing rule it leaves behind: **never infer "empty input" from a jq exit status; check the string.** ## Deviation from the proposed fix The issue suggested a step inside `decisions-guard` as "probably the cheapest home". That is the one place it must not go — `ci.decisions-lifecycle-flake` instructs sessions that a lone `decisions lifecycle` red is a known flake and must *not* be investigated, so a genuine pytest regression there would surface as exactly the red everyone is told to wave through. It is its own `script-tests` job instead. Recorded as `ci.script-tests-job`. PR: #637. Probe PR #638 is closed unmerged and its branch deleted.
Author
Owner

The second of the two reds above — the merge-consent gate failing open on jq 1.6 — is now filed separately as #643 and carries its own ## Done-when. It stays fixed in PR #637; #643 exists so the gate's security-posture change is retrievable on its own rather than only as a paragraph inside this issue.

The second of the two reds above — the merge-consent gate failing open on jq 1.6 — is now filed separately as **#643** and carries its own `## Done-when`. It stays fixed in PR #637; #643 exists so the gate's security-posture change is retrievable on its own rather than only as a paragraph inside this issue.
Author
Owner

Closing record

Outcome: Shipped in PR #637. scripts/tests/ now runs on every PR as its own script-tests job in pr-checks.yml. New decision record ci.script-tests-job.

Root cause: Nothing invoked pytest. decisions-guard runs decisions_validate.py and build_decisions_catalog.py directly — it exercises that code but never its tests — and the test job is dotnet test only. So the suite guarding the decision corpus, the #610 migration harness, the merge-consent logic and the #622 verdict poster was enforced solely by whoever remembered to run it locally.

Decisions/conventions changed: added ci.script-tests-job. The load-bearing part is where the job does not go: the issue proposed a step inside decisions-guard, which is the one job covered by ci.decisions-lifecycle-flake ("a lone decisions lifecycle red is a known flake — do not investigate"). A real pytest regression there would surface as exactly the red every session is told to wave through. A gate inherits the credibility of the job it lives in, so a job under a standing ignore-rule can host no real gate.

Reusable knowledge — what switching the light on immediately found. Four latent defects, none of them in this PR's own work:

  1. The suite is not stdlib-only (test_migration_equivalence.py → PyYAML). It passed locally only because this machine had it. The complete third-party set came from an AST import scan, not from reading the files that looked relevant.
  2. #643 — a jq-1.6 fail-open in the merge-consent hook's docs-only enumeration.
  3. #647 — the H10 verdict classifier was entirely inert on jq 1.6 (contains("�") is true for every string there), and the same fail-open sat in review-verdict.yml, the server-enforced copy. 61 of 178 tests failed under jq 1.6 on main.
  4. A record title whose unquoted # was a YAML comment, so PyYAML and our hand reader disagreed — already red on main.

The pattern: every one was invisible because the only thing that could see it never ran. A test suite that runs nowhere is not a safety net, it is a document.

Verification: measured, not asserted. Throwaway PR #638 = this branch + one deliberately broken test → Script tests (pytest) failure, every other job green; this branch → success. 216 tests green under both jq 1.8.2 and the runner's jq 1.6. Four independent cold review rounds (one cross-family Codex, three Opus cold-context, the last two in isolated worktrees); rounds 3 and 4 both MERGEABLE.

Deferred: the job is not a required status check — it reddens the run but does not block the merge button; promoting it is a branch-protection change, deliberately out of scope. Pinning or preflighting the runner's jq version is recorded as a follow-up on #647 — three version divergences in one day argue for it, and patching constructs one at a time does not scale.

Docs updated: docs/ci-cd.md (new script-tests section, corrected "three gates" → four, small-lane paragraph, dependency list), docs/decisions/records/ci/script-tests-job.md (new), docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Shipped in PR #637. `scripts/tests/` now runs on every PR as its own `script-tests` job in `pr-checks.yml`. New decision record `ci.script-tests-job`. **Root cause:** Nothing invoked `pytest`. `decisions-guard` runs `decisions_validate.py` and `build_decisions_catalog.py` *directly* — it exercises that code but never its tests — and the `test` job is `dotnet test` only. So the suite guarding the decision corpus, the #610 migration harness, the merge-consent logic and the #622 verdict poster was enforced solely by whoever remembered to run it locally. **Decisions/conventions changed:** added `ci.script-tests-job`. The load-bearing part is *where the job does not go*: the issue proposed a step inside `decisions-guard`, which is the one job covered by `ci.decisions-lifecycle-flake` ("a lone `decisions lifecycle` red is a known flake — do not investigate"). A real pytest regression there would surface as exactly the red every session is told to wave through. **A gate inherits the credibility of the job it lives in, so a job under a standing ignore-rule can host no real gate.** **Reusable knowledge — what switching the light on immediately found.** Four latent defects, none of them in this PR's own work: 1. The suite is **not stdlib-only** (`test_migration_equivalence.py` → PyYAML). It passed locally only because this machine had it. The complete third-party set came from an AST import scan, not from reading the files that looked relevant. 2. **#643** — a jq-1.6 fail-open in the merge-consent hook's docs-only enumeration. 3. **#647** — the H10 verdict classifier was *entirely inert* on jq 1.6 (`contains("�")` is true for every string there), and the same fail-open sat in `review-verdict.yml`, the **server-enforced** copy. 61 of 178 tests failed under jq 1.6 on `main`. 4. A record title whose unquoted ` #` was a YAML comment, so PyYAML and our hand reader disagreed — already red on `main`. The pattern: every one was invisible because the only thing that could see it never ran. **A test suite that runs nowhere is not a safety net, it is a document.** **Verification:** measured, not asserted. Throwaway PR #638 = this branch + one deliberately broken test → `Script tests (pytest)` **failure**, every other job green; this branch → **success**. 216 tests green under both jq 1.8.2 and the runner's jq 1.6. Four independent cold review rounds (one cross-family Codex, three Opus cold-context, the last two in isolated worktrees); rounds 3 and 4 both MERGEABLE. **Deferred:** the job is **not** a required status check — it reddens the run but does not block the merge button; promoting it is a branch-protection change, deliberately out of scope. Pinning or preflighting the runner's jq version is recorded as a follow-up on #647 — three version divergences in one day argue for it, and patching constructs one at a time does not scale. **Docs updated:** `docs/ci-cd.md` (new `script-tests` section, corrected "three gates" → four, `small`-lane paragraph, dependency list), `docs/decisions/records/ci/script-tests-job.md` (new), `docs/decisions/README.md` (regenerated).
timothy removed the in-progress label 2026-07-26 13:57:48 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#631