DO NOT MERGE — probe(631): negative control, proves script-tests can go red #638

Closed
timothy wants to merge 5 commits from probe/631-negative-control into main
5 Commits
Author SHA1 Message Date
timothy 52a8f16224 probe(631): deliberately broken test to measure that script-tests goes red in CI
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 9s
review-verdict/h10 Awaiting review verdict for 52a8f16
Review verdict / Set review-verdict status (pull_request) Successful in 3s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 15s
PR Gates / Script tests (pytest) (pull_request) Failing after 20s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m37s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 8s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 18m27s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 27m11s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Throwaway branch, never merged. Satisfies the #631 Done-when box requiring the
enforcement be verified by MEASUREMENT rather than a green tick.
2026-07-26 01:00:35 +02:00
timothy 6314646083 fix(631): close a fail-open in the merge-consent gate that only CI could see
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 15s
review-verdict/h10 Awaiting review verdict for 6314646
PR Gates / Docs update reminder (pull_request) Successful in 18s
Review verdict / Set review-verdict status (pull_request) Successful in 3s
PR Gates / decisions lifecycle (pull_request) Successful in 17s
PR Gates / Script tests (pytest) (pull_request) Successful in 22s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 20s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 16s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m35s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 23m56s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 26m57s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Running scripts/tests in CI for the first time turned up a real defect in the gate
itself, not just in the harness.

`jq -e` over EMPTY input exits 4 on jq >= 1.7 but 0 on jq 1.6 — verified against both
binaries, not inferred. The docs-only pagination guard in
pretooluse-merge-consent.sh leaned on that exit status to reject a transport failure.
On jq 1.6, which the CI runner ships:

  page 2 errors -> gq returns empty -> jq guard wrongly PASSES -> n is empty so
  [ "$n" -lt 50 ] errors into false -> the loop walks PAST the failed page -> page 3
  legitimately returns [] -> files_complete=yes over a PARTIAL list -> the docs-only
  exemption fires over unread pages that may be pure code.

That is the very defect the guard's own comment describes, reintroduced one layer down
by a jq version difference. Fixed by rejecting an empty body explicitly rather than
inferring it from jq's exit status. The same hardening is applied to the
review-verdict status read, which fell through to `vstate=""` -> deny (fail-CLOSED, so
never a hole) but would have surfaced the wrong message.

Why it survived: the existing transport-failure test asserts the right thing but can
only observe the bug where jq is 1.6, so it passes on a developer Mac with the bug
fully present — and the suite had never run anywhere else. The new test removes that
dependency by shimming ONLY jq 1.6's empty-input exit status, so it pins the property
on any host. Mutation-verified: revert the fix and it goes red on jq 1.8.2.

The shim is deliberately narrow (no `-n`): a broader first version swallowed the
`jq -n` calls `decide` uses to build its JSON, so the hook emitted nothing and every
decision read as a passthrough — the verifier manufacturing the exemption it was
meant to disprove. test_jq16_shim_actually_reproduces_the_quirk now pins the shim
itself, `-n` case included.

Suite: 113 passed under jq 1.8.2 AND under jq 1.6.

Refs #631
2026-07-26 01:00:28 +02:00
timothy b3d4759054 ci(631): install pyyaml too — the suite is not pure-stdlib, and CI proved it
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 22s
review-verdict/h10 Awaiting review verdict for b3d4759
PR Gates / Docs update reminder (pull_request) Successful in 26s
PR Gates / decisions lifecycle (pull_request) Successful in 26s
PR Gates / Script tests (pytest) (pull_request) Failing after 26s
Review verdict / Set review-verdict status (pull_request) Successful in 34s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m44s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 10s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m47s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 25m42s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
First CI run went red on `ModuleNotFoundError: No module named 'yaml'` at collection:
test_migration_equivalence.py imports scripts/migrate_decisions_split.py, which uses
PyYAML by design. It passed locally only because this machine already had PyYAML.

The original claim came from reading decisions_lib/decisions_validate/build_catalog
and the test files themselves — which is precisely the set that does NOT import yaml.
Replaced with an AST import scan over all of scripts/, which gives the complete
third-party set: pytest and yaml. (etv_client is imported only by
scripts/scripted-schedules/entrypoint.py, a container entrypoint pytest never
collects.)

This does not contradict the dependency-free decisions READ path:
decisions_lib._read_frontmatter is hand-written exactly so validation runs where
nothing is installed. The one-shot WRITE path is allowed PyYAML, and its test needs it.

Verified the way it should have been the first time: a clean venv with pytest alone
reproduces CI's collection error; adding pyyaml gives 111 passed. Comments, ci-cd.md
and the decision record all corrected to drop the "pure stdlib" claim.

Refs #631
2026-07-26 00:44:17 +02:00
timothy 8d3fbe215b ci(631): preflight jq/git in script-tests, and reflow the header comment
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 10s
PR Gates / Docs update reminder (pull_request) Successful in 12s
review-verdict/h10 Awaiting review verdict for 8d3fbe2
PR Gates / Script tests (pytest) (pull_request) Failing after 15s
Review verdict / Set review-verdict status (pull_request) Successful in 4s
PR Gates / decisions lifecycle (pull_request) Successful in 23s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Failing after 4m13s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m42s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 13s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m23s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
test_post_review_verdict.py and test_merge_consent_exemption.py exec the real
post-review-verdict.sh and pretooluse-merge-consent.sh, which shell out to jq ~26
times. Those tests shim `curl` on PATH but NOT `jq`, and no existing small-lane job
proves jq is present on that image — so a missing jq would land as ~20 opaque
assertion failures rather than a diagnosis.

Checks rather than installs: #390 deliberately removed run-time apt-get from CI, so
the correct fix for a genuine miss is baking the tool into the runner image, and the
error message says so.

Refs #631
2026-07-26 00:39:15 +02:00
timothy fedec84997 ci(631): run scripts/tests in CI as its own job, not inside the flake-covered decisions-guard
Nothing executed scripts/tests/. No workflow and no Husky hook invoked pytest:
decisions-guard runs decisions_validate.py and build_decisions_catalog.py directly
(the code, never its tests) and the `test` job is dotnet test only. The 111 tests
guarding the decision corpus, the #610 migration-equivalence harness, the
merge-consent exemption logic and the #622 review-verdict poster were enforced only
by whoever remembered to run pytest locally.

Adds a `script-tests` job to pr-checks.yml rather than the step inside decisions-guard
the issue proposed. decisions-guard is covered by ci.decisions-lifecycle-flake, a
standing instruction that a lone `decisions lifecycle` red is a known flake and must
not be investigated — so a genuine pytest regression there would surface as exactly
the red every session is told to wave through. A gate inherits the credibility of the
job it lives in.

Runs unconditionally: the suite executes the real post-review-verdict.sh and
pretooluse-merge-consent.sh, so a `scripts/**` path filter would miss a
.claude/hooks/** edit. ~10s, pure stdlib + pytest.

Negative control verified locally (broken test -> exit 1); CI-red measurement follows
on a scratch PR per the Done-when.

Refs #631
2026-07-26 00:32:57 +02:00