fix(631): run scripts/tests in CI as its own job, not inside the flake-covered decisions-guard #637

Merged
timothy merged 10 commits from fix/631-script-tests-ci into main 2026-07-26 13:56:45 +02:00
10 Commits
Author SHA1 Message Date
timothyandClaude Opus 5 37fd30dce7 fix(631): quote a record title whose unquoted # was a YAML comment
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 19s
PR Gates / Docs update reminder (pull_request) Successful in 20s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
Review verdict / Set review-verdict status (pull_request) Successful in 3s
PR Gates / Script tests (pytest) (pull_request) Successful in 37s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m29s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 20s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 15m41s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 15m57s
review-verdict/h10 Review-verdict: MERGEABLE @ 37fd30d
Third latent defect surfaced purely by running scripts/tests in CI — and this one was
already red on origin/main, where test_frontmatter_reader_matches_pyyaml_on_every_real_record
fails.

`api/paging-zero-based.md` carried an UNQUOTED plain scalar title ending
"... (#616, #633)". In YAML a space followed by `#` starts a COMMENT, so PyYAML reads the
title as truncated at "(#616," while the hand-written dependency-free reader keeps the
whole line. PyYAML is correct per spec; the record's title was genuinely losing "#633)"
for any spec-compliant consumer, which includes MemPalace's ingest.

Fixed by quoting the scalar; both parsers now agree byte-for-byte.

Asked "instance or class?" before fixing: swept every frontmatter field of all 179
records under both parsers and compared. This is the ONLY divergence in the corpus, so a
one-line fix is the right scope — no lint rule needed, and #621's structural guard
already makes a parse FAILURE loud (this was a parse-to-different, which no structural
check can see).

216 passed under jq 1.8.2 and jq 1.6.

Refs #631

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:27:05 +02:00
timothyandClaude Opus 5 bb1809fbf0 fix(647): the jq-1.6 fail-open was ALSO in review-verdict.yml — the enforced gate
Round-4 review caught the miss that matters most: #643 fixed the `jq -e`-on-empty
fail-open in .claude/hooks/pretooluse-merge-consent.sh, but the SAME construct sits in
.gitea/workflows/review-verdict.yml — and that is the copy that runs on the CI runner,
where jq is 1.6, and that feeds the branch-protection-required review-verdict/h10
status.

Reproduced: `printf "" | jq -e '.statuses | type == "array"'` exits 4 on jq 1.8.2 (guard
fires, correct) and 0 on jq 1.6 (guard passes). So on a transient API error `statusjson`
is empty, the guard lets it through, `existing` reads "", and the job posts `pending` —
or for a bot/docs-only PR an exemption `success` — over a possibly-existing human
verdict. That is precisely what the comment three lines above it says must never happen.

The hook version was harmless in practice because it runs on a dev Mac with jq 1.8. This
one is live. Fixed identically, with a comment naming why the sibling fix missed it, and
the same hardening applied to the changed-files read in the same workflow.

Also from round 4:

- LOW, reproduced — an ARRAY-valued `.status` dodged the closed allow-list. `index` is
  polymorphic: with an array argument it does SUBSEQUENCE matching, so
  `[...,"renamed",...] | index(["renamed"])` is truthy while `.status == "renamed"` is
  false — the row passed the allow-list AND skipped the `previous_filename REQUIRED`
  clause. The same `git mv code -> docs/` dodge the closed set exists to block, one type
  away. Now requires `.status` to be a string first; mutation-verified.

- The record now carries all THREE jq rules rather than the one, and notes that the
  durable fix is pinning/preflighting the runner's jq version rather than patching
  constructs one at a time (tracked on #647).

200 tests pass under BOTH jq 1.8.2 and jq 1.6.

Refs #647, #643, #631

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:24:20 +02:00
timothyandClaude Opus 5 7265fba36d fix(647): the H10 verdict classifier was inert on jq 1.6 — the runner's version
Turning on the scripts/tests suite in CI immediately paid for itself: measured on
origin/main, 61 of 178 tests FAIL under jq 1.6, which is what the CI runner ships. They
pass on a dev Mac's jq 1.8.2, which is why this was invisible — and the suite has never
run anywhere else, which is exactly #631's thesis.

Two defects in scripts/check-review-verdict.sh (from #629, the single source of truth
for H10 verdict classification):

1. `contains("<NUL>")` is TRUE FOR EVERY STRING on jq 1.6 — the escape truncates the
   literal to the empty string, and every string contains "". So the body guard errored
   "NUL in body" on every comment and the H10 grammar was entirely inert on the runner.
   Verified against both binaries: 1.6 says true for "hello", 1.7+ says false. Replaced
   with `(explode | index(0)) != null`, which involves no regex engine and agrees on
   both.

2. A parse error was indistinguishable from "no output". The script used jq's exit code
   to separate malformed input from a legitimately empty comment list, treating 4 as
   benign — but jq >= 1.7 exits 5 on a parse error while 1.6 exits 4, the same code both
   use for "filter produced no output". On 1.6 a garbage API response therefore returned
   `absent` instead of an input error. Fixed with an explicit `jq empty` pre-check, which
   is non-zero iff the input does not parse regardless of output volume.

Severity: fail-closed, not exploitable. The classifier is only invoked from the
merge-consent hook, which runs on the dev machine (jq 1.8.2), so the live gate is
unaffected. The cost is that #629's hardening was inert on the runner and would have
stayed invisible.

198 tests now pass under BOTH jq 1.8.2 and jq 1.6 (was 138/60 split under 1.6).

This is the third distinct jq-1.6 divergence found in this codebase today (the first was
#643's `jq -e` on empty input). The rule: a shell gate's behaviour is a function of its
interpreter's version — test against the version CI actually runs, or pin it.

Refs #647, #631

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:24:20 +02:00
timothyandClaude Opus 5 f4473926d4 fix(643): close the status-string dodge, pin the allow-list anchors, fix two doc claims
Round-3 review (MERGEABLE, all findings Low/Nit) — applied anyway, because each is the
over-claim or unpinned-anchor family this PR keeps hitting.

LOW — the `renamed => previous_filename REQUIRED` clause matched `.status` by exact
lowercase string, so any other value took the `else true` branch: a row with
`"Renamed"`, or with no status at all, validated fine and silently dropped its SOURCE
path, letting `git mv ErsatzTV/Program.cs -> docs/a.md` read as docs-only. `.status` is
now checked against a closed set and an unknown value fails closed.

Two things that fix taught me, both caught by my own positive control rather than by
review:
  1. The first predicate was WRONG in a way that gated everything: inside
     `[...] | index(.status)`, jq's `.` is the ARRAY, so `.status` was null and every
     row failed. `$s` is now bound from the row before the context switches. A
     security check that rejects everything looks identical to a working one from the
     failing side — only test_gitea_real_status_values_are_accepted caught it.
  2. The set includes BOTH `changed` and `modified`. Live Gitea 1.25.4 emits `changed`,
     but a closed allow-list built from the wrong vocabulary is worse than the hole it
     closes: it would gate every genuine docs-only PR. The property wanted is "reject
     what we don't recognise", not "enumerate one version exactly".

LOW — three allow-list anchors had no test at all: dropping `^` from the `docs/`
alternative (`ErsatzTV/docs/Program.cs` would exempt), dropping `$` from `.md`
(`x.md.cs` would exempt), and dropping the non-empty-list guard. Since the round-3 `..`
finding WAS an anchor subversion, they are now pinned; all three mutation-verified.

NIT — docs/ci-cd.md called this job "a checkout plus a pure-stdlib pytest run", which
the same file contradicts 450 lines later and which this PR's own record names as the
bug that turned the job red on its first CI run. Also replaced a frozen "111 tests" with
an explicitly indicative figure — the suite is ~190 after rebasing onto main, and a
frozen count has rotted four times this session.

198 passed under both jq 1.8.2 and jq 1.6.

Refs #643, #631

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 13:24:20 +02:00
timothy 54c875414c fix(643): validate every field the extraction consumes; fix a vacuous test
Re-review of the previous fix commit found it incomplete and its test vacuous. Both
reproduced before fixing.

MEDIUM — the validation domain did not match the CONSUMPTION domain. `chunk` emits
`(.previous_filename // empty)` for EVERY row regardless of status, but the guard
validated that field only when `.status == "renamed"`. So a row marked `modified` — or
Gitea's distinct `copied` — carrying a newline in previous_filename was still exempted.
Verified EXEMPT for both statuses against the previous commit. Now validated whenever
present, with the `renamed` => REQUIRED clause kept on top.

MEDIUM — test_newline_in_previous_filename_is_also_rejected asserted the right outcome
without exercising the mechanism: its payload's second segment was
`ErsatzTV/Program.cs`, which the allow-list rejects on its own, so the test passed with
the newline guard entirely removed. That is why the hole above went unnoticed — the
same filter-hides-the-defect trap the guard itself is about. Payload changed to a
segment that PASSES the allow-list, so the test now discriminates, plus parametrized
regressions for modified/copied/added.

LOW — `..` components rejected. The allow-list anchors `^docs/`, so
`docs/../ErsatzTV/Program.cs` matched it (reproduced). Git will not produce such a
path, but this guard exists to fail closed on unexpected 2xx shapes.

Added a positive control (a legitimate docs->docs rename still exempts) so the
tightened row schema cannot be satisfied by never exempting anything.

Severity calibrated in the record: the docs-only exemption ends in `decide allow ""`, a
passthrough to the normal permission prompt, NOT an auto-grant. Every bypass here
downgrades a mechanical deny/ask to a human prompt; none can silently self-merge. Real,
worth fixing, but not what an earlier framing of #643 implied.

All mutation-verified; 123 passed under BOTH jq 1.8.2 and jq 1.6.

Refs #643, #631
2026-07-26 13:24:20 +02:00
timothy c046add10a fix(643): close two more fail-opens in the docs-only enumeration, found by cold review
An independent cross-family review of the jq-1.6 fix found two further ways the
docs-only exemption can fire over an incomplete file list — both reachable with NO
transport error, so neither had anything to do with the original bug.

1. HIGH — a path containing a newline. `chunk` flattens paths into newline-delimited
   text before the allow-list grep, so a filename of "safe.md\ndocs/Program.cs" splits
   into two lines that BOTH match the allow-list, while the real single path ends in
   .cs. Git permits newlines in filenames and the reviewer reproduced the bypass
   against this hook. Now rejected outright at the row-schema guard, on both
   `filename` and `previous_filename`: no docs path contains a control character, so
   failing closed costs nothing.

2. HIGH — a short page read as the last page. `n < 50` assumed the server's page size
   is the 50 we requested, but Gitea caps `limit` at the server-wide
   MAX_RESPONSE_ITEMS (default 50, configurable) and may return fewer. A 30-row docs
   page followed by a page of code completed the enumeration over a PARTIAL list.
   Only a validated EMPTY page may now terminate it; the page<=40 cap still fails
   closed, and the cost is one extra request.

3. MEDIUM — the enumeration was not bound to one head. Paging is several round-trips,
   so a force-push between them assembles a list belonging to no single commit: page 1
   from head A plus a short docs tail from head B, with B's code page never read. The
   head sha is re-read after enumeration and the exemption refused if it moved.

All three mutation-verified: reverting each fix reddens exactly its own test and
nothing else. A positive control (short page then empty page) pins that the stricter
terminator still exempts a genuinely docs-only PR, so "never terminate early" cannot
be satisfied by never exempting anything. 118 passed under BOTH jq 1.8.2 and jq 1.6.

The record now states the generalisable lesson: every defect here was an
exhaustiveness failure in an enumeration whose completeness is load-bearing. When a
security decision depends on having seen ALL of something, the termination condition
must be positive and explicit, never inferred from a proxy.

Refs #643, #631
2026-07-26 13:24:20 +02:00
timothy 5f068a2488 fix(631): close a fail-open in the merge-consent gate that only CI could see
Running scripts/tests in CI for the first time turned up a real defect in the gate
itself, not just in the harness.

`jq -e` over EMPTY input exits 4 on jq >= 1.7 but 0 on jq 1.6 — verified against both
binaries, not inferred. The docs-only pagination guard in
pretooluse-merge-consent.sh leaned on that exit status to reject a transport failure.
On jq 1.6, which the CI runner ships:

  page 2 errors -> gq returns empty -> jq guard wrongly PASSES -> n is empty so
  [ "$n" -lt 50 ] errors into false -> the loop walks PAST the failed page -> page 3
  legitimately returns [] -> files_complete=yes over a PARTIAL list -> the docs-only
  exemption fires over unread pages that may be pure code.

That is the very defect the guard's own comment describes, reintroduced one layer down
by a jq version difference. Fixed by rejecting an empty body explicitly rather than
inferring it from jq's exit status. The same hardening is applied to the
review-verdict status read, which fell through to `vstate=""` -> deny (fail-CLOSED, so
never a hole) but would have surfaced the wrong message.

Why it survived: the existing transport-failure test asserts the right thing but can
only observe the bug where jq is 1.6, so it passes on a developer Mac with the bug
fully present — and the suite had never run anywhere else. The new test removes that
dependency by shimming ONLY jq 1.6's empty-input exit status, so it pins the property
on any host. Mutation-verified: revert the fix and it goes red on jq 1.8.2.

The shim is deliberately narrow (no `-n`): a broader first version swallowed the
`jq -n` calls `decide` uses to build its JSON, so the hook emitted nothing and every
decision read as a passthrough — the verifier manufacturing the exemption it was
meant to disprove. test_jq16_shim_actually_reproduces_the_quirk now pins the shim
itself, `-n` case included.

Suite: 113 passed under jq 1.8.2 AND under jq 1.6.

Refs #631
2026-07-26 13:24:20 +02:00
timothy 73577f484f ci(631): install pyyaml too — the suite is not pure-stdlib, and CI proved it
First CI run went red on `ModuleNotFoundError: No module named 'yaml'` at collection:
test_migration_equivalence.py imports scripts/migrate_decisions_split.py, which uses
PyYAML by design. It passed locally only because this machine already had PyYAML.

The original claim came from reading decisions_lib/decisions_validate/build_catalog
and the test files themselves — which is precisely the set that does NOT import yaml.
Replaced with an AST import scan over all of scripts/, which gives the complete
third-party set: pytest and yaml. (etv_client is imported only by
scripts/scripted-schedules/entrypoint.py, a container entrypoint pytest never
collects.)

This does not contradict the dependency-free decisions READ path:
decisions_lib._read_frontmatter is hand-written exactly so validation runs where
nothing is installed. The one-shot WRITE path is allowed PyYAML, and its test needs it.

Verified the way it should have been the first time: a clean venv with pytest alone
reproduces CI's collection error; adding pyyaml gives 111 passed. Comments, ci-cd.md
and the decision record all corrected to drop the "pure stdlib" claim.

Refs #631
2026-07-26 13:24:19 +02:00
timothy c0f4a52d7a ci(631): preflight jq/git in script-tests, and reflow the header comment
test_post_review_verdict.py and test_merge_consent_exemption.py exec the real
post-review-verdict.sh and pretooluse-merge-consent.sh, which shell out to jq ~26
times. Those tests shim `curl` on PATH but NOT `jq`, and no existing small-lane job
proves jq is present on that image — so a missing jq would land as ~20 opaque
assertion failures rather than a diagnosis.

Checks rather than installs: #390 deliberately removed run-time apt-get from CI, so
the correct fix for a genuine miss is baking the tool into the runner image, and the
error message says so.

Refs #631
2026-07-26 13:24:19 +02:00
timothy 69d8d3ccfe ci(631): run scripts/tests in CI as its own job, not inside the flake-covered decisions-guard
Nothing executed scripts/tests/. No workflow and no Husky hook invoked pytest:
decisions-guard runs decisions_validate.py and build_decisions_catalog.py directly
(the code, never its tests) and the `test` job is dotnet test only. The 111 tests
guarding the decision corpus, the #610 migration-equivalence harness, the
merge-consent exemption logic and the #622 review-verdict poster were enforced only
by whoever remembered to run pytest locally.

Adds a `script-tests` job to pr-checks.yml rather than the step inside decisions-guard
the issue proposed. decisions-guard is covered by ci.decisions-lifecycle-flake, a
standing instruction that a lone `decisions lifecycle` red is a known flake and must
not be investigated — so a genuine pytest regression there would surface as exactly
the red every session is told to wave through. A gate inherits the credibility of the
job it lives in.

Runs unconditionally: the suite executes the real post-review-verdict.sh and
pretooluse-merge-consent.sh, so a `scripts/**` path filter would miss a
.claude/hooks/** edit. ~10s, pure stdlib + pytest.

Negative control verified locally (broken test -> exit 1); CI-red measurement follows
on a scratch PR per the Done-when.

Refs #631
2026-07-26 13:24:19 +02:00