timothy
b255b7ffdc
Round 5 could not break the predicate itself: 28,930 real runs of the script across
14,465 crafted --version strings on bash 3.2.57 and 5.3.15 produced zero fail-opens, and
`{1,9}` is honoured on bash 3.2, so round 4's bound is not void on the authoring Macs.
What it did find is that two of round 4's changes were unpinned, and the tests that
looked like they covered them did not.
Reverting BOTH the first-line slice and `[[:blank:]]`→`[[:space:]]` together left the whole
suite green. The four filler cases are all killed by the SEPARATOR restriction alone, so
they attributed the fix to the wrong layer. Added three cases carrying the literal word
`version` (`jq\nversion\n9.9` and friends), which satisfy the separator rule and can only be
stopped by confining the parse to line one with a newline-free blank class.
The CR-strip test was worse: vacuous through two independent mechanisms. `str.splitlines()`
also splits on `\r`, so a per-line view dropped the stray CR; and `subprocess.run(text=True)`
translates `\r` to `\n` outright, so even a raw-string check on stdout was unfalsifiable.
The mutant demonstrably emits `... = jq-1.6<CR> (parsed 1.6; ...)` at the byte level while
the test reported green. Added `run_bytes()` and a bytes comparison.
Both gaps are now mutation-verified: reverting either change reddens exactly its own test.
Also records the operational edge this parser acquires in the follow-up: it is strictly
fail-closed by design, so once the floor mode gates the required check, a jq wrapper that
prints a banner line would deadlock merges. The fix there is to widen the accepted forms,
never to relax fail-closed.
Decisions-Edit: yes
8.4 KiB
key, title, status, since, supersedes, superseded-by, rule, signals, mechanics
| key | title | status | since | supersedes | superseded-by | rule | signals | mechanics |
|---|---|---|---|---|---|---|---|---|
| ci.jq-version-contract | 2026-07-26 — jq 1.6 is the FLOOR every shell gate must run on; `scripts/jq-preflight.sh` makes the version observable, and only `script-tests` pins it (#648) | active | 2026-07-26 | none | none | Every shell gate that shells out to `jq` is authored to the jq 1.6-compatible subset, because the CI runner ships jq 1.6 while every developer Mac ships 1.8.x. `scripts/jq-preflight.sh` (no args) prints the parsed version and asserts a floor of 1.6 in every gate job's log; `scripts/jq-preflight.sh --expect 1.6` additionally pins and fails loudly, but ONLY in the `script-tests` job. `review-verdict.yml` never pins — it writes the branch-protection-required `review-verdict/h10` status, so a hard pin there would turn any jq bump into a repo-wide merge deadlock. | jq version divergence, jq 1.6 vs 1.8, jq -e exit code on empty input, contains NUL false positive, jq parse-error exit code collision, jq-preflight, script-tests --expect, review-verdict jq floor, merge deadlock from a pinned dependency · paths: `scripts/jq-preflight.sh`, `.gitea/workflows/review-verdict.yml`, `.gitea/workflows/pr-checks.yml`, `docs/ci-cd.md` · issues: #643, #647, #648, #649 | `scripts/jq-preflight.sh` (no args) -> floor+observability in every gate job; `scripts/jq-preflight.sh --expect 1.6` -> tripwire, `script-tests` job only; `docs/ci-cd.md` -> "The jq contract" |
Three independent jq-version divergences hit inside a single day (#643, #647), all in gates written and tested on a developer Mac (jq 1.8.x) but running on the CI runner (jq 1.6):
jq -eover EMPTY input exits 4 on jq >= 1.7, but 0 on jq 1.6 — the docs-only pagination guard inferred "transport failure" from that exit status, so on 1.6 a failed page silently passed and the loop walked past unread pages while still reportingfiles_complete=yes.contains("\u0000")— the NUL escape truncates to""on jq 1.6, so the containment test is vacuously true for every string, not just ones actually containing a NUL. The H10 review-verdict classifier that relied on this was entirely inert on the runner.- Parse-error exit code:
jq emptyexits 5 on jq >= 1.7 but 4 on jq 1.6 — the same code jq 1.6 uses for "no output produced". A garbage API response and an empty-but-valid one were indistinguishable, and the garbage case was read as "no comments."
None of these are exotic jq usage — they are constructs anyone would reach for first, and each one was discovered only because a real gate broke, not because anyone thought to test jq 1.6. That is the argument for a contract, not three point fixes: the failures share one shape (a shell gate's behavior is a function of an interpreter version nobody was treating as a variable), so patching each construct as it's found does not converge — it just narrows the next surprise.
Why 1.6 is the floor and not 1.8. The runner is the binding constraint, not the author's machine.
Baking a pinned jq into docker/ci/Dockerfile was the obvious first idea and was rejected because it
provably cannot cover the gate that actually broke: review-verdict.yml is runs-on: small with no
toolchain-image pin, and per ci.small-lane-git-only the small lane is git-only — it gets the host's
jq 1.6 no matter what the toolchain image contains. That was checked against the running binary, not
assumed. So the fix has to hold at 1.6, in every gate, regardless of which lane it runs in.
Why the pin is asymmetric. scripts/jq-preflight.sh has two modes on purpose:
-
No arguments — print the parsed version and fail only below the 1.6 floor. This is pure observability: the jq version CI actually used is now in the job log, so a future divergence can be diagnosed from the log alone instead of by guessing at the runner image.
review-verdict.ymlruns this mode. It cannot run the pinned mode, because that job's output is the requiredreview-verdict/h10status check onmain— a hard version pin there means the day the runner's jq is upgraded (a base-image bump, a host reimage, anything outside this repo's control), every PR onmainstops merging until someone notices and re-pins. A required merge gate cannot have a failure mode that is "an upstream package manager did its job." -
--expect 1.6— pin and fail loudly. Used only byscript-tests.scripts/tests/currently exercises the 1.6 code path only because the runner happens to ship 1.6; if that silently changed, the 1.6 coverage this whole contract depends on would evaporate with no signal. The tripwire forces a human decision — re-pin after re-reading this record, or add a real 1.6 matrix leg — instead of letting the coverage quietly disappear.Be honest about the cost: firing this tripwire DOES block merges. An earlier draft of this record claimed the pin was safe because
script-testsis "advisory, not one of the required checks". That reasoning is wrong, and the correction is worth recording because it is easy to make twice..claude/hooks/pretooluse-merge-consent.shreads the combined commit status and denies on anything that is notsuccess/skipped— seeci.advisory-red-blocks-the-merge-gate(#598).script-testsis a Gitea Actions job, so its red is a context folded into that combined state. A jq bump therefore reddensscript-testsand blocks non-docs-only merges until someone re-pins.One qualification, so this does not over-correct in the other direction: that combined-status read is guarded by
if [ "$mwcs" != "true" ]. On themerge_when_checks_succeedpath the hook does not read the combined status at all and defers to Gitea, which gates on required checks only — andscript-testsis not one. So the blast radius is the hook-mediated merge path, not literally every merge.The pin is kept anyway, deliberately: the fix is a one-line edit to the
--expectvalue inpr-checks.yml, the failure message spells that out, and the alternative — silently losing the only coverage of the version axis that produced three bugs in one day — is worse than a visible stop. What is NOT acceptable is believing it is free. The difference fromreview-verdict.ymlis therefore one of degree and recoverability, not of "blocks merges vs doesn't": there the check is required per-sha and a jq bump would deadlock merges with no in-repo remedy at all, whereas here a human can unblock the repo in one commit.
The parse is strictly fail-closed, and that has an operational edge once it gates merges.
scripts/jq-preflight.sh accepts only a FIRST line of the form jq-<X>.<Y> or jq version <X>.<Y>;
anything else — a leading blank line, a wrapper that prints a warning first, a version reported only
on stderr — exits 1 rather than guess. That is the right default for a guard whose whole purpose is
refusing to certify a version it did not parse, and it was arrived at over four revisions in which
every permissive variant turned out to be fail-OPEN.
But when the follow-up wires the floor-only mode into review-verdict.yml, that strictness sits in
the branch-protection-required check. A jq wrapper that starts printing a banner line would then
deadlock merges repo-wide — the very failure the pin/floor asymmetry exists to avoid, arriving
through the parser instead of the pin. If that ever happens the fix is to widen the accepted forms in
jq-preflight.sh, not to relax the fail-closed behaviour: an unparsed version must never be
treated as satisfying the floor.
The three constructs to avoid, and their version-stable replacements:
- Never infer "empty input" from a jq exit status — check the string in shell before invoking jq.
- Never use
contains("\u0000")(or any raw NUL literal) for a control-character test — useexplode | index(0), which does not depend on jq's NUL-escape handling. - Never infer "parse error" from
jq's exit code on ambiguous input —jq emptyis the portable parse-only test, but its exit code collides between "no output" (1.6) and "parse error" (1.6, same code as 1.8's "no output"). Validate the response shape explicitly rather than reading one exit code as a specific failure mode.
Full narrative of how these were found (inside the #631 script-tests rollout) is in
docs/decisions/records/ci/script-tests-job.md; this record is the durable contract that came out of
it, rather than the incident log.