Pin or preflight jq in CI: the runner ships 1.6, dev machines ship 1.8.x, and that gap silently broke three shell gates #648

Closed
opened 2026-07-26 20:21:27 +02:00 by timothy · 2 comments
Owner

Follow-up from #647 (closed). Recorded there as "the durable fix", but a deferral living only in a closed issue's comment is not tracked — filing it so it exists in the queue.

The problem

Every shell gate in this repo is written and tested against a developer Mac's jq 1.8.x. The CI runner ships jq 1.6. Nothing pinned or checked that, and until #631 the one thing that could have noticed (scripts/tests/) never ran on the runner.

Three independent divergences were found in a single day:

# construct jq 1.8.2 jq 1.6 consequence
#643 jq -e over empty input exit 4 exit 0 a transport failure passed the docs-only pagination guard
#647 contains("<NUL>") false true for every string the H10 verdict classifier was entirely inert
#647 parse error exit code 5 4 — same as "no output" garbage API response read as "no comments"

All three are now fixed with version-stable constructs, and the suite passes under both. But patching constructs one at a time does not scale: the next author reaching for a natural jq idiom has nothing telling them which side of 1.7 they are writing for.

Why this is systemic, not three bugs

The failures share one shape: a shell gate's behaviour is a function of its interpreter's version, and that version was an untested axis. Fixing the three known instances leaves the axis untested. A fourth divergence is a matter of time — jq 1.7 changed NUL handling, exit codes, @base64d, number precision and more.

Options (pick one; not obviously equivalent)

  1. Bake a pinned jq into the CI toolchain image (docker/ci/Dockerfile) and use it everywhere. Makes CI deterministic, but the small lane jobs run without the toolchain image (ci.small-lane-git-only), so they would still get the host's 1.6 — this needs checking before assuming it solves it.
  2. Preflight a minimum version in the jobs that run shell gates — script-tests already preflights jq/git presence, so extending it to jq --version >= 1.7 is a few lines. Fails loudly rather than silently diverging, but does not make the two environments equal.
  3. Test against both, as this session did by hand: download a real 1.6 and run the suite under it. Highest fidelity, slowest, and needs a home in CI rather than a human remembering.

My weak preference is (2) then (1) — a loud floor first, since it is cheap and immediately removes the silent-divergence class, then pinning if the lane analysis supports it. Deliberately not deciding here.

Done-when

  • Decide which option (or combination), with the small-lane constraint checked rather than assumed
  • Implemented, and the jq version CI actually uses is observable in the job log
  • A divergence is loud: verified by measurement (e.g. temporarily assert a version the runner does not have, and confirm the job goes red) — not by a green tick
  • docs/ci-cd.md states the jq contract, and ci.script-tests-job's three jq rules point at it
  • Adversarial review passed
Follow-up from #647 (closed). Recorded there as "the durable fix", but a deferral living only in a closed issue's comment is not tracked — filing it so it exists in the queue. ## The problem Every shell gate in this repo is written and tested against a developer Mac's **jq 1.8.x**. The CI runner ships **jq 1.6**. Nothing pinned or checked that, and until #631 the one thing that could have noticed (`scripts/tests/`) never ran on the runner. Three independent divergences were found in a single day: | # | construct | jq 1.8.2 | jq 1.6 | consequence | |---|---|---|---|---| | #643 | `jq -e` over **empty** input | exit 4 | **exit 0** | a transport failure passed the docs-only pagination guard | | #647 | `contains("<NUL>")` | `false` | **`true` for every string** | the H10 verdict classifier was *entirely inert* | | #647 | parse error exit code | 5 | **4** — same as "no output" | garbage API response read as "no comments" | All three are now fixed with version-stable constructs, and the suite passes under both. But **patching constructs one at a time does not scale**: the next author reaching for a natural jq idiom has nothing telling them which side of 1.7 they are writing for. ## Why this is systemic, not three bugs The failures share one shape: *a shell gate's behaviour is a function of its interpreter's version, and that version was an untested axis.* Fixing the three known instances leaves the axis untested. A fourth divergence is a matter of time — jq 1.7 changed NUL handling, exit codes, `@base64d`, number precision and more. ## Options (pick one; not obviously equivalent) 1. **Bake a pinned jq into the CI toolchain image** (`docker/ci/Dockerfile`) and use it everywhere. Makes CI deterministic, but the `small` lane jobs run *without* the toolchain image (`ci.small-lane-git-only`), so they would still get the host's 1.6 — this needs checking before assuming it solves it. 2. **Preflight a minimum version** in the jobs that run shell gates — `script-tests` already preflights `jq`/`git` presence, so extending it to `jq --version >= 1.7` is a few lines. Fails loudly rather than silently diverging, but does not make the two environments equal. 3. **Test against both**, as this session did by hand: download a real 1.6 and run the suite under it. Highest fidelity, slowest, and needs a home in CI rather than a human remembering. My weak preference is (2) then (1) — a loud floor first, since it is cheap and immediately removes the silent-divergence class, then pinning if the lane analysis supports it. Deliberately not deciding here. ## Done-when - [x] Decide which option (or combination), with the `small`-lane constraint checked rather than assumed - [x] Implemented, and the jq version CI actually uses is *observable* in the job log - [x] A divergence is loud: verified by measurement (e.g. temporarily assert a version the runner does not have, and confirm the job goes red) — not by a green tick - [x] `docs/ci-cd.md` states the jq contract, and `ci.script-tests-job`'s three jq rules point at it - [x] Adversarial review passed
timothy added the ci-cdpriority: medium labels 2026-07-26 20:21:27 +02:00
timothy added the in-progress label 2026-07-26 20:23:44 +02:00
Author
Owner

Claiming — bundled with #644 this session as the ci-cd sibling cluster (both tracked follow-ups from #647, both shell-gate correctness). Worked in a worktree disjoint from the #644 SPA slice.

Claiming — bundled with #644 this session as the ci-cd sibling cluster (both tracked follow-ups from #647, both shell-gate correctness). Worked in a worktree disjoint from the #644 SPA slice.
timothy removed the in-progress label 2026-07-26 22:46:18 +02:00
Author
Owner

Closing record

Outcome: Merged as PR #658. scripts/jq-preflight.sh makes the jq version observable in every
gate job's log and loud when it drifts; scripts/pr-changed-files.sh is the single PR-file
enumeration both merge-gate callers use. #649 stays OPEN — its enforced half (review-verdict.yml)
is a deliberate follow-up; see Deferred.

Root cause: The three divergences in #643/#647 were not three bugs. They shared one shape: a
shell gate's behaviour is a function of its interpreter's version, and that version was an untested
axis.
Gates are authored on Macs with jq 1.8.x; the runner ships 1.6. Nothing pinned or checked
that, and until #631 the one job that could have noticed (scripts/tests/) never ran on the runner.
Patching constructs one at a time left the axis itself untested.

Decisions/conventions changed: added ci.jq-version-contract and
ci.shared-pr-file-enumeration; ci.script-tests-job now points at the contract instead of
restating its three jq rules.

Reusable knowledge:

  • The obvious fix would have missed the bug. Baking a pinned jq into docker/ci/Dockerfile
    cannot cover review-verdict.yml: it is runs-on: small, carries no toolchain-image pin, and per
    ci.small-lane-git-only the small lane is git-only, so it gets the host's jq. That is the exact
    workflow where the #647 bug was live. Checked, not assumed — which is what the issue asked for.
  • An "advisory" red is not free. The first draft of the decision record justified pinning
    script-tests on the grounds that it is not a required check. Wrong: the merge-consent hook reads
    the COMBINED status (ci.advisory-red-blocks-the-merge-gate, #598), so firing the tripwire blocks
    the hook-mediated merge path repo-wide. The pin is kept deliberately — the fix is a one-line edit
    and silent loss of the only 1.6 coverage is worse — but it is no longer described as costless.
  • A string-matching predicate took FIVE rounds, and rounds 2-4 each broke the previous fix. They
    share one signature worth memorising: the guard appears to assert something, but a crafted input
    makes the assertion silently not happen
    — usually a shell builtin erroring inside an if
    condition, which set -e exempts. Round 1: empty major -> [ "" -lt 1 ] errors -> floor never
    asserted, exit 0. Round 2 (a regression): 2>&1 || true folded stderr into the parse and dropped
    the exit status, so a glibc error containing 2.34 PASSED the floor, where round 1 had failed
    closed. Round 3: [[:space:]] matches newlines, so the "anchor" still scanned everything; and an
    over-long digit run resurrected round 1's exact mechanism. Round 4: {1,9} bounded nothing on its
    own, because the pattern was unanchored at the end.
  • Two of my own tests passed for the wrong reason, in two different ways. test_missing_jq_is_loud
    passed while PATH still exposed the Mac's real /usr/bin/jq. And the CRLF test was unfalsifiable
    through two independent mechanisms: str.splitlines() also splits on \r, and
    subprocess.run(text=True) rewrites \r to \n outright — so an assertion about a stray
    carriage return could never fail. Both were found by mutation, never by reading the code.
  • Shared MECHANISM, not shared policy. The two callers' docs-only allow-lists differ on purpose
    (the hook's is broader because it falls through to a human prompt; the workflow's is narrower
    because a match posts a green status with nobody in the loop). Extracting the enumeration fixed the
    drift; extracting the classification would have erased an intended difference.

Verification: scripts/tests 292 passed. Ten enumeration guards and six preflight guards
individually mutated, each turning exactly its own test red, file restored byte-identical each time.
An independent reviewer fetched a real jq 1.6, ran the suite under it, and diffed the row-validation
program across 1.6 vs 1.8 on 22 payloads — identical on every case except the two already
neutralised. Another ran 28,930 real invocations across 14,465 crafted --version strings on bash
3.2.57 and 5.3.15 with zero fail-opens, and confirmed {1,9} interval quantifiers are honoured on
bash 3.2. Full CI green on b255b7ff.

The runner's own log closes the observability box directly:

jq-preflight: jq version in use = jq-1.6 (parsed 1.6; floor 1.6)
jq-preflight: version matches the expected pin (1.6).

That is also the first empirical confirmation that the runner ships 1.6 (previously inferred from
the #647 symptom) and that the parser behaves under Linux/glibc regex, which the reviewers could only
test against macOS libc.

Deferred:

  • #649 remains open. review-verdict.yml is untouched here. It will check out the PR's BASE ref
    — never the head, so a PR cannot rewrite the gate that judges it — and the base is main, which
    did not contain these scripts until this merged. Wiring it in the same PR would have made the job
    exit 127 on its own PR and blocked the merge gate via the combined status. The follow-up carries a
    known env trap: the workflow exports BASE_URL and a combined owner/repo in REPO, while the
    script reads GITEA_BASE_URL and separate owner/repo args; unadapted it silently falls back to the
    hardcoded LAN address instead of erroring.
  • The Done-when box asking to prove loudness by asserting a version the runner lacks and watching the
    job go red was satisfied as a unit test (test_jq_preflight.py shims jq on PATH) rather than by
    pushing a deliberately-red commit. Stated rather than glossed: the substance is stronger — it
    re-runs on every PR instead of living in one run's history — but it is not literally the method the
    box describes.

Docs updated: docs/ci-cd.md ("The jq contract"), docs/decisions/records/ci/jq-version-contract.md
(new), docs/decisions/records/ci/shared-pr-file-enumeration.md (new),
docs/decisions/records/ci/script-tests-job.md, docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Merged as PR #658. `scripts/jq-preflight.sh` makes the jq version observable in every gate job's log and loud when it drifts; `scripts/pr-changed-files.sh` is the single PR-file enumeration both merge-gate callers use. #649 stays OPEN — its enforced half (`review-verdict.yml`) is a deliberate follow-up; see Deferred. **Root cause:** The three divergences in #643/#647 were not three bugs. They shared one shape: *a shell gate's behaviour is a function of its interpreter's version, and that version was an untested axis.* Gates are authored on Macs with jq 1.8.x; the runner ships 1.6. Nothing pinned or checked that, and until #631 the one job that could have noticed (`scripts/tests/`) never ran on the runner. Patching constructs one at a time left the axis itself untested. **Decisions/conventions changed:** added `ci.jq-version-contract` and `ci.shared-pr-file-enumeration`; `ci.script-tests-job` now points at the contract instead of restating its three jq rules. **Reusable knowledge:** - **The obvious fix would have missed the bug.** Baking a pinned jq into `docker/ci/Dockerfile` cannot cover `review-verdict.yml`: it is `runs-on: small`, carries no toolchain-image pin, and per `ci.small-lane-git-only` the small lane is git-only, so it gets the *host's* jq. That is the exact workflow where the #647 bug was live. Checked, not assumed — which is what the issue asked for. - **An "advisory" red is not free.** The first draft of the decision record justified pinning `script-tests` on the grounds that it is not a required check. Wrong: the merge-consent hook reads the COMBINED status (`ci.advisory-red-blocks-the-merge-gate`, #598), so firing the tripwire blocks the hook-mediated merge path repo-wide. The pin is kept deliberately — the fix is a one-line edit and silent loss of the only 1.6 coverage is worse — but it is no longer described as costless. - **A string-matching predicate took FIVE rounds, and rounds 2-4 each broke the previous fix.** They share one signature worth memorising: *the guard appears to assert something, but a crafted input makes the assertion silently not happen* — usually a shell builtin erroring inside an `if` condition, which `set -e` exempts. Round 1: empty major -> `[ "" -lt 1 ]` errors -> floor never asserted, exit 0. Round 2 (a regression): `2>&1 || true` folded stderr into the parse and dropped the exit status, so a glibc error containing `2.34` PASSED the floor, where round 1 had failed closed. Round 3: `[[:space:]]` matches newlines, so the "anchor" still scanned everything; and an over-long digit run resurrected round 1's exact mechanism. Round 4: `{1,9}` bounded nothing on its own, because the pattern was unanchored at the end. - **Two of my own tests passed for the wrong reason, in two different ways.** `test_missing_jq_is_loud` passed while PATH still exposed the Mac's real `/usr/bin/jq`. And the CRLF test was unfalsifiable through *two* independent mechanisms: `str.splitlines()` also splits on `\r`, and `subprocess.run(text=True)` rewrites `\r` to `\n` outright — so an assertion about a stray carriage return could never fail. Both were found by mutation, never by reading the code. - **Shared MECHANISM, not shared policy.** The two callers' docs-only allow-lists differ on purpose (the hook's is broader because it falls through to a human prompt; the workflow's is narrower because a match posts a green status with nobody in the loop). Extracting the enumeration fixed the drift; extracting the classification would have erased an intended difference. **Verification:** `scripts/tests` 292 passed. Ten enumeration guards and six preflight guards individually mutated, each turning exactly its own test red, file restored byte-identical each time. An independent reviewer fetched a real jq 1.6, ran the suite under it, and diffed the row-validation program across 1.6 vs 1.8 on 22 payloads — identical on every case except the two already neutralised. Another ran 28,930 real invocations across 14,465 crafted `--version` strings on bash 3.2.57 and 5.3.15 with zero fail-opens, and confirmed `{1,9}` interval quantifiers *are* honoured on bash 3.2. Full CI green on `b255b7ff`. The runner's own log closes the observability box directly: ``` jq-preflight: jq version in use = jq-1.6 (parsed 1.6; floor 1.6) jq-preflight: version matches the expected pin (1.6). ``` That is also the first empirical confirmation that the runner ships 1.6 (previously inferred from the #647 symptom) and that the parser behaves under Linux/glibc regex, which the reviewers could only test against macOS libc. **Deferred:** - **#649 remains open.** `review-verdict.yml` is untouched here. It will check out the PR's BASE ref — never the head, so a PR cannot rewrite the gate that judges it — and the base is `main`, which did not contain these scripts until this merged. Wiring it in the same PR would have made the job exit 127 on its own PR and blocked the merge gate via the combined status. The follow-up carries a known env trap: the workflow exports `BASE_URL` and a combined `owner/repo` in `REPO`, while the script reads `GITEA_BASE_URL` and separate owner/repo args; unadapted it silently falls back to the hardcoded LAN address instead of erroring. - The Done-when box asking to prove loudness by asserting a version the runner lacks and watching the job go red was satisfied as a unit test (`test_jq_preflight.py` shims `jq` on PATH) rather than by pushing a deliberately-red commit. Stated rather than glossed: the substance is stronger — it re-runs on every PR instead of living in one run's history — but it is not literally the method the box describes. **Docs updated:** `docs/ci-cd.md` ("The jq contract"), `docs/decisions/records/ci/jq-version-contract.md` (new), `docs/decisions/records/ci/shared-pr-file-enumeration.md` (new), `docs/decisions/records/ci/script-tests-job.md`, `docs/decisions/README.md` (regenerated).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#648