Pin or preflight jq in CI: the runner ships 1.6, dev machines ship 1.8.x, and that gap silently broke three shell gates #648
Closed
opened 2026-07-26 20:21:27 +02:00 by timothy
·
2 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#648
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Follow-up from #647 (closed). Recorded there as "the durable fix", but a deferral living only in a closed issue's comment is not tracked — filing it so it exists in the queue.
The problem
Every shell gate in this repo is written and tested against a developer Mac's jq 1.8.x. The CI runner ships jq 1.6. Nothing pinned or checked that, and until #631 the one thing that could have noticed (
scripts/tests/) never ran on the runner.Three independent divergences were found in a single day:
jq -eover empty inputcontains("<NUL>")falsetruefor every stringAll three are now fixed with version-stable constructs, and the suite passes under both. But patching constructs one at a time does not scale: the next author reaching for a natural jq idiom has nothing telling them which side of 1.7 they are writing for.
Why this is systemic, not three bugs
The failures share one shape: a shell gate's behaviour is a function of its interpreter's version, and that version was an untested axis. Fixing the three known instances leaves the axis untested. A fourth divergence is a matter of time — jq 1.7 changed NUL handling, exit codes,
@base64d, number precision and more.Options (pick one; not obviously equivalent)
docker/ci/Dockerfile) and use it everywhere. Makes CI deterministic, but thesmalllane jobs run without the toolchain image (ci.small-lane-git-only), so they would still get the host's 1.6 — this needs checking before assuming it solves it.script-testsalready preflightsjq/gitpresence, so extending it tojq --version >= 1.7is a few lines. Fails loudly rather than silently diverging, but does not make the two environments equal.My weak preference is (2) then (1) — a loud floor first, since it is cheap and immediately removes the silent-divergence class, then pinning if the lane analysis supports it. Deliberately not deciding here.
Done-when
small-lane constraint checked rather than assumeddocs/ci-cd.mdstates the jq contract, andci.script-tests-job's three jq rules point at itClaiming — bundled with #644 this session as the ci-cd sibling cluster (both tracked follow-ups from #647, both shell-gate correctness). Worked in a worktree disjoint from the #644 SPA slice.
Closing record
Outcome: Merged as PR #658.
scripts/jq-preflight.shmakes the jq version observable in everygate job's log and loud when it drifts;
scripts/pr-changed-files.shis the single PR-fileenumeration both merge-gate callers use. #649 stays OPEN — its enforced half (
review-verdict.yml)is a deliberate follow-up; see Deferred.
Root cause: The three divergences in #643/#647 were not three bugs. They shared one shape: a
shell gate's behaviour is a function of its interpreter's version, and that version was an untested
axis. Gates are authored on Macs with jq 1.8.x; the runner ships 1.6. Nothing pinned or checked
that, and until #631 the one job that could have noticed (
scripts/tests/) never ran on the runner.Patching constructs one at a time left the axis itself untested.
Decisions/conventions changed: added
ci.jq-version-contractandci.shared-pr-file-enumeration;ci.script-tests-jobnow points at the contract instead ofrestating its three jq rules.
Reusable knowledge:
docker/ci/Dockerfilecannot cover
review-verdict.yml: it isruns-on: small, carries no toolchain-image pin, and perci.small-lane-git-onlythe small lane is git-only, so it gets the host's jq. That is the exactworkflow where the #647 bug was live. Checked, not assumed — which is what the issue asked for.
script-testson the grounds that it is not a required check. Wrong: the merge-consent hook readsthe COMBINED status (
ci.advisory-red-blocks-the-merge-gate, #598), so firing the tripwire blocksthe hook-mediated merge path repo-wide. The pin is kept deliberately — the fix is a one-line edit
and silent loss of the only 1.6 coverage is worse — but it is no longer described as costless.
share one signature worth memorising: the guard appears to assert something, but a crafted input
makes the assertion silently not happen — usually a shell builtin erroring inside an
ifcondition, which
set -eexempts. Round 1: empty major ->[ "" -lt 1 ]errors -> floor neverasserted, exit 0. Round 2 (a regression):
2>&1 || truefolded stderr into the parse and droppedthe exit status, so a glibc error containing
2.34PASSED the floor, where round 1 had failedclosed. Round 3:
[[:space:]]matches newlines, so the "anchor" still scanned everything; and anover-long digit run resurrected round 1's exact mechanism. Round 4:
{1,9}bounded nothing on itsown, because the pattern was unanchored at the end.
test_missing_jq_is_loudpassed while PATH still exposed the Mac's real
/usr/bin/jq. And the CRLF test was unfalsifiablethrough two independent mechanisms:
str.splitlines()also splits on\r, andsubprocess.run(text=True)rewrites\rto\noutright — so an assertion about a straycarriage return could never fail. Both were found by mutation, never by reading the code.
(the hook's is broader because it falls through to a human prompt; the workflow's is narrower
because a match posts a green status with nobody in the loop). Extracting the enumeration fixed the
drift; extracting the classification would have erased an intended difference.
Verification:
scripts/tests292 passed. Ten enumeration guards and six preflight guardsindividually mutated, each turning exactly its own test red, file restored byte-identical each time.
An independent reviewer fetched a real jq 1.6, ran the suite under it, and diffed the row-validation
program across 1.6 vs 1.8 on 22 payloads — identical on every case except the two already
neutralised. Another ran 28,930 real invocations across 14,465 crafted
--versionstrings on bash3.2.57 and 5.3.15 with zero fail-opens, and confirmed
{1,9}interval quantifiers are honoured onbash 3.2. Full CI green on
b255b7ff.The runner's own log closes the observability box directly:
That is also the first empirical confirmation that the runner ships 1.6 (previously inferred from
the #647 symptom) and that the parser behaves under Linux/glibc regex, which the reviewers could only
test against macOS libc.
Deferred:
review-verdict.ymlis untouched here. It will check out the PR's BASE ref— never the head, so a PR cannot rewrite the gate that judges it — and the base is
main, whichdid not contain these scripts until this merged. Wiring it in the same PR would have made the job
exit 127 on its own PR and blocked the merge gate via the combined status. The follow-up carries a
known env trap: the workflow exports
BASE_URLand a combinedowner/repoinREPO, while thescript reads
GITEA_BASE_URLand separate owner/repo args; unadapted it silently falls back to thehardcoded LAN address instead of erroring.
job go red was satisfied as a unit test (
test_jq_preflight.pyshimsjqon PATH) rather than bypushing a deliberately-red commit. Stated rather than glossed: the substance is stronger — it
re-runs on every PR instead of living in one run's history — but it is not literally the method the
box describes.
Docs updated:
docs/ci-cd.md("The jq contract"),docs/decisions/records/ci/jq-version-contract.md(new),
docs/decisions/records/ci/shared-pr-file-enumeration.md(new),docs/decisions/records/ci/script-tests-job.md,docs/decisions/README.md(regenerated).