script-tests jq tripwire: runner fleet now ships jq 1.7; re-pin --expect and update the jq contract #935

Closed
opened 2026-10-29 21:15:27 +01:00 by timothy · 3 comments
Owner

script-tests (.gitea/workflows/pr-checks.yml) runs scripts/jq-preflight.sh --expect 1.6. Since 2026-10-28 it fails on every runner that has picked up the job:

  • jazz-small-runner (PR #933 runs 3393 and its rerun, job 12347) reports jq-1.7.
  • small-runner on bumblebee (PR #934 run 3396, job 12360) also reports jq-1.7 (parsed 1.7; floor 1.6).

Every other job on those PRs is green. The cause is a newer docker.gitea.com/runner-images:ubuntu-latest image cached on the runners. bumblebee was rebooted on 2026-10-28 and its host jq was already 1.7.1. Runner images are not something this repo controls (server-management).

The tripwire did what it was built for: the 1.6 coverage the suite assumed is gone, and a person has to decide what replaces it. Because the merge hook reads COMBINED status, this advisory red blocks the auto-grant on every PR, so every PR is now stuck.

Decision (owner, 2026-10-28): re-pin to what the fleet ships. Since the fleet is mixed (server-management#1080), accept {1.6, 1.7} until #1080 settles one version; see the comment below.

Scope

  • Change --expect in pr-checks.yml to the measured fleet version.
  • Re-read docs/ci-cd.md, section "The jq contract", and the record ci.jq-version-contract. Wherever they say "the CI runner ships jq 1.6", state the invariant with a date instead. Decide, with evidence, whether the 1.6 FLOOR stays (a gate still authored to the 1.6 subset is harmless) or rises. Say what 1.6 coverage is lost now that no runner ships it.
  • scripts/jq-preflight.sh header comments name 1.6 as the runner version; correct them.
  • This changes an active decision, so add a new record under docs/decisions/records/ci/ and archive its predecessor (lifecycle). Regenerate the catalog.

Done-when

  • script-tests' jq preflight accepts exactly {1.6, 1.7} until server-management#1080 pins the fleet: green on whichever runner takes the job (cite the job log), with a witnessed red for a version outside the set
  • The jq contract docs, the record and the jq-preflight.sh comments state the new invariant; no claim that runners ship 1.6 remains
  • Adversarial review passed
`script-tests` (`.gitea/workflows/pr-checks.yml`) runs `scripts/jq-preflight.sh --expect 1.6`. Since 2026-10-28 it fails on every runner that has picked up the job: - jazz-small-runner (PR #933 runs 3393 and its rerun, job 12347) reports `jq-1.7`. - small-runner on bumblebee (PR #934 run 3396, job 12360) also reports `jq-1.7 (parsed 1.7; floor 1.6)`. Every other job on those PRs is green. The cause is a newer `docker.gitea.com/runner-images:ubuntu-latest` image cached on the runners. bumblebee was rebooted on 2026-10-28 and its host jq was already 1.7.1. Runner images are not something this repo controls (server-management). The tripwire did what it was built for: the 1.6 coverage the suite assumed is gone, and a person has to decide what replaces it. Because the merge hook reads COMBINED status, this advisory red blocks the auto-grant on every PR, so every PR is now stuck. Decision (owner, 2026-10-28): re-pin to what the fleet ships. Since the fleet is mixed (server-management#1080), accept {1.6, 1.7} until #1080 settles one version; see the comment below. ## Scope - Change `--expect` in `pr-checks.yml` to the measured fleet version. - Re-read docs/ci-cd.md, section "The jq contract", and the record `ci.jq-version-contract`. Wherever they say "the CI runner ships jq 1.6", state the invariant with a date instead. Decide, with evidence, whether the 1.6 FLOOR stays (a gate still authored to the 1.6 subset is harmless) or rises. Say what 1.6 coverage is lost now that no runner ships it. - `scripts/jq-preflight.sh` header comments name 1.6 as the runner version; correct them. - This changes an active decision, so add a new record under `docs/decisions/records/ci/` and archive its predecessor (lifecycle). Regenerate the catalog. ## Done-when - [x] `script-tests`' jq preflight accepts exactly {1.6, 1.7} until server-management#1080 pins the fleet: green on whichever runner takes the job (cite the job log), with a witnessed red for a version outside the set - [x] The jq contract docs, the record and the jq-preflight.sh comments state the new invariant; no claim that runners ship 1.6 remains - [x] Adversarial review passed
timothy added the priority: highin-progress labels 2026-10-29 21:16:01 +01:00
Author
Owner

Claiming #935: branch 935-jq-repin-17, worktree /Users/timothy/orca/workspaces/ersatztv/wt-935 (no open PR, no remote branch, no prior claim found).

Claiming #935: branch 935-jq-repin-17, worktree /Users/timothy/orca/workspaces/ersatztv/wt-935 (no open PR, no remote branch, no prior claim found).
Author
Owner

Scope change (2026-10-29, from server-management, refs server-management#1080): the runner fleet's jq version is not settled yet. Both small runners map to the floating runner-images:ubuntu-latest image with no digest pin, so each host keeps whatever image it last pulled. A digest pin across the fleet is proposed in #1080 but not decided.

So do not pin to 1.7 alone. Until #1080 lands, the tripwire must accept exactly the set {1.6, 1.7}. Any other version must still fail loudly, and that needs a witnessed red. The docs and the record must name #1080 as the event that narrows the set back to one version. The first Done-when box therefore reads: "passes on whichever runner takes the job, while still failing on a version outside {1.6, 1.7}".

Scope change (2026-10-29, from server-management, refs server-management#1080): the runner fleet's jq version is not settled yet. Both `small` runners map to the floating `runner-images:ubuntu-latest` image with no digest pin, so each host keeps whatever image it last pulled. A digest pin across the fleet is proposed in #1080 but not decided. So **do not pin to 1.7 alone**. Until #1080 lands, the tripwire must accept exactly the set {1.6, 1.7}. Any other version must still fail loudly, and that needs a witnessed red. The docs and the record must name #1080 as the event that narrows the set back to one version. The first Done-when box therefore reads: "passes on whichever runner takes the job, while still failing on a version outside {1.6, 1.7}".
Author
Owner

Closing record

Outcome: PR http://192.168.1.95:3000/timothy/ersatztv/pulls/938 (head 403bf011c): the script-tests jq preflight accepts exactly {1.6, 1.7} until server-management#1080 pins the runner fleet. Issue left open pending merge.
Root cause: the preflight pinned a single version (1.6) while the runner hosts shipped jq 1.7 (both hosts measured 1.7 on 2026-10-28); the tripwire fired on a healthy fleet and its advisory red blocked the merge-hook auto-grant on every PR.
Decisions/conventions changed: added ci.jq-accepted-version-set; archived ci.jq-version-contract; catalog regenerated.
Reusable knowledge: runner image jq version is not controlled by this repo; the contract is stated as a dated invariant naming server-management#1080. review-verdict.yml stays floor-only.
Verification: full scripts/tests 1615 passed, 3 skipped (orchestrator, 403bf011c); jq/pr-changed-files tests 353 passed; ruff, bash -n, check-doc-narrative (0 warnings), catalog regen (no diff) clean. Own-PR CI pending. Review round 1 clean (9 nits); codex could not run, cold same-family agent substituted.
Deferred: none (nits carry no issue).
Docs updated: docs/ci-cd.md, docs/decisions/ (new record, archived predecessor, catalog), docs/defect-shapes-773.md, record cross-refs.

Done-when evidence

  • Box 1 (accepts exactly {1.6, 1.7}, witnessed red outside set, green on runner): PYTHONPATH=. python3 -m pytest scripts/tests/test_jq_preflight.py (out-of-set red cases); the script-tests job in this PR's own CI run on PR #938.
  • Box 2 (docs/record/comments state invariant, no 1.6-runner claim): git diff origin/main...HEAD -- docs scripts/jq-preflight.sh; docs/decisions/records/ci/jq-accepted-version-set.md; python3 scripts/check-doc-narrative.py --diff origin/main.
  • Box 3 (adversarial review passed): review round 1 clean (0 blocking, 0 should-fix, 9 nit) in the PR body; scripts/post-review-verdict.sh 938 to be posted by the orchestrator.
## Closing record **Outcome:** PR http://192.168.1.95:3000/timothy/ersatztv/pulls/938 (head 403bf011c): the script-tests jq preflight accepts exactly {1.6, 1.7} until server-management#1080 pins the runner fleet. Issue left open pending merge. **Root cause:** the preflight pinned a single version (1.6) while the runner hosts shipped jq 1.7 (both hosts measured 1.7 on 2026-10-28); the tripwire fired on a healthy fleet and its advisory red blocked the merge-hook auto-grant on every PR. **Decisions/conventions changed:** added `ci.jq-accepted-version-set`; archived `ci.jq-version-contract`; catalog regenerated. **Reusable knowledge:** runner image jq version is not controlled by this repo; the contract is stated as a dated invariant naming server-management#1080. review-verdict.yml stays floor-only. **Verification:** full scripts/tests 1615 passed, 3 skipped (orchestrator, 403bf011c); jq/pr-changed-files tests 353 passed; ruff, bash -n, check-doc-narrative (0 warnings), catalog regen (no diff) clean. Own-PR CI pending. Review round 1 clean (9 nits); codex could not run, cold same-family agent substituted. **Deferred:** none (nits carry no issue). **Docs updated:** docs/ci-cd.md, docs/decisions/ (new record, archived predecessor, catalog), docs/defect-shapes-773.md, record cross-refs. ### Done-when evidence - Box 1 (accepts exactly {1.6, 1.7}, witnessed red outside set, green on runner): `PYTHONPATH=. python3 -m pytest scripts/tests/test_jq_preflight.py` (out-of-set red cases); the `script-tests` job in this PR's own CI run on PR #938. - Box 2 (docs/record/comments state invariant, no 1.6-runner claim): `git diff origin/main...HEAD -- docs scripts/jq-preflight.sh`; docs/decisions/records/ci/jq-accepted-version-set.md; `python3 scripts/check-doc-narrative.py --diff origin/main`. - Box 3 (adversarial review passed): review round 1 clean (0 blocking, 0 should-fix, 9 nit) in the PR body; `scripts/post-review-verdict.sh 938` to be posted by the orchestrator.
timothy removed the in-progress label 2026-10-30 00:03:22 +01:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#935