scripts/tests/ is never executed by CI — the Python test suite is local-only
#631
Closed
opened 2026-07-25 21:52:42 +02:00 by timothy
·
3 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#631
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The gap
scripts/tests/holds 95 pytest tests covering the decision-corpus parser, validator, catalog builder, the #610 migration-equivalence harness and (as of #622) the review-verdict poster. None of them run in CI.Verified by grep: no workflow and no Husky hook invokes
pytestor referencesscripts/tests.decisions-guardinpr-checks.ymlrunsdecisions_validate.pyandbuild_decisions_catalog.py --checkdirectly — it exercises the code but never the tests. Thetestjob isdotnet testonly.Why it matters
These are exactly the tests guarding the machinery that gates merges and the decision corpus — the components where "reports success while doing nothing" has now bitten repeatedly (#603
stale-after, #609 marker, #621 silent record invisibility). A regression in them is currently caught only if someone happens to run pytest locally.It also means a test added "for CI enforcement" is quietly decorative. I nearly shipped #622's suite believing it was enforced.
Shape of the fix
A step in the existing
decisions-guardjob (already hassetup-python, alreadyruns-on: small) is probably the cheapest home:Worth checking first whether pytest is present on the
smalllane or needs installing, and whether the suite is fast enough to sit in a blocking gate (~4s locally).Done-when
scripts/tests/runs in CI on every PR that touchesscripts/(or unconditionally, if cheap)docs/ci-cd.mddocuments where the suite runsNegative control — measured, not asserted
Done-when box 2 asks for verification "by measurement, not a green tick — deliberately break one test on a scratch branch and confirm CI goes red." Done on throwaway PR #638 (branch
probe/631-negative-control= PR #637's branch + one deliberately failing test, never merged):PR Gates / Script tests (pytest)PR Gates / decisions lifecyclePR Gates / CI image pinPR Gates / Docs update reminderIdentical branches apart from the broken test, so the job's red is attributable to the test and nothing else.
What running the suite in CI immediately caught
Two real reds before the green, both of which had been sitting undetected precisely because nothing ran these tests:
ModuleNotFoundError: No module named 'yaml'— the suite is not stdlib-only;test_migration_equivalence.pytransitively imports PyYAML viamigrate_decisions_split.py. It passed locally only because this machine had PyYAML. Fixed by installingpytest pyyaml, with the complete third-party set established by an AST import scan rather than by reading the files that seemed relevant.A fail-open in the merge-consent gate itself.
jq -eover empty input exits 4 on jq ≥ 1.7 but 0 on jq 1.6 — verified against both binaries, and the CI runner ships jq 1.6. The docs-only pagination guard leaned on that exit status to reject a transport failure, so on jq 1.6: page errors →gqreturns empty → guard wrongly passes →nempty so[ "$n" -lt 50 ]errors into false → loop walks past the failed page → next page legitimately returns[]→files_complete=yesover a partial list → the docs-only exemption fires over unread pages that may be pure code.That is the exact defect the guard's own comment describes, reintroduced one layer down by a jq version difference. Fixed by rejecting an empty body explicitly instead of inferring it from jq's exit status.
The second one is this issue's argument in miniature: it lived in the gate that decides whether a PR skips the Done-when checks, it was already covered by a test, and that test cannot catch it on a developer Mac (jq 1.8) — only in CI, where the suite had never run. A new test now pins the property on any jq by shimming only the 1.6 behaviour; mutation-verified.
Standing rule it leaves behind: never infer "empty input" from a jq exit status; check the string.
Deviation from the proposed fix
The issue suggested a step inside
decisions-guardas "probably the cheapest home". That is the one place it must not go —ci.decisions-lifecycle-flakeinstructs sessions that a lonedecisions lifecyclered is a known flake and must not be investigated, so a genuine pytest regression there would surface as exactly the red everyone is told to wave through. It is its ownscript-testsjob instead. Recorded asci.script-tests-job.PR: #637. Probe PR #638 is closed unmerged and its branch deleted.
The second of the two reds above — the merge-consent gate failing open on jq 1.6 — is now filed separately as #643 and carries its own
## Done-when. It stays fixed in PR #637; #643 exists so the gate's security-posture change is retrievable on its own rather than only as a paragraph inside this issue.Closing record
Outcome: Shipped in PR #637.
scripts/tests/now runs on every PR as its ownscript-testsjob inpr-checks.yml. New decision recordci.script-tests-job.Root cause: Nothing invoked
pytest.decisions-guardrunsdecisions_validate.pyandbuild_decisions_catalog.pydirectly — it exercises that code but never its tests — and thetestjob isdotnet testonly. So the suite guarding the decision corpus, the #610 migration harness, the merge-consent logic and the #622 verdict poster was enforced solely by whoever remembered to run it locally.Decisions/conventions changed: added
ci.script-tests-job. The load-bearing part is where the job does not go: the issue proposed a step insidedecisions-guard, which is the one job covered byci.decisions-lifecycle-flake("a lonedecisions lifecyclered is a known flake — do not investigate"). A real pytest regression there would surface as exactly the red every session is told to wave through. A gate inherits the credibility of the job it lives in, so a job under a standing ignore-rule can host no real gate.Reusable knowledge — what switching the light on immediately found. Four latent defects, none of them in this PR's own work:
test_migration_equivalence.py→ PyYAML). It passed locally only because this machine had it. The complete third-party set came from an AST import scan, not from reading the files that looked relevant.contains("�")is true for every string there), and the same fail-open sat inreview-verdict.yml, the server-enforced copy. 61 of 178 tests failed under jq 1.6 onmain.#was a YAML comment, so PyYAML and our hand reader disagreed — already red onmain.The pattern: every one was invisible because the only thing that could see it never ran. A test suite that runs nowhere is not a safety net, it is a document.
Verification: measured, not asserted. Throwaway PR #638 = this branch + one deliberately broken test →
Script tests (pytest)failure, every other job green; this branch → success. 216 tests green under both jq 1.8.2 and the runner's jq 1.6. Four independent cold review rounds (one cross-family Codex, three Opus cold-context, the last two in isolated worktrees); rounds 3 and 4 both MERGEABLE.Deferred: the job is not a required status check — it reddens the run but does not block the merge button; promoting it is a branch-protection change, deliberately out of scope. Pinning or preflighting the runner's jq version is recorded as a follow-up on #647 — three version divergences in one day argue for it, and patching constructs one at a time does not scale.
Docs updated:
docs/ci-cd.md(newscript-testssection, corrected "three gates" → four,small-lane paragraph, dependency list),docs/decisions/records/ci/script-tests-job.md(new),docs/decisions/README.md(regenerated).