DO NOT MERGE — probe(631): negative control, proves script-tests can go red #638
Closed
timothy
wants to merge 5 commits from
probe/631-negative-control into main
pull from: probe/631-negative-control
merge into: :main
:main
:renovate/dotnet-monorepo
:renovate/meziantou.analyzer-3.x
:renovate/lucene.net
:renovate/sqlitepclraw.bundle_e_sqlite3-3.x
:renovate/cliwrap-3.x
:913-scripted-playout-401
:release/v26.15.0-notes
:issue-806-guard-populations
:scratch/767b-poisoned
:scratch/767b-control
:release/v26.14.0-notes
:release/v26.14.0
:docs/510-skill-logo-bug-policy
:fix/510-watermark-resolution-policy
:fix/629-verdict-classifier-falseopens
:fix/609-decisions-edit-token-scope
:issue-135-clear-to-none
:release/v26.12.0-notes
:fix/409b-lastscan-api-parity
:fix/401-updatechannel-mirror-422
:fix/327-playlist-rename-validation
:fix/410-scancancel-log-level
:fix/409-447-librariesscreen-neverscanned
:fix/338-zap-exit-code
:fix/367-plex-budget-message
:fix/310-debom-legacy-cs
:ci/604-lane-rebalance
:feat/388-design-mirror
:feat/247-test-ownership
:feat/247-primary-action
:feat/357-player-owned-playback
:feat/357-jellyfin-plugin-poc
:fix/289-mcp-hardening
:issue58-mcp
:feat/244-channels-extract
:ci/auto-bump-prod-compose
:feat/multi-rerun-collections-api
:feat/collections-api
:feat/quick-wins
:feat/185-docs-part2
:feat/140-collections-screen
:feat/146-channel-edit
:feat/147-classic-ui-link
:issue22-renovate-dashboard
:feat/91-cutover
:feat/63-composite-create
:feat/65-library-browse
:feat/85-epg
:feat/86-schedule-editor
:feat/109-dashboard-data
:feat/99-session-tracking
:fix/dockerfile-node-tag
:feat/59-spa-foundation
:docs/59-ui-redesign-brief
:feat/102-json-guide
:feat/111-schedule-durations
:feat/104-artwork-upload
:feat/103-media-sources-api
:feat/playouts-read-api
:feat/108-health-api
:feat/105-picker-list-endpoints
:issue-97-channel-state-api
:issue42-jellyfin-musicvideos
:issue46-rest-api-error-contract
:dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
:qsv-improvements
:hdr-vulkan-cuda-test
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
52a8f16224 |
probe(631): deliberately broken test to measure that script-tests goes red in CI
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 9s
review-verdict/h10 Awaiting review verdict for 52a8f16
Review verdict / Set review-verdict status (pull_request) Successful in 3s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 15s
PR Gates / Script tests (pytest) (pull_request) Failing after 20s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m37s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 8s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 18m27s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 27m11s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Throwaway branch, never merged. Satisfies the #631 Done-when box requiring the enforcement be verified by MEASUREMENT rather than a green tick. |
||
|
|
6314646083 |
fix(631): close a fail-open in the merge-consent gate that only CI could see
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 15s
review-verdict/h10 Awaiting review verdict for 6314646
PR Gates / Docs update reminder (pull_request) Successful in 18s
Review verdict / Set review-verdict status (pull_request) Successful in 3s
PR Gates / decisions lifecycle (pull_request) Successful in 17s
PR Gates / Script tests (pytest) (pull_request) Successful in 22s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 20s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 16s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m35s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 23m56s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 26m57s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Running scripts/tests in CI for the first time turned up a real defect in the gate itself, not just in the harness. `jq -e` over EMPTY input exits 4 on jq >= 1.7 but 0 on jq 1.6 — verified against both binaries, not inferred. The docs-only pagination guard in pretooluse-merge-consent.sh leaned on that exit status to reject a transport failure. On jq 1.6, which the CI runner ships: page 2 errors -> gq returns empty -> jq guard wrongly PASSES -> n is empty so [ "$n" -lt 50 ] errors into false -> the loop walks PAST the failed page -> page 3 legitimately returns [] -> files_complete=yes over a PARTIAL list -> the docs-only exemption fires over unread pages that may be pure code. That is the very defect the guard's own comment describes, reintroduced one layer down by a jq version difference. Fixed by rejecting an empty body explicitly rather than inferring it from jq's exit status. The same hardening is applied to the review-verdict status read, which fell through to `vstate=""` -> deny (fail-CLOSED, so never a hole) but would have surfaced the wrong message. Why it survived: the existing transport-failure test asserts the right thing but can only observe the bug where jq is 1.6, so it passes on a developer Mac with the bug fully present — and the suite had never run anywhere else. The new test removes that dependency by shimming ONLY jq 1.6's empty-input exit status, so it pins the property on any host. Mutation-verified: revert the fix and it goes red on jq 1.8.2. The shim is deliberately narrow (no `-n`): a broader first version swallowed the `jq -n` calls `decide` uses to build its JSON, so the hook emitted nothing and every decision read as a passthrough — the verifier manufacturing the exemption it was meant to disprove. test_jq16_shim_actually_reproduces_the_quirk now pins the shim itself, `-n` case included. Suite: 113 passed under jq 1.8.2 AND under jq 1.6. Refs #631 |
||
|
|
b3d4759054 |
ci(631): install pyyaml too — the suite is not pure-stdlib, and CI proved it
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 22s
review-verdict/h10 Awaiting review verdict for b3d4759
PR Gates / Docs update reminder (pull_request) Successful in 26s
PR Gates / decisions lifecycle (pull_request) Successful in 26s
PR Gates / Script tests (pytest) (pull_request) Failing after 26s
Review verdict / Set review-verdict status (pull_request) Successful in 34s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m44s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 10s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m47s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 25m42s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
First CI run went red on `ModuleNotFoundError: No module named 'yaml'` at collection: test_migration_equivalence.py imports scripts/migrate_decisions_split.py, which uses PyYAML by design. It passed locally only because this machine already had PyYAML. The original claim came from reading decisions_lib/decisions_validate/build_catalog and the test files themselves — which is precisely the set that does NOT import yaml. Replaced with an AST import scan over all of scripts/, which gives the complete third-party set: pytest and yaml. (etv_client is imported only by scripts/scripted-schedules/entrypoint.py, a container entrypoint pytest never collects.) This does not contradict the dependency-free decisions READ path: decisions_lib._read_frontmatter is hand-written exactly so validation runs where nothing is installed. The one-shot WRITE path is allowed PyYAML, and its test needs it. Verified the way it should have been the first time: a clean venv with pytest alone reproduces CI's collection error; adding pyyaml gives 111 passed. Comments, ci-cd.md and the decision record all corrected to drop the "pure stdlib" claim. Refs #631 |
||
|
|
8d3fbe215b |
ci(631): preflight jq/git in script-tests, and reflow the header comment
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 10s
PR Gates / Docs update reminder (pull_request) Successful in 12s
review-verdict/h10 Awaiting review verdict for 8d3fbe2
PR Gates / Script tests (pytest) (pull_request) Failing after 15s
Review verdict / Set review-verdict status (pull_request) Successful in 4s
PR Gates / decisions lifecycle (pull_request) Successful in 23s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Failing after 4m13s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m42s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 13s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m23s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
test_post_review_verdict.py and test_merge_consent_exemption.py exec the real post-review-verdict.sh and pretooluse-merge-consent.sh, which shell out to jq ~26 times. Those tests shim `curl` on PATH but NOT `jq`, and no existing small-lane job proves jq is present on that image — so a missing jq would land as ~20 opaque assertion failures rather than a diagnosis. Checks rather than installs: #390 deliberately removed run-time apt-get from CI, so the correct fix for a genuine miss is baking the tool into the runner image, and the error message says so. Refs #631 |
||
|
|
fedec84997 |
ci(631): run scripts/tests in CI as its own job, not inside the flake-covered decisions-guard
Nothing executed scripts/tests/. No workflow and no Husky hook invoked pytest: decisions-guard runs decisions_validate.py and build_decisions_catalog.py directly (the code, never its tests) and the `test` job is dotnet test only. The 111 tests guarding the decision corpus, the #610 migration-equivalence harness, the merge-consent exemption logic and the #622 review-verdict poster were enforced only by whoever remembered to run pytest locally. Adds a `script-tests` job to pr-checks.yml rather than the step inside decisions-guard the issue proposed. decisions-guard is covered by ci.decisions-lifecycle-flake, a standing instruction that a lone `decisions lifecycle` red is a known flake and must not be investigated — so a genuine pytest regression there would surface as exactly the red every session is told to wave through. A gate inherits the credibility of the job it lives in. Runs unconditionally: the suite executes the real post-review-verdict.sh and pretooluse-merge-consent.sh, so a `scripts/**` path filter would miss a .claude/hooks/** edit. ~10s, pure stdlib + pytest. Negative control verified locally (broken test -> exit 1); CI-red measurement follows on a scratch PR per the Done-when. Refs #631 |