decisions_validate ceiling calibration test is one line from red — any record ≥61 prose lines fails script-tests
#688
Closed
opened 2026-07-27 20:53:17 +02:00 by timothy
·
4 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#688
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found while working #668 (PR #687), where it turned a docs addition into a CI failure.
Corrected 2026-07-27 — the original write-up of this issue was wrong twice, both caught in review. It said the 60-line ceiling warning blocks, and that any record ≥60 lines reds the job. Neither is true; the accurate statement is below.
The problem
Two different things key off the number 60, and only one of them can red CI:
::warning::, 17 records currently over it) is non-blocking by design —decisions_validate.pyexits 0.scripts/tests/test_decisions_validate.py::test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distributionassertsp90 <= RECORD_CEILING_DEFAULT <= p95. That one runs in the blockingscript-testsjob.Measured on
aa1f504e0:RECORD_CEILING_DEFAULTThe ceiling clears p90 by one line. The distribution just above it is sparse — the next values are 62, 66, 71, 83 — so adding one record of ≥61 prose lines moves the p90 index into that gap, jumping p90 to 71–83 and reding the job. A record of exactly 60 makes
p90 == ceiling == 60and passes, which is the boundary #687's record was trimmed to.In #687 a new 107-line record failed the test, and the only in-scope remedy was successive trims down to exactly 60 — the ratchet the test's own v4 docstring says it exists to abolish, pointed at record authors instead of at the constant.
Why it matters
script-testsis blocking, and perci.script-tests-jobits reds are real and must not be ignored.What I did NOT do
I did not re-derive the ceiling in #687. That constant is owned by
docs.corpus-size-signal, which records it as "chosen at a natural gap in the distribution," so changing it is a decision-record change and was out of scope for a bugfix PR.Done-when
docs.corpus-size-signalupdated if the constant or its rationale changesscript-tests— demonstrated, not assumedContext: #687, #668,
docs.corpus-size-signal.`decisions_validate` record ceiling has drifted out of calibration — any new record ≥60 prose lines reds `script-tests`to `decisions_validate` ceiling calibration test is one line from red — any record ≥61 prose lines fails `script-tests`Reproduced live while working #672, which is worth recording as evidence since this issue is a prediction about what would happen.
Adding one ordinary new decision record (
ci.gate-trigger-base-resolved, 62 prose lines — not an outlier, just a normal record with a rationale section) turned the blockingscript-testsjob red:So the failure mode described here is not hypothetical and does not need a long record to trigger it — 62 lines did it, exactly as predicted ("any record ≥61 prose lines fails
script-tests"). Note the ratchet direction: the only way to get green was to trim the new record to fit the constant, which is precisely the behaviour the test's own docstring says the design abolishes.I trimmed rather than fixing this, to keep #672 in scope. No claim on this issue — it remains unclaimed and open.
Evidence from ersatztv#706 (PR #723), which hit this in practice.
On
mainat7cd71327bthe distribution wasp90 == 60 == RECORD_CEILING_DEFAULT— sitting exactly on the assertion boundary. So any new record longer than ~60 prose lines flipsp90above the ceiling and reddenstest_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distribution, in a blocking job (script-tests).Measured while writing
ci.verdict-write-retarget-fence:The v4 docstring says percentiles "move WITH the corpus, so routine growth cannot ratchet this." That holds for growth in the bulk, but not at the boundary: because the ceiling is compared against
p90andp90was already equal to it, one long record movesp90by however far the next cluster sits (60 → 81 here). The remedy the failure message offers — "Re-derive it from the distribution" — is exactly the ceiling-raising ratchet the docstring says the change abolishes, so the test's own instruction and its stated purpose point in opposite directions.The conflict worth resolving:
docs.corpus-size-signalstates that being listed by the ceiling is "an invitation to check for REDUNDANCY, never an instruction to cut: a long record that is all distinct findings is a legitimate decline, and should be recorded as one." But this test converts the advisory ceiling into a hard cap on any new record, since exceeding it fails a blocking job. A record cannot be "a legitimate decline" and also un-mergeable.In #706 the record was cut 89 → 60 lines to land. Content was condensed rather than dropped, but the cutting was driven by the threshold, not by redundancy — which is the behaviour
docs.corpus-size-signalexplicitly warns against.No fix proposed here; #706 deliberately kept its scope out of this issue. Filing the measurements so whoever takes this has the boundary case rather than having to rediscover it.
Claiming this for a session working #674 + #688 together as a
ci-cdbundle — both are defects inscripts/decisions_validate.py/ its pytest suite, so they share a worktree and land as one reviewed change.Pre-claim checks run and clear: no open PR references either number, no remote branch names them, no prior claiming comment (#688's 2026-07-28 comment explicitly states "No claim on this issue"), and
origin/mainre-fetched.Closing record
Outcome: Shipped in PR #725 (merged,
efc34a3). The calibration claim is SPLIT by robustness instead of re-derived. Blockingscript-testsnow asserts only the coarse, non-ratcheting property — the ceiling flags a meaningful minority,0.02 <= fraction_over <= 0.25. The fine tail-boundary claim (p90 <= ceiling <= p95) is measured every run and REPORTED bymain()as a::notice::, and asserted only against synthetic distributions the tests own. The ceiling stays 60.Root cause: the property was correct as a definition but wrong as a blocking assertion. An order statistic over a sparse distribution is a step function: the lengths climb to the ceiling then jump straight to 81 with nothing between, so ONE new record could move p90 by 21 lines. More fundamentally the claim is about a corpus the commit under test does not control — the same shape as
stale_records, which the same file already treats as a notice for exactly this reason.Decisions/conventions changed:
docs.corpus-size-signalupdated with the v5 split, the fraction floor, and the three reddening arms.Reusable knowledge: a guard test must depend only on the thing it guards. This defect was found FIVE times in one PR, each time somewhere new — the complement test; an
if before.marks_tail:guard whose conclusion was still live (16 ordinary records reddened it); a hardcoded ceiling of 999; a test requiring >=1 over-ceiling record (which would red the day someone consolidates the last one); and a test that appended two records to the live corpus and so crossed the bound two records before production did. The cure is to derive fixtures from the data (p90itself is always calibrated sincep90 <= p90 <= p95;max+1is always off the tail) or to own them outright — not to tune thresholds. Two related traps: a vacuity floor can itself be growth-coupled (len(recs) > 100reds after 83 legitimate retirements), and a constant that IS a documented threshold must be mutation-pinned at both bounds AND both inclusivities or the docs quietly become wrong.Verification: 432
scripts/testspass; CI 10/10 green onefc34a3. Demonstrated, not assumed: a reviewer inserted a valid new record at 5/30/59/60/61/70/81/120/300 prose lines and the suite stayed green in all nine — including the 81- and 120-line records that reddened CI under v4. Thresholds measured: 38 consecutive over-ceiling additions, 718 short ones, or consolidating 15 of the 18 offenders. Mutation batteries across two reviewers killed every constant, both inclusivities of both properties,n, the keyless filter and the quantiles.Deferred: the coarse bound is deliberately weaker than the fine one, and nothing now FORCES a re-derivation when the ceiling drifts — accepted and recorded in the decision record with the remedy named (a consolidation PR large enough to reach the floor should re-derive the ceiling in the same change). Three benign mutants survive and are documented: an unreachable not-a-mapping branch, a
None -> ""normalisation no record exercises, and a no-op return shape.Docs updated:
docs/decisions/records/docs/corpus-size-signal.md,docs/ci-cd.md,docs/decisions/README.md(regenerated).