decisions_validate ceiling calibration test is one line from red — any record ≥61 prose lines fails script-tests #688

Closed
opened 2026-07-27 20:53:17 +02:00 by timothy · 4 comments
Owner

Found while working #668 (PR #687), where it turned a docs addition into a CI failure.

Corrected 2026-07-27 — the original write-up of this issue was wrong twice, both caught in review. It said the 60-line ceiling warning blocks, and that any record ≥60 lines reds the job. Neither is true; the accurate statement is below.

The problem

Two different things key off the number 60, and only one of them can red CI:

  • The per-record ceiling warning (::warning::, 17 records currently over it) is non-blocking by design — decisions_validate.py exits 0.
  • scripts/tests/test_decisions_validate.py::test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distribution asserts p90 <= RECORD_CEILING_DEFAULT <= p95. That one runs in the blocking script-tests job.

Measured on aa1f504e0:

value
records 176
RECORD_CEILING_DEFAULT 60
p90 59
p95 112

The ceiling clears p90 by one line. The distribution just above it is sparse — the next values are 62, 66, 71, 83 — so adding one record of ≥61 prose lines moves the p90 index into that gap, jumping p90 to 71–83 and reding the job. A record of exactly 60 makes p90 == ceiling == 60 and passes, which is the boundary #687's record was trimmed to.

In #687 a new 107-line record failed the test, and the only in-scope remedy was successive trims down to exactly 60 — the ratchet the test's own v4 docstring says it exists to abolish, pointed at record authors instead of at the constant.

Why it matters

  • script-tests is blocking, and per ci.script-tests-job its reds are real and must not be ignored.
  • The failure lands on whoever writes the next substantial record, unrelated to what they were working on.
  • The number 60 serves two roles that have drifted apart: "start of the tail" and "a line 17 records are already over."

What I did NOT do

I did not re-derive the ceiling in #687. That constant is owned by docs.corpus-size-signal, which records it as "chosen at a natural gap in the distribution," so changing it is a decision-record change and was out of scope for a bugfix PR.

Done-when

  • The ceiling is re-derived from the current distribution (or the test's property restated so routine corpus growth cannot red it), with the reasoning recorded
  • docs.corpus-size-signal updated if the constant or its rationale changes
  • Adding a new record of typical length does not red script-tests — demonstrated, not assumed
  • Adversarial review passed

Context: #687, #668, docs.corpus-size-signal.

Found while working #668 (PR #687), where it turned a docs addition into a CI failure. **Corrected 2026-07-27** — the original write-up of this issue was wrong twice, both caught in review. It said the 60-line *ceiling warning* blocks, and that any record ≥60 lines reds the job. Neither is true; the accurate statement is below. ## The problem Two different things key off the number 60, and only one of them can red CI: - The **per-record ceiling warning** (`::warning::`, 17 records currently over it) is **non-blocking by design** — `decisions_validate.py` exits 0. - `scripts/tests/test_decisions_validate.py::test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distribution` asserts `p90 <= RECORD_CEILING_DEFAULT <= p95`. **That** one runs in the blocking `script-tests` job. Measured on `aa1f504e0`: | | value | |---|---| | records | 176 | | `RECORD_CEILING_DEFAULT` | 60 | | p90 | **59** | | p95 | 112 | The ceiling clears p90 by **one line**. The distribution just above it is sparse — the next values are 62, 66, 71, 83 — so adding one record of **≥61** prose lines moves the p90 index into that gap, jumping p90 to 71–83 and reding the job. A record of exactly 60 makes `p90 == ceiling == 60` and passes, which is the boundary #687's record was trimmed to. In #687 a new 107-line record failed the test, and the only in-scope remedy was successive trims down to exactly 60 — the ratchet the test's own v4 docstring says it exists to abolish, pointed at record authors instead of at the constant. ## Why it matters - `script-tests` is blocking, and per `ci.script-tests-job` its reds are real and must not be ignored. - The failure lands on whoever writes the next substantial record, unrelated to what they were working on. - The number 60 serves two roles that have drifted apart: "start of the tail" and "a line 17 records are already over." ## What I did NOT do I did not re-derive the ceiling in #687. That constant is owned by `docs.corpus-size-signal`, which records it as "chosen at a natural gap in the distribution," so changing it is a decision-record change and was out of scope for a bugfix PR. ## Done-when - [x] The ceiling is re-derived from the current distribution (or the test's property restated so routine corpus growth cannot red it), with the reasoning recorded - [x] `docs.corpus-size-signal` updated if the constant or its rationale changes - [x] Adding a new record of typical length does not red `script-tests` — demonstrated, not assumed - [x] Adversarial review passed Context: #687, #668, `docs.corpus-size-signal`.
timothy added the ci-cdpriority: medium labels 2026-07-27 20:53:17 +02:00
timothy changed title from `decisions_validate` record ceiling has drifted out of calibration — any new record ≥60 prose lines reds `script-tests` to `decisions_validate` ceiling calibration test is one line from red — any record ≥61 prose lines fails `script-tests` 2026-07-27 22:22:19 +02:00
Author
Owner

Reproduced live while working #672, which is worth recording as evidence since this issue is a prediction about what would happen.

Adding one ordinary new decision record (ci.gate-trigger-base-resolved, 62 prose lines — not an outlier, just a normal record with a rationale section) turned the blocking script-tests job red:

AssertionError: the ceiling (60) no longer marks the tail boundary: p90=62, p95=112.
assert 62 <= 60

So the failure mode described here is not hypothetical and does not need a long record to trigger it — 62 lines did it, exactly as predicted ("any record ≥61 prose lines fails script-tests"). Note the ratchet direction: the only way to get green was to trim the new record to fit the constant, which is precisely the behaviour the test's own docstring says the design abolishes.

I trimmed rather than fixing this, to keep #672 in scope. No claim on this issue — it remains unclaimed and open.

Reproduced live while working #672, which is worth recording as evidence since this issue is a prediction about what *would* happen. Adding one ordinary new decision record (`ci.gate-trigger-base-resolved`, 62 prose lines — not an outlier, just a normal record with a rationale section) turned the blocking `script-tests` job red: ``` AssertionError: the ceiling (60) no longer marks the tail boundary: p90=62, p95=112. assert 62 <= 60 ``` So the failure mode described here is not hypothetical and does not need a long record to trigger it — 62 lines did it, exactly as predicted ("any record ≥61 prose lines fails `script-tests`"). Note the ratchet direction: the only way to get green was to **trim the new record to fit the constant**, which is precisely the behaviour the test's own docstring says the design abolishes. I trimmed rather than fixing this, to keep #672 in scope. No claim on this issue — it remains unclaimed and open.
Author
Owner

Evidence from ersatztv#706 (PR #723), which hit this in practice.

On main at 7cd71327b the distribution was p90 == 60 == RECORD_CEILING_DEFAULT — sitting exactly on the assertion boundary. So any new record longer than ~60 prose lines flips p90 above the ceiling and reddens test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distribution, in a blocking job (script-tests).

Measured while writing ci.verdict-write-retarget-fence:

record prose lines resulting p90 test
89 81 FAIL
63 63 FAIL
62 62 FAIL
61 61 FAIL
60 60 pass

The v4 docstring says percentiles "move WITH the corpus, so routine growth cannot ratchet this." That holds for growth in the bulk, but not at the boundary: because the ceiling is compared against p90 and p90 was already equal to it, one long record moves p90 by however far the next cluster sits (60 → 81 here). The remedy the failure message offers — "Re-derive it from the distribution" — is exactly the ceiling-raising ratchet the docstring says the change abolishes, so the test's own instruction and its stated purpose point in opposite directions.

The conflict worth resolving: docs.corpus-size-signal states that being listed by the ceiling is "an invitation to check for REDUNDANCY, never an instruction to cut: a long record that is all distinct findings is a legitimate decline, and should be recorded as one." But this test converts the advisory ceiling into a hard cap on any new record, since exceeding it fails a blocking job. A record cannot be "a legitimate decline" and also un-mergeable.

In #706 the record was cut 89 → 60 lines to land. Content was condensed rather than dropped, but the cutting was driven by the threshold, not by redundancy — which is the behaviour docs.corpus-size-signal explicitly warns against.

No fix proposed here; #706 deliberately kept its scope out of this issue. Filing the measurements so whoever takes this has the boundary case rather than having to rediscover it.

Evidence from ersatztv#706 (PR #723), which hit this in practice. **On `main` at 7cd71327b the distribution was `p90 == 60 == RECORD_CEILING_DEFAULT` — sitting exactly on the assertion boundary.** So *any* new record longer than ~60 prose lines flips `p90` above the ceiling and reddens `test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY_of_the_distribution`, in a **blocking** job (`script-tests`). Measured while writing `ci.verdict-write-retarget-fence`: | record prose lines | resulting p90 | test | |---|---|---| | 89 | 81 | FAIL | | 63 | 63 | FAIL | | 62 | 62 | FAIL | | 61 | 61 | FAIL | | **60** | **60** | **pass** | The v4 docstring says percentiles "move WITH the corpus, so routine growth cannot ratchet this." That holds for growth in the *bulk*, but not at the boundary: because the ceiling is compared against `p90` and `p90` was already equal to it, one long record moves `p90` by however far the next cluster sits (60 → 81 here). The remedy the failure message offers — "Re-derive it from the distribution" — is exactly the ceiling-raising ratchet the docstring says the change abolishes, so the test's own instruction and its stated purpose point in opposite directions. **The conflict worth resolving:** `docs.corpus-size-signal` states that being listed by the ceiling is "an invitation to check for REDUNDANCY, never an instruction to cut: a long record that is all distinct findings is a legitimate decline, and should be recorded as one." But this test converts the advisory ceiling into a **hard cap on any new record**, since exceeding it fails a blocking job. A record cannot be "a legitimate decline" and also un-mergeable. In #706 the record was cut 89 → 60 lines to land. Content was condensed rather than dropped, but the cutting was driven by the threshold, not by redundancy — which is the behaviour `docs.corpus-size-signal` explicitly warns against. No fix proposed here; #706 deliberately kept its scope out of this issue. Filing the measurements so whoever takes this has the boundary case rather than having to rediscover it.
timothy added the in-progress label 2026-08-04 20:00:11 +02:00
Author
Owner

Claiming this for a session working #674 + #688 together as a ci-cd bundle — both are defects in scripts/decisions_validate.py / its pytest suite, so they share a worktree and land as one reviewed change.

Pre-claim checks run and clear: no open PR references either number, no remote branch names them, no prior claiming comment (#688's 2026-07-28 comment explicitly states "No claim on this issue"), and origin/main re-fetched.

Claiming this for a session working #674 + #688 together as a `ci-cd` bundle — both are defects in `scripts/decisions_validate.py` / its pytest suite, so they share a worktree and land as one reviewed change. Pre-claim checks run and clear: no open PR references either number, no remote branch names them, no prior claiming comment (#688's 2026-07-28 comment explicitly states "No claim on this issue"), and `origin/main` re-fetched.
Author
Owner

Closing record

Outcome: Shipped in PR #725 (merged, efc34a3). The calibration claim is SPLIT by robustness instead of re-derived. Blocking script-tests now asserts only the coarse, non-ratcheting property — the ceiling flags a meaningful minority, 0.02 <= fraction_over <= 0.25. The fine tail-boundary claim (p90 <= ceiling <= p95) is measured every run and REPORTED by main() as a ::notice::, and asserted only against synthetic distributions the tests own. The ceiling stays 60.

Root cause: the property was correct as a definition but wrong as a blocking assertion. An order statistic over a sparse distribution is a step function: the lengths climb to the ceiling then jump straight to 81 with nothing between, so ONE new record could move p90 by 21 lines. More fundamentally the claim is about a corpus the commit under test does not control — the same shape as stale_records, which the same file already treats as a notice for exactly this reason.

Decisions/conventions changed: docs.corpus-size-signal updated with the v5 split, the fraction floor, and the three reddening arms.

Reusable knowledge: a guard test must depend only on the thing it guards. This defect was found FIVE times in one PR, each time somewhere new — the complement test; an if before.marks_tail: guard whose conclusion was still live (16 ordinary records reddened it); a hardcoded ceiling of 999; a test requiring >=1 over-ceiling record (which would red the day someone consolidates the last one); and a test that appended two records to the live corpus and so crossed the bound two records before production did. The cure is to derive fixtures from the data (p90 itself is always calibrated since p90 <= p90 <= p95; max+1 is always off the tail) or to own them outright — not to tune thresholds. Two related traps: a vacuity floor can itself be growth-coupled (len(recs) > 100 reds after 83 legitimate retirements), and a constant that IS a documented threshold must be mutation-pinned at both bounds AND both inclusivities or the docs quietly become wrong.

Verification: 432 scripts/tests pass; CI 10/10 green on efc34a3. Demonstrated, not assumed: a reviewer inserted a valid new record at 5/30/59/60/61/70/81/120/300 prose lines and the suite stayed green in all nine — including the 81- and 120-line records that reddened CI under v4. Thresholds measured: 38 consecutive over-ceiling additions, 718 short ones, or consolidating 15 of the 18 offenders. Mutation batteries across two reviewers killed every constant, both inclusivities of both properties, n, the keyless filter and the quantiles.

Deferred: the coarse bound is deliberately weaker than the fine one, and nothing now FORCES a re-derivation when the ceiling drifts — accepted and recorded in the decision record with the remedy named (a consolidation PR large enough to reach the floor should re-derive the ceiling in the same change). Three benign mutants survive and are documented: an unreachable not-a-mapping branch, a None -> "" normalisation no record exercises, and a no-op return shape.

Docs updated: docs/decisions/records/docs/corpus-size-signal.md, docs/ci-cd.md, docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Shipped in PR #725 (merged, `efc34a3`). The calibration claim is SPLIT by robustness instead of re-derived. Blocking `script-tests` now asserts only the coarse, non-ratcheting property — the ceiling flags a meaningful minority, `0.02 <= fraction_over <= 0.25`. The fine tail-boundary claim (`p90 <= ceiling <= p95`) is measured every run and REPORTED by `main()` as a `::notice::`, and asserted only against synthetic distributions the tests own. **The ceiling stays 60.** **Root cause:** the property was correct as a definition but wrong as a blocking assertion. An order statistic over a sparse distribution is a step function: the lengths climb to the ceiling then jump straight to 81 with nothing between, so ONE new record could move p90 by 21 lines. More fundamentally the claim is about a corpus the commit under test does not control — the same shape as `stale_records`, which the same file already treats as a notice for exactly this reason. **Decisions/conventions changed:** `docs.corpus-size-signal` updated with the v5 split, the fraction floor, and the three reddening arms. **Reusable knowledge:** a guard test must depend only on the thing it guards. This defect was found FIVE times in one PR, each time somewhere new — the complement test; an `if before.marks_tail:` guard whose *conclusion* was still live (16 ordinary records reddened it); a hardcoded ceiling of 999; a test requiring >=1 over-ceiling record (which would red the day someone consolidates the last one); and a test that appended two records to the live corpus and so crossed the bound two records before production did. The cure is to derive fixtures from the data (`p90` itself is always calibrated since `p90 <= p90 <= p95`; `max+1` is always off the tail) or to own them outright — not to tune thresholds. Two related traps: a vacuity floor can itself be growth-coupled (`len(recs) > 100` reds after 83 legitimate retirements), and a constant that IS a documented threshold must be mutation-pinned at both bounds AND both inclusivities or the docs quietly become wrong. **Verification:** 432 `scripts/tests` pass; CI 10/10 green on `efc34a3`. Demonstrated, not assumed: a reviewer inserted a valid new record at 5/30/59/60/61/70/81/120/300 prose lines and the suite stayed green in all nine — including the 81- and 120-line records that reddened CI under v4. Thresholds measured: 38 consecutive over-ceiling additions, 718 short ones, or consolidating 15 of the 18 offenders. Mutation batteries across two reviewers killed every constant, both inclusivities of both properties, `n`, the keyless filter and the quantiles. **Deferred:** the coarse bound is deliberately weaker than the fine one, and nothing now FORCES a re-derivation when the ceiling drifts — accepted and recorded in the decision record with the remedy named (a consolidation PR large enough to reach the floor should re-derive the ceiling in the same change). Three benign mutants survive and are documented: an unreachable not-a-mapping branch, a `None -> ""` normalisation no record exercises, and a no-op return shape. **Docs updated:** `docs/decisions/records/docs/corpus-size-signal.md`, `docs/ci-cd.md`, `docs/decisions/README.md` (regenerated).
timothy removed the in-progress label 2026-08-04 23:45:27 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#688