timothy/ersatztv
1
0
Fork 0
Code Issues 54 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

"all N tests stayed green" claims in test comments go stale the moment anyone adds a test — 4 found, no detector #895

New Issue
Open
opened 2026-08-30 12:37:34 +02:00 by timothy · 0 comments
timothy commented 2026-08-30 12:37:34 +02:00
Owner
Copy Link
Copy Source

Found while closing #858/#859. A cold review round flagged one instance as a Medium; sweeping for the class found four, and the mechanism guarantees more.

The shape

A test's docstring records a mutation result by quoting a count:

"deleting the arm left all 57 tests green, because nothing downstream could tell them apart"

The observation is real and worth keeping — it is the evidence that a guard was a no-op. The count is the part that rots: it is falsified by the next person who adds a test to the file, which is the single most common edit those files receive. Nothing reddens when it happens, and the sentence still reads like a measurement.

This is release.state-the-invariant-not-the-measurement applied to prose that documents a mutation, and it is mechanically detectable in a way most narrative defects are not.

Population (a floor, from one grep — re-derive it)

$ git ls-files 'scripts/tests/*.py' | xargs grep -nE 'all [0-9]+ tests'

Measured 2026-08-30, four sites; three were fixed in #858/#859's PR because that PR's own new tests falsified them, one was left because it is in an unrelated file:

site claim truth at the time
test_merge_consent_required_check.py (×2) "all 57 tests", "all 29 tests" 64
test_check_required_contexts.py "all 71 tests across this file and the merge-consent file" 111
test_ci_dropped_step_guard.py:581 "all 30 tests" unverified — still open

The first three now read "the WHOLE suite" / "the ENTIRE suite" with a parenthetical saying why the count was dropped. The fourth is untouched: nothing in that PR falsified it, and editing an unrelated guard's file mid-PR is the scope creep this repo keeps rejecting.

Why it is worth a detector rather than another sweep

The #858/#859 PR fixed one instance, review found its identical twin in the same PR's own diff, and the class sweep then found two more. That is the signature of a defect whose instances are cheap to find and impossible to keep fixed by hand — docs/defect-shapes-773.md §4's criterion for promoting a class to a check.

A candidate detector is nearly trivial: fail on \ball \d+ tests\b (and likely \b\d+/\d+ (tests|green)\b) in scripts/tests/**. Before writing it, decide the carve-out, because there is a legitimate one: a count that names a specific historical commit ("at 37d86cda0 the suite was 57") is dated evidence and does not rot the same way. The rule wants "no count asserted about the CURRENT suite", not "no digits near the word tests".

Done-when

  • Population derived from git ls-files, not from the table above (which is one grep's floor)
  • A decision on the carve-out for dated/commit-pinned counts, recorded
  • Either a detector with a declared clause mutation and a docs/guard-inventory.md row, or a recorded decision that this stays a review-time judgement with the reason
  • test_ci_dropped_step_guard.py:581 resolved either way
  • Adversarial review passed

Refs #858, #859. Related: docs.no-session-narrative (#876 is the human-judgement sweep of the adjacent class in hooks and code comments).

Found while closing #858/#859. A cold review round flagged one instance as a Medium; sweeping for the *class* found four, and the mechanism guarantees more. ## The shape A test's docstring records a mutation result by quoting a **count**: > "deleting the arm left **all 57 tests** green, because nothing downstream could tell them apart" The observation is real and worth keeping — it is the evidence that a guard was a no-op. The count is the part that rots: **it is falsified by the next person who adds a test to the file**, which is the single most common edit those files receive. Nothing reddens when it happens, and the sentence still reads like a measurement. This is `release.state-the-invariant-not-the-measurement` applied to prose that documents a mutation, and it is mechanically detectable in a way most narrative defects are not. ## Population (a floor, from one grep — re-derive it) ``` $ git ls-files 'scripts/tests/*.py' | xargs grep -nE 'all [0-9]+ tests' ``` Measured 2026-08-30, four sites; three were fixed in #858/#859's PR because that PR's own new tests falsified them, one was left because it is in an unrelated file: | site | claim | truth at the time | |---|---|---| | `test_merge_consent_required_check.py` (×2) | "all 57 tests", "all 29 tests" | 64 | | `test_check_required_contexts.py` | "all 71 tests across this file and the merge-consent file" | 111 | | **`test_ci_dropped_step_guard.py:581`** | "all 30 tests" | **unverified — still open** | The first three now read "the WHOLE suite" / "the ENTIRE suite" with a parenthetical saying why the count was dropped. The fourth is untouched: nothing in that PR falsified it, and editing an unrelated guard's file mid-PR is the scope creep this repo keeps rejecting. ## Why it is worth a detector rather than another sweep The #858/#859 PR fixed one instance, review found its identical twin **in the same PR's own diff**, and the class sweep then found two more. That is the signature of a defect whose instances are cheap to find and impossible to keep fixed by hand — `docs/defect-shapes-773.md` §4's criterion for promoting a class to a check. A candidate detector is nearly trivial: fail on `\ball \d+ tests\b` (and likely `\b\d+/\d+ (tests|green)\b`) in `scripts/tests/**`. Before writing it, decide the **carve-out**, because there is a legitimate one: a count that names a *specific historical commit* ("at `37d86cda0` the suite was 57") is dated evidence and does not rot the same way. The rule wants "no count asserted about the CURRENT suite", not "no digits near the word tests". ## Done-when - [ ] Population derived from `git ls-files`, not from the table above (which is one grep's floor) - [ ] A decision on the carve-out for dated/commit-pinned counts, recorded - [ ] Either a detector with a declared clause mutation and a `docs/guard-inventory.md` row, or a recorded decision that this stays a review-time judgement with the reason - [ ] `test_ci_dropped_step_guard.py:581` resolved either way - [ ] Adversarial review passed Refs #858, #859. Related: `docs.no-session-narrative` (#876 is the human-judgement sweep of the adjacent class in hooks and code comments).
timothy added the ci-cdpriority: low labels 2026-08-30 12:37:40 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels ci-cd priority: low
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#895