timothy/ersatztv
1
0
Fork 0
Code Issues 54 Pull Requests 6 Actions Packages Projects Releases 9 Wiki Activity

Mutation claims in PROSE are outside testing.mutation-claims-are-executed — 4 false ones in #812, each caught by a separate review round #881

New Issue
Open
opened 2026-08-29 21:21:06 +02:00 by timothy · 0 comments
timothy commented 2026-08-29 21:21:06 +02:00
Owner
Copy Link
Copy Source

Spawned by #812, where the documentation half was clean from review round 5 onward (two independent rounds traced every deleted line and found no loss) but four consecutive rounds each found one defect in the same narrow place: prose asserting what a proof establishes. Every fix round introduced the next one.

The four, and what each drifted from

# The claim The referent it drifted from
1 a docstring: the real-corpus arm reddens "including the depth caps" measurement — a count("/") <= 3 cap alters no real path's verdict; only the synthetic arm catches it. A comment 32 lines below in the same function said so.
2 a record: the script comment's "written once and never revised" another file's text. The phrase existed in an earlier draft of that comment, which the same branch had rewritten three commits earlier. git grep found it in exactly one place: the record line asserting it.
3 a docstring: the mutant is keyed on the "first segment" path.split("/", 2)[1] — the second. The quoted fragment also omitted the path.count("/") > 1 guard that makes the claim true; without it the expression fails 25 of 38 tests.
4 a record: revisions "all predate the last edit to that path" repo state. The commit asserting it modifies five files under that path, so the last edit to it is a revision — that one. False from the moment it shipped.

Not four unrelated slips. One shape: a prose assertion about a verifiable referent, with nothing binding the prose to the referent.

The mechanism already exists — its population is too narrow

testing.mutation-claims-are-executed is exactly the right rule:

A MUTATION row … is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in scripts/tests/mutation_manifest.py, and test_mutation_harness.py applies that mutation … on every run and requires the row's OWN named test to go red.

But mutation_manifest.py's own docstring scopes it: "one per MUTATION-graded row of docs/guard-inventory.md". So a mutation claim written in a test docstring, a decision record, or a commit message is outside it by construction — and that is where all four of #812's defects lived.

This is the #773 shape: a rule established where it was first noticed and never extended to its class. Which is the same shape #812 itself was remediating, and the same shape as #784 and #743. Worth stating plainly because the rule that would have prevented these was already written, already good, and simply not reachable from where the claims were being made.

Why the existing advice did not prevent it

"Verify claims about other files before asserting" is already recorded (#609), and it did not fire — because advice fires when you remember it, and these were all written while answering a review finding, which is the moment of least slack. docs.no-session-narrative names that mechanism itself:

A review finding creates pressure to answer in the artifact. The correction and the justification-for-the-correction get written in the same keystroke.

The rule predicted the defects in the commentary about applying the rule. So the fix has to be mechanical, not exhortative.

Proposed — to be argued, not assumed

  1. Extend the executed-claim rule to its class. A "mutant X reddens / does not redden arm Y" claim is either a manifest entry that executes, or it is not written — wherever it appears. In #812 the sentence "the <=3 cap is caught only by the synthetic arm" should have been a parametrized case; prose would then point at the manifest and could not go stale.
  2. A quotation of another file is a claim about that file. Check it or paraphrase without quote marks. The trigger that actually fires is concrete: after editing a file, grep for quotations of it.
  3. Never anchor to a state your own commit changes. "the last edit to that path", "currently", "the only X" — anchor to a DATE or a SHA. This is the existing relative-time ban extended: an anchor the commit moves is relative time in disguise.
  4. Prose written to answer a review finding is new code and gets the same scrutiny — cf. process.consistency-fix-new-code-scrutiny, which already says this for code fixes.

The carve-out, stated so this does not get over-applied

Over-application is the likelier failure, exactly as in docs.no-session-narrative. Not every sentence near a test becomes a manifest entry. The trigger is narrow: a claim that a specific mutation produces a specific outcome, which is checkable and therefore must be checked. "This arm exists because the corpus cannot see rules it has no instance of" is rationale and stays prose — it is the why behind a non-obvious choice, which the carve-out protects.

Note also the manifest's own doctrine, which any implementation must not violate: "no count is kept here, because a count of the entries below is a second copy of them." Do not add a count of prose claims either.

Done-when

  • The population of executed-claim enforcement stated and DERIVED from source, not from the table above (which is #812's four, not the corpus)
  • A decision on whether testing.mutation-claims-are-executed is extended in place or a sibling record is added, with the choice argued — extending in place is the #773-shaped fix, adding a sibling is what created the gap
  • Proposals 2 and 3 each either adopted with a mechanism or rejected with a reason, not left as advice
  • Adversarial review passed
Spawned by #812, where the *documentation* half was clean from review round 5 onward (two independent rounds traced every deleted line and found no loss) but **four consecutive rounds each found one defect in the same narrow place: prose asserting what a proof establishes.** Every fix round introduced the next one. ## The four, and what each drifted from | # | The claim | The referent it drifted from | |---|---|---| | 1 | a docstring: the real-corpus arm reddens "including the **depth caps**" | measurement — a `count("/") <= 3` cap alters no real path's verdict; only the synthetic arm catches it. A comment **32 lines below in the same function** said so. | | 2 | a record: ``the script comment's "written once and never revised"`` | another file's text. The phrase existed in an earlier draft of that comment, which the same branch had rewritten three commits earlier. `git grep` found it in **exactly one place: the record line asserting it**. | | 3 | a docstring: the mutant is keyed on the "**first** segment" | `path.split("/", 2)[1]` — the second. The quoted fragment also omitted the `path.count("/") > 1` guard that makes the claim true; without it the expression fails 25 of 38 tests. | | 4 | a record: revisions "all predate **the last edit to that path**" | repo state. The commit asserting it modifies five files under that path, so the last edit to it *is* a revision — that one. False from the moment it shipped. | Not four unrelated slips. One shape: **a prose assertion about a verifiable referent, with nothing binding the prose to the referent.** ## The mechanism already exists — its population is too narrow `testing.mutation-claims-are-executed` is exactly the right rule: > A `MUTATION` row … is not a statement that someone once witnessed a red. It carries a DECLARED clause mutation in `scripts/tests/mutation_manifest.py`, and `test_mutation_harness.py` applies that mutation … on every run and requires the row's OWN named test to go red. But `mutation_manifest.py`'s own docstring scopes it: *"one per `MUTATION`-graded row of `docs/guard-inventory.md`"*. So a mutation claim written in a **test docstring**, a **decision record**, or a **commit message** is outside it by construction — and that is where all four of #812's defects lived. **This is the #773 shape**: a rule established where it was first noticed and never extended to its class. Which is the same shape #812 itself was remediating, and the same shape as #784 and #743. Worth stating plainly because the rule that would have prevented these was already written, already good, and simply not reachable from where the claims were being made. ## Why the existing advice did not prevent it "Verify claims about other files before asserting" is already recorded (#609), and it did not fire — because advice fires when you remember it, and these were all written *while answering a review finding*, which is the moment of least slack. `docs.no-session-narrative` names that mechanism itself: > A review finding creates pressure to answer **in the artifact**. The correction and the justification-for-the-correction get written in the same keystroke. The rule predicted the defects in the commentary about applying the rule. So the fix has to be mechanical, not exhortative. ## Proposed — to be argued, not assumed 1. **Extend the executed-claim rule to its class.** A "mutant X reddens / does not redden arm Y" claim is either a manifest entry that executes, or it is not written — wherever it appears. In #812 the sentence *"the `<=3` cap is caught only by the synthetic arm"* should have been a parametrized case; prose would then point at the manifest and could not go stale. 2. **A quotation of another file is a claim about that file.** Check it or paraphrase without quote marks. The trigger that actually fires is concrete: *after editing a file, grep for quotations of it.* 3. **Never anchor to a state your own commit changes.** "the last edit to that path", "currently", "the only X" — anchor to a DATE or a SHA. This is the existing relative-time ban extended: an anchor the commit moves is relative time in disguise. 4. **Prose written to answer a review finding is new code** and gets the same scrutiny — cf. `process.consistency-fix-new-code-scrutiny`, which already says this for code fixes. ## The carve-out, stated so this does not get over-applied Over-application is the likelier failure, exactly as in `docs.no-session-narrative`. Not every sentence near a test becomes a manifest entry. The trigger is narrow: **a claim that a specific mutation produces a specific outcome**, which is checkable and therefore must be checked. "This arm exists because the corpus cannot see rules it has no instance of" is *rationale* and stays prose — it is the why behind a non-obvious choice, which the carve-out protects. Note also the manifest's own doctrine, which any implementation must not violate: *"no count is kept here, because a count of the entries below is a second copy of them."* Do not add a count of prose claims either. ## Done-when - [ ] The population of executed-claim enforcement stated and DERIVED from source, not from the table above (which is #812's four, not the corpus) - [ ] A decision on whether `testing.mutation-claims-are-executed` is extended in place or a sibling record is added, with the choice argued — extending in place is the #773-shaped fix, adding a sibling is what created the gap - [ ] Proposals 2 and 3 each either adopted with a mechanism or rejected with a reason, not left as advice - [ ] Adversarial review passed
timothy added the priority: medium label 2026-08-29 21:21:29 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
ad-hoc
One-off / ad-hoc work not tracked by a dedicated issue
 api
REST API / HTTP endpoints
 bug
Something isn't working
 ci-cd
Build, test, deploy pipeline
 content
Channel content / schedules / playlists
 dependencies
Dependency updates (Renovate)
 enhancement
New feature or improvement
 frontend
ChicoryTV React SPA frontend
 in-progress
Claimed by an active session — do not pick up
 jellyfin
Jellyfin tuner / IPTV integration
 parked
Excluded from automatic queue pickup; work only when explicitly selected
 priority: high
priority: low
priority: medium
review
Adversarial review finding
 security
Security / vulnerability fix
No labels priority: medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate timothy (Timothy)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#881