docs(773): root-cause the recurring defect shapes across the full closed-issue corpus #782

Merged

timothy merged 8 commits from issue/773-defect-shapes into main 2026-08-13 21:16:51 +02:00

Conversation 1 Commits 8 Files Changed 2

+537

timothy commented 2026-08-13 20:02:02 +02:00

Owner

Copy Link

Copy Source

fixes #773

Adds docs/defect-shapes-773.md and indexes it in docs/README.md (index entry + a task-signal row for "proposing a new guard / CI check").

Method

All 349 closed issues; 95 carry a ## Closing record. Each was classified by one of five independent raters against a written taxonomy that allowed NEW:<name>, so the taxonomy could not be self-fulfilling. Two controls:

• Inter-rater: a sixth rater re-classified a stratified 15-issue sample blind — 12/15 raw agreement across 11 categories. The two disagreements at the "was there a process failure" boundary point in opposite directions, so that boundary is noisy but unbiased.
• Backward generalization: a seventh rater sampled 25 of the 254 pre-convention issues from raw bodies/comments.

What changed relative to the issue's own analysis

• The ranking reverses. Vacuous verification is the most common shape (17/69), not twin-missed (14/69). The issue's 50-issue sample had it the other way.
• The central hypothesis holds and extends. Twin-missed, vacuous-by-sampling and symptom-keyed guards are one error — reasoning about a representative instead of the population. ~32/69 (46%), one detector. The doc gives 46% and 58% and says which assumption produces which.
• The shapes predate the closing-record convention (#1, #215, #232, #234, #403, #473), so they are not an artifact of recent guard-building. I had flagged that confound mid-session; the backward check refuted it.
• A class the taxonomy missed entirely: check-and-use races over mutable state (#536, #622, #632, #706, #707) — five instances, larger than four of the issue's own classes. Proposed independently by three raters and re-derived by a fourth, blind.
• Overclaim drops to 4% as a primary cause — it is a modifier riding on another failure, not a class with its own detector.

Part 2 — tooling audit (measured, not assumed)

• csharp-lsp cannot initialize (libhostfxr.dylib not found in a Homebrew dotnet path that does not exist); typescript-lsp cannot resolve typescript from the repo root; pyright-lsp works.
• The LSP tool has 0 calls across 811 transcripts — verified with a positive control (23,661 Bash, 5,102 Read). My first attempt at that count returned 0 for Bash too, from an rg -c output-format slip; the doc notes it, because a zero without a positive control is the exact failure this document is about.
• ruff/pyright are enforced nowhere in CI or hooks despite the global instruction, and there is no committed ruff config — so lint behaviour depends on one laptop's ~/.config.
• No dead hook scripts — all 13 are wired, contrary to the issue's suspicion.
• PreToolUse/PostToolUse hooks leave no durable execution trace, so we cannot currently tell whether merge-consent, worktree, BOM or agent-model guards fire at all.

Honest limits, stated in the doc rather than smoothed over

Record coverage is not uniform (31/32 for #600–699 vs 4/69 for #1–99); corpus composition shifts from feature work to CI hardening; a closing record is a self-report and cannot contain a defect nobody noticed. §4 also names the classes where no mechanical detector is plausible — overclaim, per-task review blindness, omitted brief constraints — rather than inventing weak ones.

Spawned

#774 (detector A, high) · #775 (detector B, high) · #776 (hook observability, high) · #777 (LSPs, medium) · #778 (check-and-use races, medium) · #779 (fail-open + destructive replace, medium) · #780 (Python tooling, medium) · #781 (config hygiene, low).

Detector C (dedup by construction) deliberately has no issue — it is the shape #774/#778's fixes should take, and filing it separately would invite the parity check the corpus already rejected twice.

Docs-only; no code paths touched.

fixes #773 Adds `docs/defect-shapes-773.md` and indexes it in `docs/README.md` (index entry + a task-signal row for "proposing a new guard / CI check"). ## Method All **349 closed issues**; **95 carry a `## Closing record`**. Each was classified by one of five independent raters against a written taxonomy that allowed `NEW:<name>`, so the taxonomy could not be self-fulfilling. Two controls: - **Inter-rater**: a sixth rater re-classified a stratified 15-issue sample **blind** — 12/15 raw agreement across 11 categories. The two disagreements at the "was there a process failure" boundary point in **opposite** directions, so that boundary is noisy but unbiased. - **Backward generalization**: a seventh rater sampled 25 of the 254 pre-convention issues from raw bodies/comments. ## What changed relative to the issue's own analysis - **The ranking reverses.** Vacuous verification is the most common shape (17/69), not twin-missed (14/69). The issue's 50-issue sample had it the other way. - **The central hypothesis holds and extends.** Twin-missed, vacuous-by-sampling and symptom-keyed guards are one error — reasoning about a representative instead of the population. ~32/69 (46%), one detector. The doc gives 46% and 58% and says which assumption produces which. - **The shapes predate the closing-record convention** (#1, #215, #232, #234, #403, #473), so they are *not* an artifact of recent guard-building. I had flagged that confound mid-session; the backward check refuted it. - **A class the taxonomy missed entirely**: check-and-use races over mutable state (#536, #622, #632, #706, #707) — five instances, larger than four of the issue's own classes. Proposed independently by three raters and re-derived by a fourth, blind. - **Overclaim drops to 4% as a primary cause** — it is a modifier riding on another failure, not a class with its own detector. ## Part 2 — tooling audit (measured, not assumed) - `csharp-lsp` **cannot initialize** (`libhostfxr.dylib` not found in a Homebrew dotnet path that does not exist); `typescript-lsp` cannot resolve `typescript` from the repo root; `pyright-lsp` works. - **The `LSP` tool has 0 calls across 811 transcripts** — verified with a positive control (23,661 `Bash`, 5,102 `Read`). My first attempt at that count returned 0 for `Bash` too, from an `rg -c` output-format slip; the doc notes it, because a zero without a positive control is the exact failure this document is about. - `ruff`/`pyright` are enforced **nowhere** in CI or hooks despite the global instruction, and there is no committed ruff config — so lint behaviour depends on one laptop's `~/.config`. - **No dead hook scripts** — all 13 are wired, contrary to the issue's suspicion. - **PreToolUse/PostToolUse hooks leave no durable execution trace**, so we cannot currently tell whether merge-consent, worktree, BOM or agent-model guards fire at all. ## Honest limits, stated in the doc rather than smoothed over Record coverage is not uniform (31/32 for #600–699 vs 4/69 for #1–99); corpus composition shifts from feature work to CI hardening; a closing record is a self-report and cannot contain a defect nobody noticed. §4 also names the classes where **no** mechanical detector is plausible — overclaim, per-task review blindness, omitted brief constraints — rather than inventing weak ones. ## Spawned #774 (detector A, high) · #775 (detector B, high) · #776 (hook observability, high) · #777 (LSPs, medium) · #778 (check-and-use races, medium) · #779 (fail-open + destructive replace, medium) · #780 (Python tooling, medium) · #781 (config hygiene, low). Detector C (dedup by construction) deliberately has no issue — it is the shape #774/#778's fixes should take, and filing it separately would invite the parity check the corpus already rejected twice. Docs-only; no code paths touched.

timothy added 1 commit 2026-08-13 20:02:02 +02:00

docs(773): root-cause the recurring defect shapes across the full closed-issue corpus ... c3ccfa95e0

Analysis over all 349 closed issues (95 carry a `## Closing record`), classified
by five independent raters against a written taxonomy, with an inter-rater control
(12/15 blind agreement) and a backward-generalization check over the 254
pre-convention issues.

Findings that change the picture the issue started from:

- The ranking reverses. Vacuous verification is the most common shape (17/69),
  not twin-missed (14/69), over the full corpus vs the issue's 50-issue sample.
- The issue's central hypothesis holds and extends: twin-missed, vacuous-by-
  sampling and symptom-keyed guards are one error (reasoning about a
  representative instead of the population) — ~32/69 (46%), one detector.
- The shapes predate the closing-record convention (#1, #215, #232, #403), so
  they are not an artifact of recent guard-building work.
- A class the taxonomy missed entirely: check-and-use races over mutable state
  (#536, #622, #632, #706, #707) — five instances, proposed independently by
  three raters and re-derived by a fourth, blind.
- Overclaim drops to 4% as a primary cause; it is a modifier, not a class.

Part 2 (tooling audit), all measured rather than assumed: csharp-lsp cannot
initialize and typescript-lsp cannot resolve typescript, the `LSP` tool has 0
calls across 811 transcripts, ruff/pyright are enforced nowhere despite the
global instruction, and PreToolUse/PostToolUse hook execution leaves no durable
trace — so we cannot currently tell whether our own guards fire.

Names the classes where no mechanical detector is plausible rather than
inventing weak ones. Spawns #774-#781; this issue is analysis, not implementation.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:15:59 +02:00

docs(773): fix the double-count, two fabricated quotes, and a miscited precedent ... af3126edbd

Both cold reviews found real defects. Fixes, in severity order:

FABRICATED QUOTES (worst). Two strings presented as verbatim do not appear in
the records they cite: "even a no-op" (#252) and "one bug papering over
another" (#232/#234). I lifted paraphrases from a subagent's summary table and
re-presented them inside quotation marks. Replaced with the actual wording,
verified against the raw comments. A third invented figure ("~40 duplicated
lines" for #649) is removed the same way — the record names the extracted
script and its decision record, not a line count.

MISCITED PRECEDENT. #711 was cited as a case where the corpus "rejected" a
parity check. Its record argues the opposite and deliberately KEEPS its
enumerative list, because deriving it would evaluate the rule against the very
file list being classified. Family C and the §6 rationale now carry it as the
reasoned exception it is.

DOUBLE-COUNT. Family A claimed all 14 twin-missed records while Family C
separately claimed 4 of them — the exact error §3 Family A is about, and
undisclosed, unlike the vacuous split. §3.6 now states the partition explicitly
and asserts it covers all 69 records exactly once, disjointly. Family A is 27
(39%), not ~32 (46%). The completeness assertion caught a transcription slip
(#438) while being written.

CAVEAT THAT DID NOT SURVIVE. The 39% is now bounded, where it is asserted, to
the recent guard-heavy corpus; the backward sample establishes the shapes
existed earlier, not that the proportions hold.

Also: scope detector A to populations with an external authoritative source and
send the code-structural residue (#403, #510) to #777; mark detector D as a fix
pattern whose detector is A over an enumerated inventory; soften Family D's
"independently proposed" to "unprompted by the taxonomy", and note #632 was
rated twin-missed with the race secondary; trim the over-broad
insufficient-evidence list to #157 alone; fix a dangling reference in 5.4.

MISSED DELIVERABLE. #773 asked for tools we do NOT have; the draft skipped it.
New 5.5 proposes four, each tied to specific issues: Stryker.NET (#621, #685,
#719), shellcheck (#445), a duplicate-code detector (#440, #649, #711), and an
enum-exhaustiveness rule for switch statements (#503).

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:26:49 +02:00

docs(773): fix round 3 — #510 miscited, a letter collision, and a tool I never tested ... 2404cefde1

The re-review of the fix commit found the fix had introduced adjacent defects,
which is the pattern this document records. Three fixed, plus one I found by
testing a claim the reviewer flagged as unverifiable.

#510 MISCITED, contradicting this document's own partition. The scope-limit
paragraph cited it as a code-structural population with "no external list",
two sentences before saying an enum IS that list. Verified: its switch keys on
`ChannelWatermarkImageSource`, a plain 3-value enum, and its record names the
root cause as "Duplication" — so it is Family C, where 3.6 already puts it.
Replaced with the real distinction: a population of VALUES always has an
external source; a population of SITES IN CODE does not. #403 is the honest
example of the latter (PlaybackOrder's values enumerate; its 6 dispatch sites
do not). Swept the same error out of 5.1 and out of #774/#777, which had
inherited it.

LETTER COLLISION. 3.6's family table reused F/G/H, which 4 had already assigned
to different detectors, after teaching the reader that Family X maps to
Detector X for A-E. Families below E are now unlettered, with the independence
stated.

SHELLCHECK: PROPOSED, TESTED, REJECTED. 5.5 claimed it "flags this class
directly" for #445. It does not. ShellCheck 0.11.0 reports nothing on that
exact construct, `-o all` yields only an unrelated brace nit, and against our
13 hook scripts it finds 2 SC2034s. Kept as a struck-through negative result so
it is not re-proposed on plausibility, since a tool justified by "it plausibly
catches this" is the same species of claim this document criticises.

Also: 3.6's percentage column floor-rounds to 99%, now stated.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:34:09 +02:00

docs(773): fix round 4 — a false rounding claim and an overclaim about my own evidence ... 7a8e0e09d3

Round 3 introduced two defects; both are the failure modes this document is
about, which is the point.

FALSE ROUNDING CLAIM. 3.6's total row said "percentages floor-rounded, so the
column reads 99%". Verified: true floors sum to 97, not 99 — the displayed
values are round-half-up (2/69 shows as 3%, 4/69 as 6%). The sentence asserted
a method that, applied to its own table, contradicts the total in the same
sentence. Now: the n column is exact, percentages are rounded to nearest and
sum to 99%.

OVERCLAIM ABOUT MY OWN EVIDENCE, inside the paragraph warning against exactly
that. 5.5 said "each row above was tested rather than reasoned about". Only the
shellcheck row was ever executed — and it was struck through precisely because
running it refuted its rationale. The three surviving rows (Stryker.NET, jscpd,
the enum-exhaustiveness analyzer rule) are reasoned from precedent and have not
been run against this repo. They are now labelled candidates to test, with the
weaker warrant stated, and #775 carries the Stryker candidate with the same
caveat rather than as a recommendation.

Also tightened 3.6's cross-table wording: this table assigns no letters below E,
so an F or G in 4 refers to nothing in it.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:40:51 +02:00

docs(773): round-churn is a property, not a class — and review scope is the 4th sampling target ... 0916b3b770

Prompted by a direct question: is "each fix round introduces the next defect"
a new shape, or one already recorded? Measured rather than argued.

ROUND-CHURN IS MIS-SCOPED IN THIS DOCUMENT. It inherited #773's "string-
predicate churn", which names a substrate. Counting records that narrate >=3
review rounds: 18 of 69, spread across EVERY family — A=5, C=3, E=3, D=2,
identity=2, and one each in B, overclaim and string-predicate. Only 1 of the 18
sits in the class named after the phenomenon. #697 settles it independently:
three BLOCKED rounds on credential scoping, no parser involved. So it is a
modifier riding on another failure, like overclaim — new 3.7 says so, and the
2 and 3.6 rows now point at it.

THE FOURTH SAMPLING TARGET. 3 named three things sampled instead of enumerated
— the fix, the verification, the guard. Review scope is the fourth: a re-review
briefed to check the reported finding samples, while the text written to fix the
last round is unreviewed by construction. This session is the demonstration —
rounds 2 and 3 were briefed on the delta and each shipped a new defect; round 4
was briefed to sweep the whole file and came back clean. 4 now lists it as the
one detector with no build step.

TWO LIMITS STATED. The count pattern-matches narrative prose, so 18 is a floor,
not a total. And #767 — the eight-round case that motivates this whole issue —
is absent from the corpus entirely, because it was still open and has no
closing record. The headline example is invisible to the measurement meant to
characterise it.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:42:33 +02:00

docs(773): #767 closed 19 minutes after the corpus snapshot — state the boundary, use its record ... cc2bf7128f

Caught by the user. 3.7 said #767 "was still open when the corpus was drawn, so
it has no `## Closing record`." First half true, second half stale within the
hour: the corpus was drawn 19:45, #767 closed 20:04, and it now carries a full
closing record. A claim that was true when written and silently stopped being
true — the exact class this document ranks 7th — inside the sentence
complaining that #767 was missing.

Fixed by stating the snapshot boundary rather than softening the claim: the
corpus is a 19:45 snapshot, #767 is excluded by timing, and it is named as an
exclusion instead of quietly folded in, because re-drawing would move every
denominator in 2 and 3.6 and the boundary has to sit somewhere. The header now
carries the snapshot time, not just the date.

Reading its record strengthens Family A rather than complicating it. #767
reinvents detector A independently — the fifth and sixth times in this repo:
"Refuting two variants of a channel is not clearing the channel" (the working
attacks through that channel were never tried), and "Enumerating SHAPES of a
command loses. Nine disarms across two rounds; running the command settles them
together." The latter is detector A in a sentence.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:50:55 +02:00

docs(773): the round-count table sampled instead of enumerating — 33, not 18 ... 9fc334e6e6

A cold reviewer BLOCKED 0916b3b77 on the section I had just added. The finding
is correct and lands where it hurts most.

I built the >=3-rounds table by pattern-matching a regex I invented against the
corpus, in the section arguing that guards must enumerate a population rather
than sample it. The regex found 18. Reading all 69 records finds 33. Fifteen
missed records narrate their rounds in the same words as the counted ones — #578
("Six cold review rounds on #676, nine on #678"), #651 ("Nine"), #706 ("six
rounds, five real defects"), #756 ("Five"), #698 ("Six"), #671, #616, #621,
#631, #685, #445, #643 among them.

Worst of it: #706 is named as a Family D member three paragraphs earlier in this
document, and my own D row omitted it. The table contradicted its own section 3.

The hedge was wrong too, in a way that mattered. It blamed "records that ran
rounds without narrating them" — but that is not what happened; the missing
records narrate plainly. It misdescribed an incomplete sweep as an inherent
limit, which is how a sampling error gets laundered into a stated caveat.

Corrected by reading all 69 rather than writing a better pattern, with the
extraction rule and two loose attributions (#647, #629) stated. New distribution:
A=10, B=5, C=5, E=5, D=3, identity=2, string-predicate=2, overclaim=1.

The conclusion survives and is stronger — 2 of 33 sit in the class named after
the phenomenon, against 1 of 18 before. Added the qualification that string
predicates ARE disproportionately round-prone (2 of that class's 3 records); the
naming error was defining a class by the substrate where a cross-cutting property
was noticed, leaving 31 instances nowhere to be counted.

Also swept a stale "the 18 above" in the #767 bullet.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy added 1 commit 2026-08-13 20:53:26 +02:00

docs(773): strip session narrative — the reader never saw the earlier drafts ... 1a2afd2f0f

Six places narrated this document's own revision history ("an earlier draft
counted...", "the first version of this table said 18", "the first attempt
returned 0 for Bash"). A reader coming to this cold never saw those drafts, so
the confessions carry no information they can act on — it is session narrative
leaking into a reference artifact.

Each was rewritten to keep what a reader needs and drop what only the author
knows:

- 3.6: was "an earlier draft double-counted" → now states why the families
  overlap conceptually and that membership is a mechanically verified partition,
  so an n is read as disjoint.
- 3.7: was "the first version said 18 and was wrong by half" → now two reusable
  limits: 33 is a floor (it counts records that STATE a count), and re-deriving
  it by pattern under-reports by roughly half, so read all 69.
- 5.3: was "my first attempt returned 0 for Bash too" → now the actionable form:
  re-derive the zero WITH the positive control, because `rg -c` over multiple
  files prints path:count and the obvious total sums the paths.
- 5.5 x2 and 5.5's evidence paragraph: dropped "an earlier draft skipped this"
  and "because an earlier draft overclaimed"; the evidence-status distinction
  itself is what matters and stays.

Deliberately KEPT, because these are findings rather than history: the struck-
through shellcheck row (a tool tested and rejected, so nobody re-proposes it),
the dated measurement stamps, and the 19:45 corpus-snapshot boundary.

The revision narrative is not lost — it lives in these commit messages and in
the closing record, which is where per-issue provenance belongs.

Refs #773

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

timothy commented 2026-08-13 21:01:13 +02:00

Author

Owner

Copy Link

Copy Source

Review-verdict: MERGEABLE @ 1a2afd2

Six cold rounds, worktree-isolated. Final pass whole-file per the doc's own 3.7. Independent re-derivation of the 33-round count and ~15 quotes verified verbatim; partition sums 69 exactly.

Review-verdict: MERGEABLE @ 1a2afd2 Six cold rounds, worktree-isolated. Final pass whole-file per the doc's own 3.7. Independent re-derivation of the 33-round count and ~15 quotes verified verbatim; partition sums 69 exactly.

timothy merged commit e1937674cd into main 2026-08-13 21:16:51 +02:00

timothy deleted branch issue/773-defect-shapes 2026-08-13 21:16:53 +02:00

timothy referenced this pull request 2026-08-13 21:17:36 +02:00

Root-cause the recurring defect shapes across closed issues, then build hooks/tooling that make them hard to repeat — plus an audit of the tooling we already have and don't use #773

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

ad-hoc

One-off / ad-hoc work not tracked by a dedicated issue

api

REST API / HTTP endpoints

bug

Something isn't working

ci-cd

Build, test, deploy pipeline

content

Channel content / schedules / playlists

dependencies

Dependency updates (Renovate)

enhancement

New feature or improvement

frontend

ChicoryTV React SPA frontend

in-progress

Claimed by an active session — do not pick up

jellyfin

Jellyfin tuner / IPTV integration

parked

Excluded from automatic queue pickup; work only when explicitly selected

priority: high

priority: low

priority: medium

review

Adversarial review finding

security

Security / vulnerability fix

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

renovate timothy (Timothy)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: timothy/ersatztv#782