docs(773): root-cause the recurring defect shapes across the full closed-issue corpus
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 12s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 16s
review-verdict/h10 Exempt: docs-only change (no code, no protected path)
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 10s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 8s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 8s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
PR Gates / Script tests (pytest) (pull_request) Successful in 2m14s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 12s
PR Gates / Docs update reminder (pull_request) Successful in 15s
PR Gates / decisions lifecycle (pull_request) Successful in 16s
review-verdict/h10 Exempt: docs-only change (no code, no protected path)
Review verdict / Set review-verdict status (pull_request_target) Successful in 11s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 10s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 8s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 8s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 7s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
PR Gates / Script tests (pytest) (pull_request) Successful in 2m14s
Analysis over all 349 closed issues (95 carry a `## Closing record`), classified by five independent raters against a written taxonomy, with an inter-rater control (12/15 blind agreement) and a backward-generalization check over the 254 pre-convention issues. Findings that change the picture the issue started from: - The ranking reverses. Vacuous verification is the most common shape (17/69), not twin-missed (14/69), over the full corpus vs the issue's 50-issue sample. - The issue's central hypothesis holds and extends: twin-missed, vacuous-by- sampling and symptom-keyed guards are one error (reasoning about a representative instead of the population) — ~32/69 (46%), one detector. - The shapes predate the closing-record convention (#1, #215, #232, #403), so they are not an artifact of recent guard-building work. - A class the taxonomy missed entirely: check-and-use races over mutable state (#536, #622, #632, #706, #707) — five instances, proposed independently by three raters and re-derived by a fourth, blind. - Overclaim drops to 4% as a primary cause; it is a modifier, not a class. Part 2 (tooling audit), all measured rather than assumed: csharp-lsp cannot initialize and typescript-lsp cannot resolve typescript, the `LSP` tool has 0 calls across 811 transcripts, ruff/pyright are enforced nowhere despite the global instruction, and PreToolUse/PostToolUse hook execution leaves no durable trace — so we cannot currently tell whether our own guards fire. Names the classes where no mechanical detector is plausible rather than inventing weak ones. Spawns #774-#781; this issue is analysis, not implementation. Refs #773 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,6 +24,7 @@ doc below, or that changes which sections a task signal points to.**
|
||||
| Concurrency / optimistic-locking work | `docs/api-conventions.md` §7a/b/c + `docs/decisions/optimistic-concurrency.md` |
|
||||
| Auth / security-surface work | `docs/decisions/api-auth-security.md` |
|
||||
| CI / release pipeline work | `docs/ci-cd.md` + `docs/decisions/release-ci-governance.md` |
|
||||
| Proposing a new guard / CI check / regression test convention | `docs/defect-shapes-773.md` §4 (detector menu + the classes where no detector is plausible) |
|
||||
| Live local run / Playwright-MCP verification | `docs/e2e-local.md` + `scripts/e2e-local.sh` |
|
||||
| Adding/changing a UI-E2E browser flow | `docs/e2e-local.md` → "UI-E2E harness" + `scripts/e2e-ui.sh` |
|
||||
| What does a test suite cover | `docs/testing.md` |
|
||||
@@ -98,6 +99,12 @@ bounds, what's mined per issue): `docs/handoffs/chicorytv-issue-queue.md` → "K
|
||||
that file's standing kickoff for the two concurrent tracks (orientation ‖ selection). ersatztv#237
|
||||
is a closed, archival historical tracker (superseded by `startup.parallel-orientation` in
|
||||
`docs/decisions.md`) — not a live pointer.
|
||||
- **`docs/defect-shapes-773.md`** — root-cause analysis of the recurring defect shapes across the
|
||||
whole closed-issue corpus (#773): the measured class ranking, the four families they consolidate
|
||||
into, the cheapest mechanical detector per class, the classes where **no** detector is plausible,
|
||||
and an audit of which configured hooks/MCP servers/LSPs are actually invoked. Read it before
|
||||
proposing a new guard or CI check — §4 is the detector menu, and it argues against enumerating
|
||||
cases one incident at a time.
|
||||
- **`docs/tracker-retrofit-triage-237.md`** — audit trail for the #524 triage of ersatztv#237's 111
|
||||
comments (method, per-comment classification, totals). Evidence for the
|
||||
`docs.tracker-comment-retrofit` decision; read it only when triaging another over-cap tracker.
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
# Recurring defect shapes across the closed-issue corpus (ersatztv#773)
|
||||
|
||||
Root-cause analysis over every closed issue in the tracker, the detectors that would have caught
|
||||
each class, and an audit of the tooling already configured in this repo. This is an **analysis
|
||||
artifact**, not a rulebook: the rules it argues for land as decision records and follow-up issues,
|
||||
which are linked per class below.
|
||||
|
||||
Measured 2026-08-13 against the tracker as of issue #773 and `origin/main` at `f9f8f65ce`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Method, and what these numbers can and cannot support
|
||||
|
||||
**Corpus.** All 349 closed issues (`state=closed&type=issues`, 7 pages, 349 distinct numbers,
|
||||
#1–#757 — pull requests excluded). Of those, **95 carry a `## Closing record`** comment. Each
|
||||
record was classified by one of five independent agents against a written taxonomy, with
|
||||
`NEW:<name>` available so the taxonomy could not be self-fulfilling. A sixth agent re-rated a
|
||||
stratified 15-issue sample blind, as an inter-rater control. A seventh sampled 25 of the 254
|
||||
pre-convention closed issues from their raw bodies and comments, to test whether the shapes exist
|
||||
outside the era that writes records.
|
||||
|
||||
**Three limits, stated up front because they bound every number below.**
|
||||
|
||||
1. **Record coverage is not uniform.** The `## Closing record` convention starts at #520
|
||||
(2026-07-21). Coverage is 31/32 for #600–699 and 4/69 for #1–99. So frequencies over the 95
|
||||
records are frequencies over *recent* practice.
|
||||
2. **Corpus composition shifts underneath the measurement.** The share of records describing no
|
||||
process failure at all falls from 12/19 (#60–#438) to 0/19 (#616–#671). Early issues are
|
||||
feature work; recent issues are largely CI/process hardening. Guard-shaped defects therefore
|
||||
dominate the recent records *partly by construction* — we have lately been building guards, so
|
||||
our defects are in guards.
|
||||
3. **A closing record is the closing author's self-report.** Raters were told to flag records whose
|
||||
own claims outrun what they describe, and several did (#60, #651, #691, #720). But the corpus
|
||||
systematically cannot contain a defect nobody noticed.
|
||||
|
||||
**What limit 2 does *not* undermine.** The obvious worry — that these classes are an artifact of
|
||||
recent guard-building — was tested and **refuted**. The backward sample found high-confidence
|
||||
instances of the same shapes well before the convention existed: twin-missed at #215 (REST playout
|
||||
mutations skip the build-lock gating the Blazor path had) and #403 (an unhandled enum silent at 5 of
|
||||
6 dispatch sites); wrapper drift at #198 and #287 (OpenAPI spec diverged from runtime JSON);
|
||||
masking guards at #232 and #234 ("one bug papering over another"); overclaim at #1 ("the earlier
|
||||
'confirmed' was based on one curl test and several unverified assertions"). The convention changed
|
||||
the **density and legibility** of the evidence, not the existence of the shapes. Several older
|
||||
issues (#94, #157, #172, #265, #377) genuinely cannot be classified from what they record, and are
|
||||
reported as `insufficient-evidence` rather than forced into a bucket.
|
||||
|
||||
---
|
||||
|
||||
## 2. What the corpus actually says
|
||||
|
||||
95 records; **69 describe a process failure**, 26 describe none. Percentages are of the 69.
|
||||
|
||||
| Rank | Class | n | % | Shape |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **Vacuous verification** | 17 | 25% | A check was green having done no work |
|
||||
| 2 | **Twin-missed** | 14 | 20% | The fix hit one instance and missed its structural twin |
|
||||
| 3 | **Symptom-keyed guard** | 9 | 13% | The guard keyed on the symptom seen, not the defect's mechanism |
|
||||
| 4 | *(new classes — see §3)* | 8 | 12% | Proposed by raters as fitting nothing above |
|
||||
| 5 | **Environment divergence** | 7 | 10% | Behaviour differs by interpreter/provider/hardware, untested there |
|
||||
| 6 | **Wrapper drift** | 5 | 7% | A hand-maintained mirror drifted from the contract it mirrors |
|
||||
| 7 | **Overclaim / stale claim** | 3 | 4% | An assertion stronger than its evidence, or since gone false |
|
||||
| 7 | **String-predicate churn** | 3 | 4% | A regex/grep predicate needing repeated rounds |
|
||||
| 9 | **Identity-not-capability** | 2 | 3% | Authorization scoped by *who*, not by *what is being changed* |
|
||||
| 10 | **Masking guards** | 1 | 1% | Two guards on one condition; one hid the other's total failure |
|
||||
|
||||
**This reorders the issue's own ranking.** #773 put twin-missed first (9 instances) and vacuous
|
||||
verification second (8), from a 50-issue sample. Over the full record corpus they swap: **vacuous
|
||||
verification is the most common single shape.** Overclaim also drops sharply as a *primary* cause
|
||||
(4%) — raters overwhelmingly assigned it as a *secondary*. It is better understood as a modifier
|
||||
that rides on another failure than as a class with its own detector.
|
||||
|
||||
**Inter-rater control (measured, not asserted).** A sixth rater re-classified a stratified 15-issue
|
||||
sample blind. **Raw agreement 12/15 (80%)** across an 11-category scheme. The three disagreements
|
||||
are informative rather than noise:
|
||||
|
||||
- **#177** (twin-missed → no-failure) and **#521** (no-failure → twin-missed) disagree at the
|
||||
"did a process failure occur" boundary and point in **opposite directions**, so that boundary is
|
||||
noisy but not systematically biased. The 26 clean-record count should be read as ±3.
|
||||
- **#649** was rated twin-missed by one and wrapper-drift by the other — which is precisely the blur
|
||||
that Family C in §3 exists to resolve. Both raters named the other class as secondary.
|
||||
|
||||
The top-three separation is far larger than the disagreement. Treat ±1 rank as noise.
|
||||
|
||||
---
|
||||
|
||||
## 3. Consolidation: four families, one of which the issue's taxonomy missed
|
||||
|
||||
### Family A — reasoning about a representative instead of the population
|
||||
|
||||
**#773's central hypothesis, confirmed, and it extends further than proposed.** The issue asked
|
||||
whether twin-missed and vacuous verification are the same underlying error. They are — and
|
||||
symptom-keyed guards belong with them. All three are the same mistake at different targets:
|
||||
|
||||
- **twin-missed** — the *fix* was applied to a sample of the population,
|
||||
- **vacuous-by-sampling** — the *verification* sampled an empty or unrepresentative subset,
|
||||
- **symptom-keyed guard** — the *guard* matched only the member that was noticed.
|
||||
|
||||
**Two numbers, because the honest one is smaller than the headline.** Merging the three classes as
|
||||
rated gives **40 of 69 records (58%)**. But vacuous verification splits: only part of it is a
|
||||
sampling error, and the rest belongs to Family B below. Assigning each of the 17 vacuous records to
|
||||
one half or the other — **my split, not the raters'** — puts ~9 in the sampling half and ~8 in the
|
||||
never-ran half. So Family A is **~32 of 69 (46%)**, and 58% is the upper bound that holds only if
|
||||
you accept the issue's original hypothesis wholesale. Either way it is the largest family by a wide
|
||||
margin.
|
||||
|
||||
The merge is not merely rhetorical; it predicts a single detector, and the corpus already contains
|
||||
that detector, independently reinvented several times:
|
||||
|
||||
> "the by-id handler covered 4 of 10 media types… **Sweep by FIELD, not by the call site the issue
|
||||
> names**" — #671
|
||||
> "#644's guard keyed on the *symptom* (an inflated pageSize) rather than the *defect*… an at-cap
|
||||
> request was structurally invisible to it" — #650
|
||||
> "filtered on tools that *already* declared `pageNum`, so a tool wrapping a paged endpoint with no
|
||||
> paging args escaped it entirely" — #616
|
||||
> "the pre-existing test filters `Where(t => t.QueryParameters is {Count: > 0})`, so a tool that
|
||||
> lost its query parameters escaped it entirely" — #757
|
||||
|
||||
Four sessions, four reinventions of one rule → **detector A** in §4.
|
||||
|
||||
### Family B — the check never ran at all
|
||||
|
||||
The other ~8 of the 17 vacuous-verification records are not sampling errors: the check was *dead*. `scripts/tests/`
|
||||
invoked by no CI job (#631); a validator whose call site could be deleted with the suite still green
|
||||
(#621); a `${{ }}` in a shell **comment** silently dropping a step while the job reported success in
|
||||
6s (#751); `if ! cmd; then status=$?` reading bash's logical negation, so a failing spec run exited 0
|
||||
(#445); new pre-push logic never wired to receive stdin, "dead code that every unit test still passes
|
||||
over" (#719). A sampling detector does nothing here. This needs **detector B**: proof the guard can
|
||||
go red.
|
||||
|
||||
### Family C — two copies of one thing
|
||||
|
||||
Wrapper drift plus a chunk of twin-missed: `.claude/` and its byte-identical `.codex/` mirror where
|
||||
only one was in the `PROTECTED` list (#711); an advisory local hook and the server-enforced workflow
|
||||
where four rounds of hardening landed on the copy with the lower stakes (#649); a hand-maintained MCP
|
||||
schema drifting from the generated OpenAPI DTO by one field, silently clearing it (#754, again #757);
|
||||
a hand-rolled frontmatter parser accepting YAML that PyYAML rejects (#674); two byte-identical Lucene
|
||||
escapers in different files (#440). The honest detector here is not a parity check — it is
|
||||
**deletion**: one implementation, called from both places.
|
||||
|
||||
### Family D — check-and-use race over mutable state *(the taxonomy missed this entirely)*
|
||||
|
||||
Three raters independently proposed overlapping new classes that consolidate into one — and the
|
||||
blind re-rater, having seen none of their output, proposed the same class again for #536 under
|
||||
its own name (`toctou-partial-atomicity` vs `half-atomic-toctou`). That is convergent validity, not
|
||||
a prompt echo:
|
||||
|
||||
| # | The race |
|
||||
|---|---|
|
||||
| 536 | Check-then-act split by an `await`; the write side was atomic, the read side a stale `Volatile.Read` |
|
||||
| 622 | Merge consent bound to a head sha, then Gitea's async auto-merge evaluated against a later head |
|
||||
| 632 | Head sha bound, but retargeting the PR base changes the diff while moving neither sha nor status |
|
||||
| 706 | Status writes are read-then-write with no compare-and-set; an older run can finish last and win |
|
||||
| 707 | Paged file enumeration diffs each page against the base's *live* tip, so a mid-paging advance drops rows |
|
||||
|
||||
**Five instances — larger than wrapper drift, identity-not-capability, string-predicate churn and
|
||||
masking guards individually**, and #773's taxonomy had no bucket for it. The unifying property:
|
||||
a check and the action it authorizes are separated in time over state that can change in between,
|
||||
with nothing pinning a version. #622's own record puts it exactly: *"The gate was never bypassed —
|
||||
it was satisfied against a snapshot that stops being true."*
|
||||
|
||||
### Two further cross-era classes, from the pre-convention sample
|
||||
|
||||
- **Fail-open by default** — a surface that defaults to permissive when config is absent or a step
|
||||
vanishes: API writes open when `Api:WriteKey` is unset (#280); a dropped step is fail-CLOSED in
|
||||
`review-verdict.yml` but fail-OPEN in `docker-build.yml` (#751, #756, #768); a jq-1.6-inert
|
||||
classifier (#647). Distinct from identity-not-capability: nothing is mis-scoped, the *default*
|
||||
is wrong.
|
||||
- **Destructive replace** — a full-replace path that silently destroys state a reconcile-by-id
|
||||
would have kept: a schedule PUT cascade-deleting fill-group progression on every save, "even a
|
||||
no-op" (#252); a full-replace wrapper missing one field and clearing it (#754); a dedup fix that
|
||||
turned a duplicate row into permanent data loss because the remove filter used a different key
|
||||
(#500).
|
||||
|
||||
---
|
||||
|
||||
## 4. Detectors
|
||||
|
||||
Ranked by instances covered per unit of build cost. "Rule status" distinguishes a **docs problem**
|
||||
(no rule exists) from a **hooks problem** (the rule exists and nothing enforces it) — they have
|
||||
different fixes, and conflating them is why several of these recurred.
|
||||
|
||||
| # | Detector | Covers | Rule status |
|
||||
|---|---|---|---|
|
||||
| **A** | **A guard derives its expected set from the authoritative source and asserts set equality — never `filter`, never a sample.** The population comes from the enum / the OpenAPI doc / the workflow YAML / the provider list, and the assertion is equality against it. A filter over the population cannot see the member that is missing. | Family A: ~32 records (46%); 40 (58%) if the whole vacuous class merges in | **rule-missing** as a general rule; instantiated ad hoc in #616, #644→#650, #671, #757 |
|
||||
| **B** | **Every guard ships with a proof it can go red: delete or disarm *that guard alone* and the suite must fail.** Not "a test exists" — a mutation. | Family B + masking guards: ~9 records | **rule-present-unenforced** — stated in #685's record and in project memory, enforced nowhere |
|
||||
| **C** | **Dedup by construction.** When two copies of one rule exist, delete one and have both callers invoke it. Prefer this over any parity check; a parity check is itself a Family-A guard that can be keyed wrong. | Family C: ~9 records | **rule-missing** |
|
||||
| **D** | **Anything read-then-written against live Gitea/remote state pins a version or uses compare-and-set** — bind the base sha as well as the head, page against a pinned tip, make the status write conditional. | Family D: 5 records | **rule-missing** (partially addressed for the merge path by the per-sha required check from #622) |
|
||||
| **E** | **Run the check under the interpreter/provider it will actually run under**, and preflight-log the version. | Environment divergence: 7 records | **rule-present** and working — `scripts/jq-preflight.sh` + `ci.jq-version-contract` closed the jq axis after #643/#647/#648. **Unclosed axes**: SQLite-vs-MySQL query semantics (#668), GPU generation (#505), CI-VM speed (#512) |
|
||||
| **F** | **Test the DENY path with the production config value.** #756's lesson generalised: a fixture that omits a field tests only the default, so a fail-open in the production value stays invisible. Parametrise the whole matrix, including "config absent". | Fail-open-by-default: ~5 records | **rule-missing** |
|
||||
| **G** | **Full-replace endpoints assert their complete field list in a test**, and prefer reconcile-by-id over delete-and-reinsert where child state exists. | Destructive replace: ~3 records | **rule-missing** |
|
||||
|
||||
**Detector A is the highest-value single change in this analysis** — one convention, ~46% of the
|
||||
recorded process failures, and it is already proven four times in this repo under four different
|
||||
names.
|
||||
|
||||
### Classes where no mechanical detector is plausible — stated rather than papered over
|
||||
|
||||
The issue explicitly asked for this, and inventing a weak detector here would itself be the
|
||||
symptom-keyed-guard mistake.
|
||||
|
||||
- **Overclaim / stale claim.** No check can tell that a sentence is stronger than the evidence
|
||||
behind it. Partial mitigations exist and should not be oversold: `stale-after` frontmatter (#603)
|
||||
dates a claim, and #578's retracted-term grep catches a *specific known* retraction propagating
|
||||
into generated artifacts. Neither detects a fresh overclaim. This stays a review responsibility.
|
||||
- **Per-task review blindness** (#60). A review scoped to one task's diff structurally cannot see a
|
||||
defect that only exists once several tasks compose — five Important findings at #60 were invisible
|
||||
to six per-task reviews and surfaced only in a whole-branch pass. The fix is a *process* step
|
||||
(a whole-branch review before close), not a check.
|
||||
- **Omitted brief constraint** (#586). A delegated brief silent on a hazard gets a plausible-but-wrong
|
||||
default — "an omitted rule isn't an unenforced rule, it's a rule replaced by whatever default the
|
||||
agent reaches for." A brief lint is conceivable but would be a keyword matcher, i.e. exactly the
|
||||
string-predicate class. A hazards checklist in the brief template is the honest ceiling.
|
||||
- **Guard-parity by verb** (#458), and re-deriving an inherited exemption against a new failure mode
|
||||
(#484). Both need judgment about whether a prior rationale still applies.
|
||||
|
||||
### A meta-finding about this repo's own knowledge base
|
||||
|
||||
`docs/decisions/records/` holds **189 active records**, and the two most relevant to this analysis
|
||||
(`testing.enumerating-guard-identity-not-position`, `ci.required-job-step-execution-markers`) are
|
||||
extraordinarily detailed — each a full account of one incident. The corpus grows **one record per
|
||||
instance**. That is Family A operating on our own process: we are enumerating cases rather than
|
||||
removing the mechanism. Detector A, C and B are class-level rules precisely because the per-instance
|
||||
record has already been tried 189 times.
|
||||
|
||||
---
|
||||
|
||||
## 5. Part 2 — audit of the tooling already configured
|
||||
|
||||
Judged against the measured classes above, per the issue's sequencing. Everything here was run, not
|
||||
assumed.
|
||||
|
||||
### 5.1 The LSPs: two of three are broken, and the one that matters most is the most broken
|
||||
|
||||
#773 asked whether routine LSP use would catch Family A (find *all* references rather than the one
|
||||
in front of you). It would help — **and it is not available.** Measured in this session:
|
||||
|
||||
| LSP | State | Evidence |
|
||||
|---|---|---|
|
||||
| `csharp-lsp` | **Broken — cannot initialize** | `findReferences` on `ChannelPlaylist.ToM3U()` → `System.InvalidOperationException: .NET SDK cannot be resolved, because libhostfxr.dylib cannot be found inside /opt/homebrew/Cellar/dotnet/10.0.302/bin/host/fxr`. That directory **does not exist** — Homebrew's dotnet layout is not what MSBuildLocator expects. `dotnet --version` works (10.0.302), so builds are fine; only the language server is dead. |
|
||||
| `typescript-lsp` | **Broken at the repo root** | `Could not find a valid TypeScript installation… ensure that the "typescript" dependency is installed in the workspace`. `typescript` lives in `web/node_modules`, not at the workspace root. Configuration problem, not a missing dependency. |
|
||||
| `pyright-lsp` | **Works** | `documentSymbol` on `scripts/decisions_lib.py` returned the full symbol tree; `findReferences` on `active_files` correctly returned 4 references across 3 files, including cross-file hits in `decisions_validate.py` and `migrate_decisions_split.py`. |
|
||||
|
||||
**Two conclusions, and the second is the sharper one.**
|
||||
|
||||
1. The LSP that would help most is the one that is dead. C# is where Family A bites hardest — #671
|
||||
(4 of 10 media types), #510 (three switch arms, not one), #403 (5 of 6 dispatch sites) are all
|
||||
"find every site" problems in C#, and `csharp-lsp` cannot answer a single query.
|
||||
2. **A dispatched subagent could not reach the LSP tool at all.** The agent tasked with testing the
|
||||
three LSPs reported `ToolSearch` returning "No matching deferred tools found" for every query,
|
||||
while the same tool resolved immediately in this main session. So the standing note
|
||||
*"workflow agents must use csharp-lsp"* is doubly rotten: the server is broken **and** the agents
|
||||
it addresses cannot invoke it even when it works. This is a live instance of overclaim/stale
|
||||
claim (§2 rank 7) sitting in our own guidance.
|
||||
|
||||
### 5.2 Python: the global instruction and this repo disagree, and the repo is silent
|
||||
|
||||
`~/.claude/CLAUDE.md` instructs `ruff check`, `ruff format --check`, and `pyright` after modifying
|
||||
Python. Measured against reality:
|
||||
|
||||
| Claim | Measured |
|
||||
|---|---|
|
||||
| ruff/pyright run in CI or hooks | **No.** `grep -rn "ruff\|pyright" .gitea/workflows/ .husky/` returns **nothing**. |
|
||||
| What *does* run on Python | `pr-checks.yml` → `decisions-guard` (`decisions_validate.py`, `build_decisions_catalog.py --check`) and `script-tests` (`pytest scripts/tests`). Neither Husky hook touches Python. |
|
||||
| `ruff check scripts/` | **47 errors** — but 28 are `E702` (semicolons) and **7 `S105` "hardcoded password" are false positives** on test stubs (`env["ETV_GITEA_TOKEN"] = "stub"`). The real cleanup is small. |
|
||||
| `ruff format --check scripts/` | **10 of 20 files would be reformatted.** |
|
||||
| `pyright scripts/` | **2 errors**, both `reportMissingImports` for `etv_client` in `scripts/scripted-schedules/entrypoint.py` — a package resolvable only in that script's deploy environment. Effectively clean. |
|
||||
| Repo-level ruff config | **None.** No `ruff.toml`/`pyproject.toml`. Ruff silently falls back to whichever `~/.config/ruff/ruff.toml` the operator's machine happens to have. |
|
||||
|
||||
**The last row is the finding that matters**, and it is Family E (environment divergence) in our own
|
||||
toolchain: Python lint behaviour here is a function of an un-versioned file on one laptop. A second
|
||||
machine lints differently, or not at all. One of the two must move — either the repo adopts a
|
||||
committed ruff config and enforces it, or the global instruction stops claiming this repo enforces
|
||||
something it does not.
|
||||
|
||||
### 5.3 Configured vs actually invoked
|
||||
|
||||
Measured over the session transcript corpus (811 files under
|
||||
`~/.claude/projects/-Users-timothy-ersatztv/`, this session excluded).
|
||||
|
||||
| Tool | Configured | Actually invoked | Verdict |
|
||||
|---|---|---|---|
|
||||
| `gitea` MCP | project `.mcp.json` **and** user scope (duplicate, identical values) | Heavy, same-day | Keep; de-duplicate the config |
|
||||
| `ssh-mcp` | project `.mcp.json` | 16 transcripts, same-day | Keep |
|
||||
| `mempalace` MCP | user scope | 10 transcripts, **last hit 2026-07-25** (19 days) | Keep, but see below |
|
||||
| **`LSP` (all three servers)** | 3 plugins enabled + `csharp-lsp` in `enabledMcpjsonServers` | **0 calls in 811 transcripts** | Broken *and* unused |
|
||||
| `context7` | enabled, `CONTEXT7_API_KEY` set | 0 calls | Dead |
|
||||
| `playwright` | enabled | 145 calls, last 2026-07-25 | Keep |
|
||||
| `superpowers` | enabled | 29 transcripts, last 2026-08-04 | Keep |
|
||||
| `feature-dev`, `ralph-loop`, `security-guidance` | enabled | 0 skill invocations | Dead as configured |
|
||||
| `codex` plugin | enabled | 0 *skill* invocations — but the documented workflow is `codex exec` via Bash, which this search cannot see | Not dead; measured the wrong surface |
|
||||
| `nuget`, `docker-mcp` | in `.mcp.json`, explicitly **disabled** | 0 (expected) | Correctly off |
|
||||
|
||||
The zero for `LSP` is verified rather than assumed: the same query shape returns 23,661 `Bash` calls
|
||||
and 5,102 `Read` calls over the same corpus, so the search was demonstrably capable of finding hits.
|
||||
*(The first attempt at this count returned 0 for `Bash` as well — a `rg -c` output-format slip that
|
||||
summed the file path instead of the count. A zero without a positive control is exactly the vacuous
|
||||
verification this document is about.)*
|
||||
|
||||
**Hooks are the good news.** All 13 scripts in `.claude/hooks/` are wired from either
|
||||
`.claude/settings.json` or `.husky/*`, and no settings entry points at a missing path — there are
|
||||
**no dead hook scripts**, contrary to the issue's suspicion. Husky hooks have genuine fired-output
|
||||
evidence in transcripts (`husky - dotnet format found whitespace/BOM issues`, `husky - commit message
|
||||
missing Co-Authored-By trailer`, `husky - refusing to commit root-level screenshot(s)`).
|
||||
|
||||
### 5.4 We cannot currently tell whether our own hooks fire
|
||||
|
||||
The most consequential Part 2 finding, and it was volunteered by the inventory as a limit on its own
|
||||
evidence. In this harness version only **`Stop`** hooks emit a structured record
|
||||
(`stop_hook_summary`/`hookInfos`). `PreToolUse` and `PostToolUse` hooks — which is *every* guard that
|
||||
matters here: merge consent, worktree, BOM, agent model/RAM — leave **no durable execution trace**.
|
||||
The counts for those rows are filename mentions in settings dumps and prior audits, not proof of
|
||||
execution. Only `design-sync-reminder.sh`'s Stop path and the three husky message strings are
|
||||
confirmed fires.
|
||||
|
||||
So the guards this repo relies on most are exactly the ones whose execution we cannot observe. That
|
||||
is Family B (the check never ran) applied to the hook layer itself, and this project has already
|
||||
paid for it twice at the CI layer — #751's guard that reported success in 6s having run nothing, and
|
||||
#756's fail-open twin. The standing lesson from #756 applies unchanged: **make the system report it
|
||||
rather than infer it.** A one-line append to a session-scoped log from each hook would convert every
|
||||
row in that table from inference to measurement, and it is cheap.
|
||||
|
||||
**On MemPalace**, the 19-day gap deserves care rather than a verdict: `CLAUDE.md` makes it the
|
||||
documented discovery path for decisions, so either sessions are skipping the documented path, or
|
||||
they are correctly reading `docs/decisions/README.md` directly (which the same contract permits, and
|
||||
which is authoritative). The measurement cannot distinguish those, and I am not going to guess.
|
||||
|
||||
---
|
||||
|
||||
## 6. Follow-ups
|
||||
|
||||
This issue is analysis; it spawns implementation rather than doing it.
|
||||
|
||||
| Issue | Detector / finding | Covers | Priority |
|
||||
|---|---|---|---|
|
||||
| #774 | **A** — a guard derives its population from the authoritative source and asserts set equality | Family A, ~32 records (46%) | high |
|
||||
| #775 | **B** — every guard ships a mutation proof: delete that guard alone, see red | Family B + masking guards, ~9 records | high |
|
||||
| #776 | Make hooks report that they fired — PreToolUse/PostToolUse execution is unobservable | the whole hook layer | high |
|
||||
| #777 | `csharp-lsp` and `typescript-lsp` are broken; the "workflow agents must use csharp-lsp" note is stale | partial mitigation for Family A | medium |
|
||||
| #778 | **D** — pin a version or use compare-and-set for read-then-write against live remote state | Family D, 5 records | medium |
|
||||
| #779 | **F + G** — test the deny path with the production config value; assert full-replace field lists | fail-open + destructive-replace, ~8 records | medium |
|
||||
| #780 | Reconcile Python tooling — commit a ruff config and enforce it, or stop claiming we do | environment divergence in our own toolchain | medium |
|
||||
| #781 | Retire enabled-but-never-invoked plugins/MCP servers; de-duplicate the `gitea` server | config that reads as coverage | low |
|
||||
|
||||
**Detector C (dedup by construction) has no issue of its own on purpose.** It is not a thing to build;
|
||||
it is the shape the fixes in #774 and #778 should take when they find two copies of one rule. Filing
|
||||
it separately would invite a parity check, which is the weaker construction the corpus already
|
||||
rejected twice (#649, #711).
|
||||
|
||||
Not filed, deliberately: overclaim/stale-claim, per-task review blindness, and omitted brief
|
||||
constraints. §4 argues no plausible mechanical detector exists for these, and filing an issue for
|
||||
each would produce exactly the weak enumerating guard this analysis recommends against.
|
||||
Reference in New Issue
Block a user