Generalize the no-session-narrative rule from the kickoff file to all docs — the reader never saw the earlier draft #784

Closed
opened 2026-08-13 20:54:00 +02:00 by timothy · 4 comments
Owner

Spawned by #773. Found while writing docs/defect-shapes-773.md, which had accumulated six places narrating its own revision history ("an earlier draft counted...", "the first version of this table said 18", "my first attempt returned 0 for Bash too"). All six were stripped in that PR; this issue is about the boundary, not those sites.

The rule already exists, scoped to one file

docs/handoffs/chicorytv-issue-queue.md says it, correctly:

Keep it that way — this file is pasted into every session's kickoff, so a paragraph of narrative here is a tax paid by every future session, while a record is retrieved only when it is needed (#542).

That reason is file-specific (this file is pasted every session). The general reason is different and broader: a reader coming to any doc cold never saw the earlier draft, so "we previously got this wrong" carries no information they can act on.

This is the recurring shape #773 measures — a rule established where it was first noticed and never extended to its class. Same as #743 (a control that defended the merge path while the push path stayed open) and as #767's own record puts it: "'the ban is enforced' and 'the ban is enforced where it matters' were never separated."

Why it leaks (mechanism, not sites)

A review finding creates pressure to answer in the artifact. The correction and the justification-for-the-correction are written in the same keystroke, and the artifact is the file already open — so the justification lands there too. Every one of the six instances came out of a review round.

Proposed rule

A review finding is answered in the commit message; only the corrected claim enters the doc. The commit is written at the same moment, is permanent, and is where provenance is actually looked for. Per-issue narrative belongs in the ## Closing record.

Stating the destination is load-bearing: "don't write it in the doc" without a home loses the knowledge, and this repo has the inverse failure on record too (pruning a narrative that turned out to be the only copy — #542).

The carve-out, which must be explicit or the rule gets over-applied

History that is reader-facing and must survive:

  • a decision record's supersedes / superseded-by
  • a dated measurement or a stated snapshot boundary (docs/defect-shapes-773.md carries both)
  • a tested-and-rejected negative result — e.g. that PR's struck-through shellcheck row, which exists so nobody re-proposes it on plausibility

The test is who benefits. If only the author's timeline explains why the sentence is there, it is narrative. If a reader would act differently knowing it, it is a finding.

On mechanising it — advisory only, and #773 §3.7 says why

A grep over added lines in docs/** for earlier draft|first version|previously said|used to say|we were wrong is a string predicate. #773 §3.7 and ci.* precedent say those need 3+ rounds and fail in unanticipated ways, so as a blocking gate it is a bad bet. As a non-blocking nudge it is nearly free, and the repo already has that exact pattern in the docs-reminder job.

So: extend docs-reminder, do not add a gate.

Done-when

  • Decision record written generalizing the rule beyond the kickoff file, with the who-benefits test and the carve-out stated (suggested key docs.no-session-narrative)
  • The kickoff file's file-specific version updated to reference the general rule rather than restating it
  • docs-reminder extended with the advisory check, explicitly non-blocking, with its false-positive behaviour on the carved-out cases checked against docs/defect-shapes-773.md (which deliberately contains legitimate history and must not be flagged into deletion)
  • A sweep of existing docs/*.md for the same leak, reporting what it found rather than assuming this document was the only offender
  • Adversarial review passed
Spawned by #773. Found while writing `docs/defect-shapes-773.md`, which had accumulated **six** places narrating its own revision history ("an earlier draft counted...", "the first version of this table said 18", "my first attempt returned 0 for `Bash` too"). All six were stripped in that PR; this issue is about the boundary, not those sites. ## The rule already exists, scoped to one file `docs/handoffs/chicorytv-issue-queue.md` says it, correctly: > **Keep it that way** — this file is pasted into every session's kickoff, so a paragraph of narrative here is a tax paid by every future session, while a record is retrieved only when it is needed (#542). That reason is **file-specific** (this file is pasted every session). The general reason is different and broader: **a reader coming to any doc cold never saw the earlier draft**, so "we previously got this wrong" carries no information they can act on. This is the recurring shape #773 measures — a rule established where it was first noticed and never extended to its class. Same as #743 (a control that defended the merge path while the push path stayed open) and as #767's own record puts it: *"'the ban is enforced' and 'the ban is enforced **where it matters**' were never separated."* ## Why it leaks (mechanism, not sites) A review finding creates pressure to answer **in the artifact**. The correction and the justification-for-the-correction are written in the same keystroke, and the artifact is the file already open — so the justification lands there too. Every one of the six instances came out of a review round. ## Proposed rule **A review finding is answered in the commit message; only the corrected claim enters the doc.** The commit is written at the same moment, is permanent, and is where provenance is actually looked for. Per-issue narrative belongs in the `## Closing record`. Stating the destination is load-bearing: "don't write it in the doc" without a home loses the knowledge, and this repo has the inverse failure on record too (pruning a narrative that turned out to be the only copy — #542). ## The carve-out, which must be explicit or the rule gets over-applied History that is **reader-facing** and must survive: - a decision record's `supersedes` / `superseded-by` - a dated measurement or a stated snapshot boundary (`docs/defect-shapes-773.md` carries both) - a **tested-and-rejected** negative result — e.g. that PR's struck-through `shellcheck` row, which exists so nobody re-proposes it on plausibility **The test is who benefits.** If only the author's timeline explains why the sentence is there, it is narrative. If a reader would act differently knowing it, it is a finding. ## On mechanising it — advisory only, and #773 §3.7 says why A grep over *added* lines in `docs/**` for `earlier draft|first version|previously said|used to say|we were wrong` is a **string predicate**. `#773` §3.7 and `ci.*` precedent say those need 3+ rounds and fail in unanticipated ways, so as a **blocking gate** it is a bad bet. As a **non-blocking nudge** it is nearly free, and the repo already has that exact pattern in the `docs-reminder` job. So: extend `docs-reminder`, do not add a gate. ## Done-when - [x] Decision record written generalizing the rule beyond the kickoff file, with the who-benefits test and the carve-out stated (suggested key `docs.no-session-narrative`) - [x] The kickoff file's file-specific version updated to reference the general rule rather than restating it - [x] `docs-reminder` extended with the advisory check, explicitly non-blocking, with its false-positive behaviour on the carved-out cases checked against `docs/defect-shapes-773.md` (which deliberately contains legitimate history and must not be flagged into deletion) - [x] A sweep of existing `docs/*.md` for the same leak, reporting what it found rather than assuming this document was the only offender - [x] Adversarial review passed
timothy added this to the Defect-shape hardening milestone 2026-08-13 20:54:00 +02:00
timothy added the priority: medium label 2026-08-13 20:54:00 +02:00
Author
Owner

Scope note (from a #776 session, 2026-08-14): if this ever becomes a hook, it belongs at USER scope, not in this repo. The rule — a reader coming to any doc cold never saw the earlier draft — is not ersatztv-specific; it is true of every repo, and an ersatztv-only hook would enforce it exactly where it was first noticed and nowhere else, which is the very shape this issue is about.

Mechanically that means ~/.claude/hooks/ + the user settings.json, not .claude/hooks/ here. Worth stating because ~/.claude/hooks/ currently holds only cosmetic hooks (terminal title, statusline) — there is no precedent there for a content rule, so someone would have to decide that deliberately.

Two caveats that should survive into whatever gets built:

  1. A narrative detector is a string predicate over prose, and this repo's own docs/defect-shapes-773.md §4 argues against weak string-matching detectors — testing.guard-derives-population-from-source calls a weak detector the symptom-keyed mistake, and the withdrawn test_review_verdict_vocabulary_parity.py (six rounds, then deleted) is the empirical case. So this is plausible as an advisory PostToolUse reminder on doc writes, and not plausible as a blocking guard. Budget for it being wrong sometimes, and make that acceptable by not letting it block.
  2. Decision records are a deliberate exception. docs/decisions/records/** narrate how a rule was got wrong on purpose — that IS the rationale a record exists to carry (see testing.guard-ships-with-mutation-proof, which recounts failing its own rule three times). A detector that does not exempt them would fight the convention it is meant to serve.
Scope note (from a #776 session, 2026-08-14): **if this ever becomes a hook, it belongs at USER scope, not in this repo.** The rule — a reader coming to any doc cold never saw the earlier draft — is not ersatztv-specific; it is true of every repo, and an ersatztv-only hook would enforce it exactly where it was first noticed and nowhere else, which is the very shape this issue is about. Mechanically that means `~/.claude/hooks/` + the user `settings.json`, not `.claude/hooks/` here. Worth stating because `~/.claude/hooks/` currently holds only cosmetic hooks (terminal title, statusline) — there is no precedent there for a content rule, so someone would have to decide that deliberately. Two caveats that should survive into whatever gets built: 1. **A narrative detector is a string predicate over prose**, and this repo's own `docs/defect-shapes-773.md` §4 argues against weak string-matching detectors — `testing.guard-derives-population-from-source` calls a weak detector the symptom-keyed mistake, and the withdrawn `test_review_verdict_vocabulary_parity.py` (six rounds, then deleted) is the empirical case. So this is plausible as an **advisory** PostToolUse reminder on doc writes, and not plausible as a blocking guard. Budget for it being wrong sometimes, and make that acceptable by not letting it block. 2. **Decision records are a deliberate exception.** `docs/decisions/records/**` narrate how a rule was got wrong on purpose — that IS the rationale a record exists to carry (see `testing.guard-ships-with-mutation-proof`, which recounts failing its own rule three times). A detector that does not exempt them would fight the convention it is meant to serve.
Author
Owner

Claiming this (Claude Code session, 2026-08-21). Checked first: no open PR references #784, no remote branch names it, and the only prior comment is the '#776 scope note', not a claim.

Plan: decision record docs.no-session-narrative (who-benefits test + carve-out), point the kickoff file's file-specific wording at it, extend the non-blocking docs-reminder job with the advisory check (verified against docs/defect-shapes-773.md and exempting docs/decisions/records/**), and sweep docs/*.md reporting what it finds.

Claiming this (Claude Code session, 2026-08-21). Checked first: no open PR references #784, no remote branch names it, and the only prior comment is the '#776 scope note', not a claim. Plan: decision record `docs.no-session-narrative` (who-benefits test + carve-out), point the kickoff file's file-specific wording at it, extend the non-blocking `docs-reminder` job with the advisory check (verified against `docs/defect-shapes-773.md` and exempting `docs/decisions/records/**`), and sweep `docs/*.md` reporting what it finds.
timothy added the in-progress label 2026-08-21 22:33:12 +02:00
Author
Owner

Sweep report (Done-when box 4) — reporting, not remediating

Two independent passes over the tracked docs corpus at 706674272, excluding docs/decisions/** (exempt by the rule):

  • a reading pass applying the who-benefits test — 18 sites
  • scripts/check-doc-narrative.sh --all, the advisory string predicate this PR adds — 10 sites

Union 21 sites across 6 files. Each pass missed sites the other caught: the detector's phrase list has no entry for "the first draft called that", "this row previously said", or "this paragraph used to describe", all of which the reading pass found; the reading pass stopped at its 25-file cap and missed three the detector caught. That divergence is the measured argument for keeping the check advisory rather than a gate — it is the same shape as the withdrawn test_review_verdict_vocabulary_parity.py, arrived at by measurement instead of by six review rounds.

Every cited line was re-read in the file rather than taken from a summary. Full table and per-file breakdown: #812, which is the remediation issue.

This document was not the only offender — docs/ci-cd.md (6), docs/remote-state-inventory.md (7) and docs/guard-inventory.md (4) each carry more than docs/defect-shapes-773.md does now (1).

Remediation is deliberately NOT in this PR. A material fraction of the 21 pass the who-benefits test and must stay — the remote-state-inventory cluster is largely "this row was graded down by cold review and the earlier wording was circular", which exists to stop the next reader regrading it back. That is 21 individual judgements, and #542 is this repo's record of pruning a narrative that turned out to be the only copy. Batching them behind the rule that authorized them is the failure mode the rule's own carve-out warns about.

False-positive check against docs/defect-shapes-773.md

The issue asked specifically whether the check would flag that file's legitimate history into deletion. It does not:

  • the dated measurements and the stated snapshot boundary: not flagged
  • the struck-through shellcheck row (a tested negative result): not flagged
  • one hit at line 583, "…and an earlier draft of this paragraph quietly contradicted it" — a genuine hit, correctly identified

And in CI the check scans only lines added in the PR diff, so the existing corpus of legitimate history is structurally out of its reach. docs/decisions/** is exempt wholesale in both modes.

## Sweep report (Done-when box 4) — reporting, not remediating Two independent passes over the tracked docs corpus at `706674272`, excluding `docs/decisions/**` (exempt by the rule): - a **reading** pass applying the who-benefits test — **18 sites** - `scripts/check-doc-narrative.sh --all`, the advisory string predicate this PR adds — **10 sites** Union **21 sites across 6 files**. Each pass missed sites the other caught: the detector's phrase list has no entry for "the first draft called that", "this row previously said", or "this paragraph used to describe", all of which the reading pass found; the reading pass stopped at its 25-file cap and missed three the detector caught. That divergence is the measured argument for keeping the check advisory rather than a gate — it is the same shape as the withdrawn `test_review_verdict_vocabulary_parity.py`, arrived at by measurement instead of by six review rounds. Every cited line was re-read in the file rather than taken from a summary. Full table and per-file breakdown: **#812**, which is the remediation issue. **This document was not the only offender** — `docs/ci-cd.md` (6), `docs/remote-state-inventory.md` (7) and `docs/guard-inventory.md` (4) each carry more than `docs/defect-shapes-773.md` does now (1). **Remediation is deliberately NOT in this PR.** A material fraction of the 21 pass the who-benefits test and must stay — the `remote-state-inventory` cluster is largely "this row was graded down by cold review and the earlier wording was circular", which exists to stop the next reader regrading it back. That is 21 individual judgements, and #542 is this repo's record of pruning a narrative that turned out to be the only copy. Batching them behind the rule that authorized them is the failure mode the rule's own carve-out warns about. ## False-positive check against `docs/defect-shapes-773.md` The issue asked specifically whether the check would flag that file's legitimate history into deletion. It does not: - the dated measurements and the stated snapshot boundary: **not flagged** - the struck-through `shellcheck` row (a *tested* negative result): **not flagged** - one hit at line 583, `"…and an earlier draft of this paragraph quietly contradicted it"` — a genuine hit, correctly identified And in CI the check scans only lines **added** in the PR diff, so the existing corpus of legitimate history is structurally out of its reach. `docs/decisions/**` is exempt wholesale in both modes.
Author
Owner

Closing record

Outcome: Shipped in PR #811 (squash-merged as one commit on main). The rule is generalized from docs/handoffs/chicorytv-issue-queue.md to every durable artifact, with a named destination and an explicit carve-out: docs/decisions/records/docs/no-session-narrative.md (docs.no-session-narrative). The kickoff file now cites the general rule and keeps only its extra file-specific cost. scripts/check-doc-narrative.py runs as a second, non-blocking step in docs-reminder.

Root cause: Not a bug — the shape #773 measures. The rule was established where it was first noticed (a file pasted into every session, whose file-specific reason was "narrative here taxes every future session") and never extended to its class. The general reason is different and broader: a reader coming to any doc cold never saw the earlier draft. Same shape as #743 and #767.

Decisions/conventions changed: Added docs.no-session-narrative. Nothing superseded.

Reusable knowledge:

  • A review finding creates pressure to answer IN THE ARTIFACT. The correction and the justification-for-the-correction get written in the same keystroke, and the artifact is the file already open. That is the mechanism, and it is why the rule names a destination (the commit message) rather than only a prohibition — "don't write it in the doc" with no home loses the knowledge, which is the #542 failure.
  • A detector over git diff output is a string predicate too, one level down. Deciding what a diff line IS from its prefix, without hunk state, produced four defects in one bash implementation (the no-newline marker as content; an added line whose text began ++ eaten by the +++ arm; core.quotePath hiding non-ASCII paths; read dropping a final unterminated line). Four sites, one mistake. Replacing the mechanism converged; patching the sites would not have.
  • Git's OUTPUT FORMAT is configurable, and that is a channel, not a list of knobs. Three review rounds each found exactly one more config key turning a real hit into a clean-looking scanned 0 file(s) — core.quotePath, then diff.dstPrefix, then color.diff/color.ui. Pinning the third one would have been a fourth variant patch. What closed it: remove the surface (GIT_CONFIG_GLOBAL/SYSTEM/NOSYSTEM → devnull), pin each named knob on the command line where it also beats a repo-local config, and put every demonstrated knob in a FORMAT_KNOBS table so the next one found becomes a row instead of a round. An independent 46-configuration matrix then found zero false cleans.
  • A "coverage boundary" sentence can itself mis-sort. Writing "the parse path and the FORMAT PINS carry mutation proofs" put three pins on the proven side that survive mutation — including one the script's own comment already disclaimed, two files apart. Fix: name the clauses, not the category, and measure the sort. All 19 were disarmed individually; 12 witnessed red, 7 green.
  • grep -c for hit-counting, and git diff -U0 hunk headers give correct new-file line numbers if and only if you track hunk state.

Verification: scripts/tests/ 807 passed, 2 skipped. New scripts/tests/test_check_doc_narrative.py (36 tests) with a PATTERNS mutation proof, and every load-bearing clause disarmed and witnessed red. prove-fix reports PROVEN for a behaviour reason. Positive control for --diff (a narrative line in a doc, in a decision record, and in the script → only the doc flagged, correct line number, exit 0) and negative control (this PR's own diff → nothing to flag). CI green on d938493d5; the new step's execution confirmed in the real job log (doc-narrative: scanned 5 file(s)) rather than inferred from a green job. ruff check clean.

Deferred: Remediation of the 21 existing sites the sweep found — #812. They are 21 individual who-benefits judgements, not a regex: a material fraction (the docs/remote-state-inventory.md cluster especially) are load-bearing findings that must stay. Also deferred, and recorded in the record rather than implied covered: the error/degradation arms (git()'s OSError return, run_all's sentinel, open(errors=), the top-level except, -U0, --src-prefix, the env overrides) are defensive and unproven by design.

Docs updated: docs/decisions/records/docs/no-session-narrative.md (new), docs/decisions/README.md (regenerated), docs/README.md (task-signal row), docs/ci-cd.md (docs-reminder now documents both steps), docs/handoffs/chicorytv-issue-queue.md, docs/guard-inventory.md (TOOLING + PROOF rows, counts), docs/remote-state-inventory.md (N/A row).

Review: eight cold rounds in isolated worktrees. Rounds 1–5 returned BLOCKED and each was right; the last three verdicts were MERGEABLE. Two findings were ones I had reached independently (the colour knob, and two of my own tests being vacuous), which is the useful signal — the rest I would not have found.

## Closing record **Outcome:** Shipped in PR #811 (squash-merged as one commit on `main`). The rule is generalized from `docs/handoffs/chicorytv-issue-queue.md` to every durable artifact, with a named destination and an explicit carve-out: `docs/decisions/records/docs/no-session-narrative.md` (`docs.no-session-narrative`). The kickoff file now cites the general rule and keeps only its extra file-specific cost. `scripts/check-doc-narrative.py` runs as a second, non-blocking step in `docs-reminder`. **Root cause:** Not a bug — the shape #773 measures. The rule was established where it was first noticed (a file pasted into every session, whose *file-specific* reason was "narrative here taxes every future session") and never extended to its class. The general reason is different and broader: a reader coming to any doc cold never saw the earlier draft. Same shape as #743 and #767. **Decisions/conventions changed:** Added `docs.no-session-narrative`. Nothing superseded. **Reusable knowledge:** - **A review finding creates pressure to answer IN THE ARTIFACT.** The correction and the justification-for-the-correction get written in the same keystroke, and the artifact is the file already open. That is the mechanism, and it is why the rule names a destination (the commit message) rather than only a prohibition — "don't write it in the doc" with no home loses the knowledge, which is the #542 failure. - **A detector over `git diff` output is a string predicate too, one level down.** Deciding what a diff line IS from its prefix, without hunk state, produced four defects in one bash implementation (the no-newline marker as content; an added line whose text began `++ ` eaten by the `+++ ` arm; `core.quotePath` hiding non-ASCII paths; `read` dropping a final unterminated line). Four sites, one mistake. Replacing the mechanism converged; patching the sites would not have. - **Git's OUTPUT FORMAT is configurable, and that is a channel, not a list of knobs.** Three review rounds each found exactly one more config key turning a real hit into a clean-looking `scanned 0 file(s)` — `core.quotePath`, then `diff.dstPrefix`, then `color.diff`/`color.ui`. Pinning the third one would have been a fourth variant patch. What closed it: remove the surface (`GIT_CONFIG_GLOBAL`/`SYSTEM`/`NOSYSTEM` → devnull), pin each *named* knob on the command line where it also beats a repo-local config, and put every demonstrated knob in a `FORMAT_KNOBS` table so the next one found becomes a row instead of a round. An independent 46-configuration matrix then found zero false cleans. - **A "coverage boundary" sentence can itself mis-sort.** Writing "the parse path and the FORMAT PINS carry mutation proofs" put three pins on the proven side that survive mutation — including one the script's own comment already disclaimed, two files apart. Fix: name the clauses, not the category, and *measure* the sort. All 19 were disarmed individually; 12 witnessed red, 7 green. - **`grep -c` for hit-counting, and `git diff -U0` hunk headers give correct new-file line numbers** if and only if you track hunk state. **Verification:** `scripts/tests/` 807 passed, 2 skipped. New `scripts/tests/test_check_doc_narrative.py` (36 tests) with a `PATTERNS` mutation proof, and every load-bearing clause disarmed and witnessed red. `prove-fix` reports PROVEN for a behaviour reason. Positive control for `--diff` (a narrative line in a doc, in a decision record, and in the script → only the doc flagged, correct line number, exit 0) and negative control (this PR's own diff → nothing to flag). CI green on `d938493d5`; the new step's execution confirmed in the real job log (`doc-narrative: scanned 5 file(s)`) rather than inferred from a green job. `ruff check` clean. **Deferred:** Remediation of the 21 existing sites the sweep found — **#812**. They are 21 individual who-benefits judgements, not a regex: a material fraction (the `docs/remote-state-inventory.md` cluster especially) are load-bearing findings that must stay. Also deferred, and recorded in the record rather than implied covered: the error/degradation arms (`git()`'s `OSError` return, `run_all`'s sentinel, `open(errors=)`, the top-level `except`, `-U0`, `--src-prefix`, the env overrides) are defensive and unproven by design. **Docs updated:** `docs/decisions/records/docs/no-session-narrative.md` (new), `docs/decisions/README.md` (regenerated), `docs/README.md` (task-signal row), `docs/ci-cd.md` (`docs-reminder` now documents both steps), `docs/handoffs/chicorytv-issue-queue.md`, `docs/guard-inventory.md` (TOOLING + PROOF rows, counts), `docs/remote-state-inventory.md` (`N/A` row). **Review:** eight cold rounds in isolated worktrees. Rounds 1–5 returned BLOCKED and each was right; the last three verdicts were MERGEABLE. Two findings were ones I had reached independently (the colour knob, and two of my own tests being vacuous), which is the useful signal — the rest I would not have found.
timothy removed the in-progress label 2026-08-22 02:24:05 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#784