feat(610): split the decision corpus into one YAML-frontmatter file per record #619

2026-07-25T19:16:44+02:00

timothy commented

2026-07-25 19:16:44 +02:00

fixes #610

168 records → docs/decisions/records/<area>/<topic>.md (164 active across 23 dirs) and docs/decisions/archive/<area>/<topic>.md. The filename is the key, so one-active-record-per-key becomes a filesystem property rather than a validator check, and supersession becomes a git mv.

Read the harness, not the diff

This is a ~190-file diff. It cannot be meaningfully reviewed by reading it, and it isn't meant to be — correctness rests on a field-level equivalence proof:

168 records before and after · zero lost, zero gained, zero field mismatches, zero rationale bodies differing

The load-bearing design choice is that the parser was taught both formats first, dispatching on a leading ---. That means the body-diff guard parses the old form at the merge-base and the new form at head, both keyed on key — so the migration validates itself, with no bypass and no one-time escape hatch. decisions-validate --base origin/main --head HEAD exits 0 with the guard live (no Decisions-Edit trailer in the range).

The equivalence harness runs against a checked-in fixture, not the live corpus. The first version migrated the real tree, which made it a one-shot: the moment the migration landed there was nothing left to move and the tests failed for reasons unrelated to the code. A fixture keeps them testing the script rather than the repo's current state. The one-off proof against the real corpus was run separately and is quoted above.

What measuring caught that reading would not have

~500 lines sit outside any record — decisions.md's lifecycle schema and each topic file's preamble, mostly the only copy of that narrative. Source files are kept and stripped, never deleted. They also can't be filed per-area: topic files hold several areas, and 4 of 23 areas span several files.
Archive discovery was a non-recursive glob. After the split it found zero archived records, surfacing as four bogus "supersedes points to unknown key" errors rather than an obvious failure.
~32 live docs point into the corpus by date (see decisions.md 2026-07-10), which the split dangles. Each stripped file now ends with a generated "Records formerly in this file" index — which also rescues the identical breadcrumbs in historical issue comments that cannot be rewritten at all.
decisions.md's "In this file:" list was 97 same-file anchor bullets that the split makes wrong, not merely stale. Dropped; the generated index replaces them with links that resolve.
.gitignore's [Rr]elease/ build-output rule silently swallowed records/release/ — all 9 release records were excluded from the first commit. Caught only by the body-diff guard reporting them as removed-without-archive. The directory is now explicitly un-ignored (negating the files alone wouldn't work — git never descends into an excluded directory). Same class as the core rule that swallows files under any Core/.
The generated index heading is itself an H2, so the parser counted one unmigrated record per stripped file — the notice went 0 → 6 while the corpus still validated OK.

Deliberate, and stated rather than buried

Keys preserved verbatim, warts included. sched (12) and scheduling (1) are now two directories for one concept, alongside seven single-record areas. Renaming a key is not a move — it changes identity, breaks the equivalence proof, and invalidates MemPalace's per-key drawers. Taxonomy normalisation is separate work.
The budget now counts prose, excluding frontmatter. This flatters the number, so: it is a change of metric, not a consolidation. The corpus reads 5228/5600 — under budget for the first time — but no knowledge was removed. Consolidation is still outstanding and now has an owner: #620, filed precisely because a future session reading only the quiet warning would infer the corpus got consolidated when it did not. It is blocked-by #610, so it surfaces in the queue on merge.

Blocking follow-up before this is useful

MemPalace's exporter (server-management#642) must be updated in step — filed as server-management#663. Keys survive, so identity is intact, but paths and file format both change. An un-updated exporter does not error — it silently finds nothing, which is the exact failure mode the retrieval contract exists to prevent. Cross-repo, so it cannot land here; deliberately NOT a Done-when box on #610 because it is circular — an exporter cannot be verified against a layout that has not landed.

Verification

70 tests; ruff, format and pyright clean; decisions-validate: OK; catalog --check clean. Trial-run repeatedly against disposable worktree copies before touching the real corpus.

🤖 Generated with Claude Code

fixes #610 168 records → `docs/decisions/records/<area>/<topic>.md` (164 active across 23 dirs) and `docs/decisions/archive/<area>/<topic>.md`. **The filename is the key**, so one-active-record-per-key becomes a filesystem property rather than a validator check, and supersession becomes a `git mv`. ## Read the harness, not the diff This is a ~190-file diff. It cannot be meaningfully reviewed by reading it, and it isn't meant to be — **correctness rests on a field-level equivalence proof**: > 168 records before and after · **zero lost, zero gained, zero field mismatches, zero rationale bodies differing** The load-bearing design choice is that the parser was taught **both** formats first, dispatching on a leading `---`. That means the body-diff guard parses the *old* form at the merge-base and the *new* form at head, both keyed on `key` — so **the migration validates itself**, with no bypass and no one-time escape hatch. `decisions-validate --base origin/main --head HEAD` exits 0 with the guard live (no `Decisions-Edit` trailer in the range). The equivalence harness runs against a **checked-in fixture**, not the live corpus. The first version migrated the real tree, which made it a one-shot: the moment the migration landed there was nothing left to move and the tests failed for reasons unrelated to the code. A fixture keeps them testing the *script* rather than the repo's current state. The one-off proof against the real corpus was run separately and is quoted above. ## What measuring caught that reading would not have 1. **~500 lines sit outside any record** — `decisions.md`'s lifecycle schema and each topic file's preamble, mostly the only copy of that narrative. Source files are **kept and stripped**, never deleted. They also can't be filed per-area: topic files hold several areas, and 4 of 23 areas span several files. 2. **Archive discovery was a non-recursive glob.** After the split it found *zero* archived records, surfacing as four bogus "supersedes points to unknown key" errors rather than an obvious failure. 3. **~32 live docs point into the corpus by date** (`see decisions.md 2026-07-10`), which the split dangles. Each stripped file now ends with a generated **"Records formerly in this file"** index — which also rescues the identical breadcrumbs in historical issue comments that cannot be rewritten at all. 4. **`decisions.md`'s "In this file:" list was 97 same-file anchor bullets** that the split makes *wrong*, not merely stale. Dropped; the generated index replaces them with links that resolve. 5. **`.gitignore`'s `[Rr]elease/` build-output rule silently swallowed `records/release/`** — all 9 release records were excluded from the first commit. Caught *only* by the body-diff guard reporting them as removed-without-archive. The directory is now explicitly un-ignored (negating the files alone wouldn't work — git never descends into an excluded directory). Same class as the `core` rule that swallows files under any `Core/`. 6. **The generated index heading is itself an H2**, so the parser counted one unmigrated record per stripped file — the notice went 0 → 6 while the corpus still validated `OK`. ## Deliberate, and stated rather than buried - **Keys preserved verbatim, warts included.** `sched` (12) and `scheduling` (1) are now two directories for one concept, alongside seven single-record areas. Renaming a key is not a move — it changes identity, breaks the equivalence proof, and invalidates MemPalace's per-key drawers. Taxonomy normalisation is separate work. - **The budget now counts prose, excluding frontmatter.** This flatters the number, so: it is a change of *metric*, not a consolidation. The corpus reads 5228/5600 — under budget for the first time — but no knowledge was removed. **Consolidation is still outstanding and now has an owner: #620**, filed precisely because a future session reading only the quiet warning would infer the corpus got consolidated when it did not. It is blocked-by #610, so it surfaces in the queue on merge. ## Blocking follow-up before this is useful **MemPalace's exporter (server-management#642) must be updated in step** — filed as **[server-management#663](https://gitea.tblindustries.be/timothy/server-management/issues/663)**. Keys survive, so identity is intact, but **paths and file format both change**. An un-updated exporter does not error — it silently finds nothing, which is the exact failure mode the retrieval contract exists to prevent. Cross-repo, so it cannot land here; deliberately NOT a Done-when box on #610 because it is circular — an exporter cannot be verified against a layout that has not landed. ## Verification 70 tests; ruff, format and pyright clean; `decisions-validate: OK`; catalog `--check` clean. Trial-run repeatedly against disposable worktree copies before touching the real corpus. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

timothy added 7 commits 2026-07-25 19:16:45 +02:00

feat(610): dual-format decision-record parser (backtick line + YAML frontmatter) e063fd065f

First step of the one-file-per-record split. `decisions_lib.parse_text` now
dispatches on a leading `---`: the legacy "many ## records per file, metadata on
a backtick line" form and the new "one record per file, YAML frontmatter" form
both produce the same Record.

This is the load-bearing move for the migration. Because the parser understands
both, the body-diff guard can parse the OLD form at the merge-base and the NEW
form at head -- both keyed on `key` -- so the commit that relocates 166 records
validates itself with no one-time bypass and no escape hatch.

Details:
- Frontmatter uses the SAME field vocabulary as the backtick line
  (`superseded-by`, `stale-after`), so on-disk names don't shift mid-migration.
- YAML `None` (a key with no value) is preserved as "" rather than collapsing to
  None, keeping the empty-vs-absent distinction `stale-after` depends on (#603).
- `active_files()` also walks `docs/decisions/records/**`, so both layouts can
  coexist while the migration lands.
- Malformed frontmatter (unterminated fence, non-dict, YAML error) yields no
  record rather than raising; the validator then reports it as a missing record.

feat(610): migration script + field-level equivalence harness d52268f2ed

The migration is a MOVE, so correctness is provable rather than reviewable:
parse the corpus before, migrate, parse after, assert the Record sets are
identical field-for-field with byte-identical rationale.

scripts/migrate_decisions_split.py
  166 records -> docs/decisions/records/<area>/<topic>.md (active) or
  docs/decisions/archive/<area>/<topic>.md (superseded/retired), 26 directories.
  Path is DERIVED from the key, so identity stays the key. Refuses to run if any
  record lacks a key, and aborts on a destination collision.

  Source files are KEPT, stripped to their narrative -- not deleted. 506 lines of
  the corpus sit outside any record: decisions.md's lifecycle-schema header (the
  status vocabulary, supersession rules, the edit-token contract) and each topic
  file's preamble explaining why those records exist. For most of it that is the
  only copy. It also cannot be filed per-area -- topic files hold several areas
  and 4 of 23 areas span several files -- so the files themselves stay.

scripts/tests/test_migration_equivalence.py
  Runs the real migration against a COPY of the real corpus in tmp_path, never
  the working tree. Asserts: all 166 records survive with the same keys, every
  metadata field round-trips, titles round-trip from headings, every rationale
  body is byte-identical, path matches key, archived records land in the archive
  wing, the legacy files keep their narrative, and no parseable record is left
  behind in them.

Proven non-vacuous: corrupting one migrated record's prose is caught by the
byte-identical check, and deleting one is caught by the survival check.

One test-authoring note: an early assertion string-matched "## " to prove no
records were left in decisions.md. That is wrong -- the schema header quotes an
illustrative "## 2026-07-17 ..." example in prose. Whether records remain is a
PARSING question, so the parser-based leftover test is the real invariant.

feat(610): teach the validator and catalog the split layout ef67be924c

Both surfaced in a trial migration against a disposable copy of the real corpus,
not from reading:

- Archive discovery was a NON-recursive glob. After the split, archived records
  live at archive/<area>/<topic>.md, so every one of them became invisible --
  which surfaced as four bogus "supersedes points to unknown key" errors rather
  than as an obvious "no archive found". rglob at both sites.

- Catalog links: a record is now a FILE, so the link is a plain relative path
  with no anchor -- nothing to slug, nothing to keep in sync with a heading. The
  legacy anchor form is kept for records still living inside a multi-record file,
  so the catalog is correct on either side of the migration.

feat(610): budget counts PROSE, excluding YAML frontmatter 1af65b7bee

The line budget exists to bound how much narrative a reader or agent must get
through. Under the split each record carries ~11 frontmatter lines plus two
fences -- 1789 lines across 166 records -- which are the structured restatement
of what used to be one dense backtick line. Counting them inflates the metric
without any new knowledge being added.

Stated plainly because it flatters the number: this is a change of METRIC, not a
consolidation. It re-measures the same corpus, it does not shrink it. Whole-file
counting put the migrated corpus at 6837 against a 5600 budget; prose-only puts
the same content at ~5048. The consolidation work is still worth doing -- it is
simply no longer being signalled by a warning that was partly measuring
punctuation.

Inert pre-migration: no legacy file has frontmatter, so the branch is never
taken and today's number is unchanged.

fix(610): catalog path links only for the split layout, not every file under docs/decisions 8ec1f527e4

The first cut tested `relative_to(TOPIC_DIR)`, which also matches the LEGACY
multi-record topic files -- so every record in workflow-process.md et al. lost its
anchor and linked to the top of the file instead of to its own record. Caught by
`build_decisions_catalog --check` going stale on the unmigrated corpus, not by
reading. Narrowed to `RECORDS_DIR in src.parents`, which by construction only
matches one-record-per-file.

feat(610): generated where-did-it-go index on each stripped file 64b65fd2db

Live docs point into the corpus BY DATE -- "see `decisions.md` 2026-07-10" --
about 32 such references across 12 files, plus the same form in historical issue
comments. The split would dangle every one of them.

Each stripped file now ends with a generated "Records formerly in this file"
index: date, title, and a link to the record's new path. A reader following a
date pointer lands on the file it names and resolves from there. That is far
cheaper and less error-prone than rewriting 32 references by hand, and it also
covers the issue-comment breadcrumbs, which cannot be rewritten at all.

Caught while verifying it: the generated `## Records formerly in this file`
heading is itself an H2, so the record parser counted one legacy-unmigrated
record per stripped file -- the notice went 0 -> 6. Same treatment as the
existing `## Index` section: skip it by name. SKIP_HEADINGS moved to
decisions_lib as the single source of truth, since three modules now need it.

Found by reading the validator's notice output on a trial migration, not by
inspection -- the corpus still validated OK, so nothing else would have flagged it.

fix(610): de-brittle the count assertion; stop conflating index lines with preserved prose 8578dc1ca7

Both surfaced when main gained two records mid-flight.

- test_record_count_is_the_expected_166 hardcoded the total, so it failed the
  moment a record landed on main -- a merge turning an unrelated test red. The
  real invariant is before == after; the count only needs to prove the harness
  isn't parsing a stub corpus, so it is now equality plus a floor.

- The migration's "lines preserved" figure silently absorbed the generated
  where-did-it-go index once that was threaded into the preamble string, jumping
  507 -> 759 with no new prose preserved. It now reports the two separately:
  514 lines of original prose, plus 245 generated index lines. A number that
  quietly changes meaning is worse than no number.

#	finding	fix
1	105 dangling links	links now `os.path.relpath(dest, src.parent)` — relative to the file they live in
2	link test resolved from the wrong base	walks every file with the index, resolves each link from `f.parent`, + vacuity guard
3	byte-identity proof was circular — baseline used `mig._rationale_body`, the same function the migration writes with	baseline now uses a deliberately separate `_independent_strip`
4	`.gitignore` fix was a spot-fix (`release/` only)	nested `docs/decisions/.gitignore` with `!*/` — fixes the class
5	`_archive_records_at` didn't filter `SKIP_HEADINGS`	filtered, matching `_records_at`

scenario	result
key renamed, same heading + body	`demoted` — blocks
key + heading + body all changed	`removed` — no escape via total identity swap
archive-move + retitle + rewrite (laundering)	`rewritten` — caught
archive-move + retitle, body byte-kept	clean, correctly
retitle only	clean — the trap it was built to retire

feat(610): split the decision corpus into one YAML-frontmatter file per record #619

Read the harness, not the diff

What measuring caught that reading would not have

Deliberate, and stated rather than buried

Blocking follow-up before this is useful

Verification

Adversarial review — BLOCKED, then fixed and re-verified

Five findings, all fixed at the class level

Re-review at fba5233c — each fix mutation-proven

The lossless-move proof re-holds

Re-review of the final two commits — MERGEABLE

The riskiest change survived every escape attempt

#609's work is intact

The hand-rolled reader held up better than expected

Re-review at `fba5233c` — each fix mutation-proven