feat(610): split the decision corpus into one YAML-frontmatter file per record #619

Merged
timothy merged 10 commits from feat/610-decisions-one-file-per-record into main 2026-07-25 20:36:08 +02:00
10 Commits
Author SHA1 Message Date
timothy 02c82b35ea fix(610): make the frontmatter READ path dependency-free — CI has no PyYAML
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 12s
PR Gates / Docs update reminder (pull_request) Successful in 17s
PR Gates / decisions lifecycle (pull_request) Successful in 17s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m15s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 6s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 16m41s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 20m6s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
This is why `decisions lifecycle` went red, and it was NOT the known flake. I
came close to dismissing it as one for the second time this session, because an
earlier red on another branch genuinely was.

The dual-format parser imported PyYAML to read frontmatter. `decisions-guard`
does `setup-python` and installs NOTHING, so once the corpus was migrated every
record became unparseable there: ModuleNotFoundError, job fails. The same would
hit the Husky pre-commit hook and every contributor's machine.

Installing PyYAML in CI is the wrong fix: READING happens everywhere -- CI, the
hook, every dev -- while WRITING happens once, in a migration a human runs
deliberately. So the read path is now dependency-free and only
`migrate_decisions_split` (the writer) still imports yaml.

A hand-rolled parser is only safe if it provably matches the library that WROTE
the files, so `test_frontmatter_reader_matches_pyyaml_on_every_real_record`
compares the two field-by-field across all 169 real records (importorskip, so it
is skipped rather than failing where PyYAML is absent) with a >100-file guard
against near-vacuity. It is narrow by construction: the frontmatter is
machine-generated with default_flow_style=False and width=10**9, so every value
is a single-line scalar, and the reader bails to None on anything nested.

Verified by running all four affected entry points against a shim that makes
`import yaml` raise: validate --base/--head, catalog --check, the kickoff guard,
and the plain validate the pre-commit hook calls. All exit 0.

refs #610
2026-07-25 19:57:22 +02:00
timothy 52786a545c feat(610): re-key the body-diff guard to key; enforce path<->key; pin the filesystem invariant
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Failing after 14s
PR Gates / Docs update reminder (pull_request) Successful in 19s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m24s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 21m19s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 22m41s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
Three Done-when items that were genuinely unfinished. I audited the checklist
before ticking it rather than after, and these were not done.

1. BODY-DIFF GUARD RE-KEYED FROM HEADING TO `key`. Heading-keying made a pure
   RENAME look like a removal plus an unrelated addition, so retitling a record
   failed CI as an "unlogged removal" -- a trap that has bitten this repo before.
   Records are now MATCHED by key and REPORTED by heading (a key alone is not
   enough for a human to find the record). Demotion, which is inherently about a
   record losing its key, is detected on the heading axis instead, and a demoted
   record is no longer double-reported as removed. Key-matching is also what lets
   the comparison work straight through the legacy->frontmatter migration, since
   `key` survives both the format change and the move between files.

   Pinned by a test driving the REAL git-backed diff engine: retitle a record,
   same key, byte-identical prose -> removed/rewritten/demoted all empty.
   Mutation-verified: restoring heading-keying makes it fail with exactly
   "a retitle was reported as a removal".

2. PATH <-> KEY ENFORCED BY THE VALIDATOR. The filename is derived from the key,
   which is what makes one-active-per-key a filesystem property rather than a
   check -- but only if the two cannot drift. Now an error when they disagree,
   with legacy multi-record files explicitly exempt (they have no key-derived
   path to match).

3. ONE-ACTIVE-PER-KEY PINNED AS A FILESYSTEM PROPERTY. Two records sharing a key
   derive the same path, so the filesystem refuses the second; the migration
   aborts on a destination collision rather than silently overwriting. Both are
   now tested, the latter end-to-end through `main()`.

Also fixed a vacuous assertion ruff caught in my own new test -- `assert X or
True` is always true.

Note on verification: my first positive control for the guard reported it NOT
firing. That was the probe, not the code -- "load 340" appears in both the
`sources:` frontmatter field and the body, and replace(...,1) hit the metadata
copy, which is correctly token-free. Re-run against genuine body prose, the guard
fires with exit 1.

refs #610
2026-07-25 19:53:08 +02:00
timothy fba5233caf feat(610): split the decision corpus into one YAML-frontmatter file per record
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 11s
PR Gates / Docs update reminder (pull_request) Successful in 16s
PR Gates / decisions lifecycle (pull_request) Failing after 23s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 1m17s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 1m29s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 8m5s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 16m5s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 17m6s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Has been skipped
168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and
docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key,
so one-active-record-per-key becomes a filesystem property rather than a
validator check, and supersession becomes a `git mv`.

WHY: the monolith was a concurrency problem before an aesthetic one. A
3,900-line append target made parallel sessions collide -- PR #605 and PR #614
both hit append-vs-append conflicts during routine rebases, and hand-resolving
those inside the corpus is exactly the operation the rationale-rewrite guard
exists to police.

HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness
does not rest on reading it. The parser was taught BOTH formats first, so the
body-diff guard parses the old form at the merge-base and the new form at head --
the migration validates itself, no bypass. The proof is a field-level equivalence
harness: 168 records before and after, zero lost, zero gained, zero field
mismatches, zero rationale bodies differing. Reviewers should scrutinise the
harness; it is the actual evidence.

What measuring caught that reading would not have:

- ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each
  topic file's preamble, mostly the only copy. Source files are kept and
  stripped, never deleted. They also cannot be filed per-area: topic files hold
  several areas and 4 of 23 areas span several files.
- Archive discovery was a non-recursive glob; after the split it found ZERO
  archived records, surfacing as four bogus "supersedes points to unknown key"
  errors rather than an obvious failure.
- ~32 live docs point into the corpus BY DATE, which the split dangles. Each
  stripped file now ends with a generated "Records formerly in this file" index,
  which also rescues the identical breadcrumbs in old issue comments.
- decisions.md's "In this file:" list was 97 same-file anchor bullets that the
  split makes WRONG, not merely stale. Dropped; the generated index replaces
  them with links that resolve.

The equivalence harness now runs against a checked-in FIXTURE, not the live
corpus. The earlier version migrated the real tree, which made it a one-shot:
the moment the migration landed there was nothing left to move and the tests
failed for reasons unrelated to the code. A fixture keeps them testing the
SCRIPT rather than the repo's current state.

Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain
two directories for one concept. Renaming a key is not a move -- it changes
identity, breaks the equivalence proof, and invalidates MemPalace's per-key
drawers. Taxonomy normalisation is separate work.

refs #610
2026-07-25 19:45:09 +02:00
timothy 8578dc1ca7 fix(610): de-brittle the count assertion; stop conflating index lines with preserved prose
Both surfaced when main gained two records mid-flight.

- test_record_count_is_the_expected_166 hardcoded the total, so it failed the
  moment a record landed on main -- a merge turning an unrelated test red. The
  real invariant is before == after; the count only needs to prove the harness
  isn't parsing a stub corpus, so it is now equality plus a floor.

- The migration's "lines preserved" figure silently absorbed the generated
  where-did-it-go index once that was threaded into the preamble string, jumping
  507 -> 759 with no new prose preserved. It now reports the two separately:
  514 lines of original prose, plus 245 generated index lines. A number that
  quietly changes meaning is worse than no number.
2026-07-25 19:08:58 +02:00
timothy 64b65fd2db feat(610): generated where-did-it-go index on each stripped file
Live docs point into the corpus BY DATE -- "see `decisions.md` 2026-07-10" --
about 32 such references across 12 files, plus the same form in historical issue
comments. The split would dangle every one of them.

Each stripped file now ends with a generated "Records formerly in this file"
index: date, title, and a link to the record's new path. A reader following a
date pointer lands on the file it names and resolves from there. That is far
cheaper and less error-prone than rewriting 32 references by hand, and it also
covers the issue-comment breadcrumbs, which cannot be rewritten at all.

Caught while verifying it: the generated `## Records formerly in this file`
heading is itself an H2, so the record parser counted one legacy-unmigrated
record per stripped file -- the notice went 0 -> 6. Same treatment as the
existing `## Index` section: skip it by name. SKIP_HEADINGS moved to
decisions_lib as the single source of truth, since three modules now need it.

Found by reading the validator's notice output on a trial migration, not by
inspection -- the corpus still validated OK, so nothing else would have flagged it.
2026-07-25 19:08:58 +02:00
timothy 8ec1f527e4 fix(610): catalog path links only for the split layout, not every file under docs/decisions
The first cut tested `relative_to(TOPIC_DIR)`, which also matches the LEGACY
multi-record topic files -- so every record in workflow-process.md et al. lost its
anchor and linked to the top of the file instead of to its own record. Caught by
`build_decisions_catalog --check` going stale on the unmigrated corpus, not by
reading. Narrowed to `RECORDS_DIR in src.parents`, which by construction only
matches one-record-per-file.
2026-07-25 19:08:58 +02:00
timothy 1af65b7bee feat(610): budget counts PROSE, excluding YAML frontmatter
The line budget exists to bound how much narrative a reader or agent must get
through. Under the split each record carries ~11 frontmatter lines plus two
fences -- 1789 lines across 166 records -- which are the structured restatement
of what used to be one dense backtick line. Counting them inflates the metric
without any new knowledge being added.

Stated plainly because it flatters the number: this is a change of METRIC, not a
consolidation. It re-measures the same corpus, it does not shrink it. Whole-file
counting put the migrated corpus at 6837 against a 5600 budget; prose-only puts
the same content at ~5048. The consolidation work is still worth doing -- it is
simply no longer being signalled by a warning that was partly measuring
punctuation.

Inert pre-migration: no legacy file has frontmatter, so the branch is never
taken and today's number is unchanged.
2026-07-25 19:08:58 +02:00
timothy ef67be924c feat(610): teach the validator and catalog the split layout
Both surfaced in a trial migration against a disposable copy of the real corpus,
not from reading:

- Archive discovery was a NON-recursive glob. After the split, archived records
  live at archive/<area>/<topic>.md, so every one of them became invisible --
  which surfaced as four bogus "supersedes points to unknown key" errors rather
  than as an obvious "no archive found". rglob at both sites.

- Catalog links: a record is now a FILE, so the link is a plain relative path
  with no anchor -- nothing to slug, nothing to keep in sync with a heading. The
  legacy anchor form is kept for records still living inside a multi-record file,
  so the catalog is correct on either side of the migration.
2026-07-25 19:08:58 +02:00
timothy d52268f2ed feat(610): migration script + field-level equivalence harness
The migration is a MOVE, so correctness is provable rather than reviewable:
parse the corpus before, migrate, parse after, assert the Record sets are
identical field-for-field with byte-identical rationale.

scripts/migrate_decisions_split.py
  166 records -> docs/decisions/records/<area>/<topic>.md (active) or
  docs/decisions/archive/<area>/<topic>.md (superseded/retired), 26 directories.
  Path is DERIVED from the key, so identity stays the key. Refuses to run if any
  record lacks a key, and aborts on a destination collision.

  Source files are KEPT, stripped to their narrative -- not deleted. 506 lines of
  the corpus sit outside any record: decisions.md's lifecycle-schema header (the
  status vocabulary, supersession rules, the edit-token contract) and each topic
  file's preamble explaining why those records exist. For most of it that is the
  only copy. It also cannot be filed per-area -- topic files hold several areas
  and 4 of 23 areas span several files -- so the files themselves stay.

scripts/tests/test_migration_equivalence.py
  Runs the real migration against a COPY of the real corpus in tmp_path, never
  the working tree. Asserts: all 166 records survive with the same keys, every
  metadata field round-trips, titles round-trip from headings, every rationale
  body is byte-identical, path matches key, archived records land in the archive
  wing, the legacy files keep their narrative, and no parseable record is left
  behind in them.

Proven non-vacuous: corrupting one migrated record's prose is caught by the
byte-identical check, and deleting one is caught by the survival check.

One test-authoring note: an early assertion string-matched "## " to prove no
records were left in decisions.md. That is wrong -- the schema header quotes an
illustrative "## 2026-07-17 ..." example in prose. Whether records remain is a
PARSING question, so the parser-based leftover test is the real invariant.
2026-07-25 19:08:58 +02:00
timothy e063fd065f feat(610): dual-format decision-record parser (backtick line + YAML frontmatter)
First step of the one-file-per-record split. `decisions_lib.parse_text` now
dispatches on a leading `---`: the legacy "many ## records per file, metadata on
a backtick line" form and the new "one record per file, YAML frontmatter" form
both produce the same Record.

This is the load-bearing move for the migration. Because the parser understands
both, the body-diff guard can parse the OLD form at the merge-base and the NEW
form at head -- both keyed on `key` -- so the commit that relocates 166 records
validates itself with no one-time bypass and no escape hatch.

Details:
- Frontmatter uses the SAME field vocabulary as the backtick line
  (`superseded-by`, `stale-after`), so on-disk names don't shift mid-migration.
- YAML `None` (a key with no value) is preserved as "" rather than collapsing to
  None, keeping the empty-vs-absent distinction `stale-after` depends on (#603).
- `active_files()` also walks `docs/decisions/records/**`, so both layouts can
  coexist while the migration lands.
- Malformed frontmatter (unterminated fence, non-dict, YAML error) yields no
  record rather than raising; the validator then reports it as a missing record.
2026-07-25 19:08:58 +02:00