feat(610): split the decision corpus into one YAML-frontmatter file per record

168 records -> docs/decisions/records/<area>/<topic>.md (163 active, 23 dirs) and docs/decisions/archive/<area>/<topic>.md (5 archived). The filename IS the key, so one-active-record-per-key becomes a filesystem property rather than a validator check, and supersession becomes a `git mv`. WHY: the monolith was a concurrency problem before an aesthetic one. A 3,900-line append target made parallel sessions collide -- PR #605 and PR #614 both hit append-vs-append conflicts during routine rebases, and hand-resolving those inside the corpus is exactly the operation the rationale-rewrite guard exists to police. HOW IT IS VERIFIED: a ~170-file diff cannot be meaningfully read, so correctness does not rest on reading it. The parser was taught BOTH formats first, so the body-diff guard parses the old form at the merge-base and the new form at head -- the migration validates itself, no bypass. The proof is a field-level equivalence harness: 168 records before and after, zero lost, zero gained, zero field mismatches, zero rationale bodies differing. Reviewers should scrutinise the harness; it is the actual evidence. What measuring caught that reading would not have: - ~500 lines sit OUTSIDE any record -- decisions.md's lifecycle schema and each topic file's preamble, mostly the only copy. Source files are kept and stripped, never deleted. They also cannot be filed per-area: topic files hold several areas and 4 of 23 areas span several files. - Archive discovery was a non-recursive glob; after the split it found ZERO archived records, surfacing as four bogus "supersedes points to unknown key" errors rather than an obvious failure. - ~32 live docs point into the corpus BY DATE, which the split dangles. Each stripped file now ends with a generated "Records formerly in this file" index, which also rescues the identical breadcrumbs in old issue comments. - decisions.md's "In this file:" list was 97 same-file anchor bullets that the split makes WRONG, not merely stale. Dropped; the generated index replaces them with links that resolve. The equivalence harness now runs against a checked-in FIXTURE, not the live corpus. The earlier version migrated the real tree, which made it a one-shot: the moment the migration landed there was nothing left to move and the tests failed for reasons unrelated to the code. A fixture keeps them testing the SCRIPT rather than the repo's current state. Keys preserved verbatim, warts included: `sched` (12) and `scheduling` (1) remain two directories for one concept. Renaming a key is not a move -- it changes identity, breaks the equivalence proof, and invalidates MemPalace's per-key drawers. Taxonomy normalisation is separate work. refs #610
2026-07-25 19:14:48 +02:00
parent 8578dc1ca7
commit 2dcc88a325
179 changed files with 6843 additions and 5983 deletions
@@ -13,15 +13,19 @@ Two properties this script exists to guarantee, both checked by
 1. **Nothing is rewritten.** Every record's rationale body is copied byte-for-byte, and every
   metadata field round-trips unchanged. The migration is a MOVE, so a field-level before/after
   comparison is a complete correctness proof.
-2. **No prose is lost.** 506 lines of the corpus are NOT inside any record — file preambles that
-   explain why a topic file exists, cross-reference notes, and decisions.md's lifecycle schema
-   header. Those files are kept and stripped to exactly that prose, never deleted. (They cannot be
-   filed per-area: topic files hold several areas, and 4 of 23 areas span several files.)
+2. **No prose is lost.** A few hundred lines of the corpus are NOT inside any record — file
+   preambles that explain why a topic file exists, cross-reference notes, and decisions.md's
+   lifecycle schema header. Those files are kept and stripped to exactly that prose, never deleted.
+   (They cannot be filed per-area: topic files hold several areas, and 4 of 23 areas span several
+   files.) The one thing deliberately dropped is same-file anchor bullets, which the split makes
+   *wrong* rather than merely stale — the generated index replaces them with links that resolve.
+   Run with `--dry-run` for the current counts rather than trusting a number written here.
 """

 from __future__ import annotations

 import argparse
+import re
 from pathlib import Path

 import yaml
@@ -121,8 +125,19 @@ def _moved_index(records: list[dl.Record]) -> str:
    return "\n".join(lines)


+# A bullet whose link is a bare same-file anchor, e.g. `- [2026-07-10 — Title](#2026-07-10--title)`.
+# decisions.md's "In this file:" index is ~97 of these. Once the records move out, every one of them
+# points at a heading that no longer exists in that file — and the generated "Records formerly in
+# this file" section replaces them with links that actually resolve. Anything else keeps its bullet.
+_SAME_FILE_ANCHOR_BULLET = re.compile(r"^\s*[-*]\s*\[[^\]]*\]\(#[^)]*\)\s*$")
+
+
 def _preamble(path: Path, records: list[dl.Record]) -> str:
-    """Everything in `path` that is not inside a record — kept, never deleted."""
+    """Everything in `path` that is not inside a record — kept, never deleted.
+
+    Two exceptions are dropped rather than kept, because the split makes them *wrong* rather than
+    merely stale: same-file anchor bullets (see above).
+    """
    text = path.read_text(encoding="utf-8")
    lines = text.splitlines()
    drop: set[int] = set()
@@ -130,10 +145,16 @@ def _preamble(path: Path, records: list[dl.Record]) -> str:
        start = r.lineno - 1  # heading line, 1-indexed in Record
        for n in range(start, start + 1 + len(r.body.splitlines())):
            drop.add(n)
-    kept = [ln for n, ln in enumerate(lines) if n not in drop]
-    while kept and not kept[-1].strip():
-        kept.pop()
-    return "\n".join(kept)
+    kept = [ln for n, ln in enumerate(lines) if n not in drop and not _SAME_FILE_ANCHOR_BULLET.match(ln)]
+    # collapse the blank runs left behind where a bullet list used to be
+    out: list[str] = []
+    for ln in kept:
+        if not ln.strip() and out and not out[-1].strip():
+            continue
+        out.append(ln)
+    while out and not out[-1].strip():
+        out.pop()
+    return "\n".join(out)


 def plan() -> tuple[list[tuple[dl.Record, Path]], dict[Path, str], dict[Path, int]]:
@@ -0,0 +1,28 @@
+# Decisions — lifecycle log
+
+Purpose prose that must survive the split. This paragraph is the only copy of it.
+
+## Index
+
+**In this file:**
+
+- [2026-01-01 — First example](#2026-01-01--first-example)
+- [2026-01-02 — Second example](#2026-01-02--second-example)
+
+## 2026-01-01 — First example (#1)
+`key: alpha.first` · `status: active` · `since: 2026-01-01` · `supersedes: none` · `superseded-by: none` · `stale-after: 2027-01-01`
+**Rule:** The first rule.
+**Signals:** alpha · paths: a/b.py · issues: #1
+**Mechanics:** docs/x.md → y
+**Sources:** a measurement
+
+First rationale paragraph.
+
+Second paragraph, with an interior blank line above it and `backticks`.
+
+## 2026-01-02 — Second example (#2)
+`key: beta.second` · `status: active` · `since: 2026-01-02` · `supersedes: gamma.old` · `superseded-by: none`
+**Rule:** The second rule.
+**Signals:** beta · issues: #2
+
+Rationale for the second.
@@ -0,0 +1,8 @@
+# Archive
+
+## 2025-12-01 — Retired predecessor (#0)
+`key: gamma.old` · `status: superseded` · `since: 2025-12-01` · `supersedes: none` · `superseded-by: beta.second`
+**Rule:** The old rule.
+**Signals:** gamma · issues: #0
+
+Old rationale that must survive relocation byte-for-byte.
@@ -0,0 +1,10 @@
+# Topic file
+
+Preamble explaining why this topic file exists — the only copy of this narrative.
+
+## 2026-01-03 — Third example (#3)
+`key: alpha.third` · `status: active` · `since: 2026-01-03` · `supersedes: none` · `superseded-by: none`
+**Rule:** The third rule.
+**Signals:** alpha, topic · issues: #3
+
+Third rationale.
@@ -4,8 +4,14 @@ The migration is a MOVE: content relocates, nothing is rewritten. That makes cor
 *provable* rather than reviewable — parse the corpus before, run the migration, parse it after,
 and assert the two Record sets are identical field-for-field with byte-identical rationale.

-These tests run the real migration against a COPY of the real corpus in tmp_path. They never touch
-the working tree.
+**Runs against a checked-in FIXTURE, not the live corpus.** An earlier version migrated the real
+`docs/` tree, which made it a one-shot: the moment the real migration landed, the tree was already
+split, the harness had nothing to move, and the tests failed for a reason that had nothing to do
+with the code under test. A fixture keeps these tests exercising the *script* rather than the
+repo's current state, so they stay meaningful and green after the migration ships.
+
+The one-off proof against the real corpus was run separately at migration time: 168 records before
+and after, zero lost, zero gained, zero field mismatches, zero rationale bodies differing.
 """

 from __future__ import annotations
@@ -18,6 +24,8 @@ import pytest
 import scripts.decisions_lib as dl
 import scripts.migrate_decisions_split as mig

+FIXTURE = Path(__file__).parent / "fixtures" / "premigration"
+
 COMPARED_FIELDS = (
    "key",
    "status",
@@ -30,26 +38,17 @@ COMPARED_FIELDS = (
    "mechanics",
    "sources",
 )
-_SKIP = dl.SKIP_HEADINGS

-
-def _corpus_files() -> list[Path]:
-    files = [p for p in dl.active_files() if p.exists()]
-    files += [p for p in sorted(dl.ARCHIVE_DIR.glob("*.md")) if p.name not in dl._NON_DECISION_FILES]
-    return files
+# The fixture is static, so an exact count is a real completeness guard here rather than a number
+# that rots every time a record lands on main.
+EXPECTED_RECORDS = 4


@pytest.fixture(scope="module")
 def migrated(tmp_path_factory):
-    """Copy the real docs/ into tmp, point decisions_lib at it, run the migration."""
+    """Copy the fixture corpus to tmp, point decisions_lib at it, run the real migration."""
    tmp = tmp_path_factory.mktemp("corpus")
-    shutil.copytree(dl.REPO_ROOT / "docs", tmp / "docs")
-
-    before = {}
-    for f in _corpus_files():
-        for r in dl.parse_file(f):
-            if r.key and r.heading not in _SKIP:
-                before[r.key] = (r, mig._rationale_body(r))
+    shutil.copytree(FIXTURE, tmp / "docs")

    orig = (dl.REPO_ROOT, dl.DECISIONS_MD, dl.TOPIC_DIR, dl.ARCHIVE_DIR, dl.RECORDS_DIR)
    dl.REPO_ROOT = tmp
@@ -58,10 +57,16 @@ def migrated(tmp_path_factory):
    dl.ARCHIVE_DIR = dl.TOPIC_DIR / "archive"
    dl.RECORDS_DIR = dl.TOPIC_DIR / "records"
    try:
+        before = {}
+        for f in [dl.DECISIONS_MD, *sorted(dl.TOPIC_DIR.glob("*.md")), *sorted(dl.ARCHIVE_DIR.rglob("*.md"))]:
+            for r in dl.parse_file(f):
+                if r.key and r.heading not in dl.SKIP_HEADINGS:
+                    before[r.key] = (r, mig._rationale_body(r))
+
        mig.main([])
-        after_files = [p for p in dl.RECORDS_DIR.rglob("*.md")] + [p for p in dl.ARCHIVE_DIR.rglob("*.md")]
+
        after = {}
-        for f in after_files:
+        for f in [*dl.RECORDS_DIR.rglob("*.md"), *dl.ARCHIVE_DIR.rglob("*.md")]:
            for r in dl.parse_file(f):
                if r.key:
                    after[r.key] = (r, r.body)
@@ -70,25 +75,19 @@ def migrated(tmp_path_factory):
        dl.REPO_ROOT, dl.DECISIONS_MD, dl.TOPIC_DIR, dl.ARCHIVE_DIR, dl.RECORDS_DIR = orig


+def test_fixture_is_actually_pre_migration(migrated):
+    """Guard against the harness silently testing nothing if the fixture ever gets migrated."""
+    before, _, _ = migrated
+    assert len(before) == EXPECTED_RECORDS, f"fixture parsed {len(before)}, expected {EXPECTED_RECORDS}"
+
+
 def test_every_record_survives_with_the_same_key(migrated):
    before, after, _ = migrated
-    assert before, "harness parsed nothing before the migration — would pass vacuously"
    assert set(before) == set(after), (
-        f"lost: {sorted(set(before) - set(after))[:5]} / gained: {sorted(set(after) - set(before))[:5]}"
+        f"lost: {sorted(set(before) - set(after))} / gained: {sorted(set(after) - set(before))}"
    )


-def test_record_count_is_stable_across_the_migration(migrated):
-    """Equality is the invariant; the floor only proves the harness isn't parsing a stub corpus.
-
-    Deliberately NOT a hardcoded total — the corpus grows whenever a record lands on main, and a
-    magic number turns every such merge into a spurious failure here.
-    """
-    before, after, _ = migrated
-    assert len(before) == len(after)
-    assert len(before) > 150, f"only {len(before)} records parsed — harness is not seeing the corpus"
-
-
 def test_every_metadata_field_round_trips(migrated):
    before, after, _ = migrated
    diffs = []
@@ -97,7 +96,7 @@ def test_every_metadata_field_round_trips(migrated):
        for f in COMPARED_FIELDS:
            if getattr(b, f) != getattr(a, f):
                diffs.append(f"{key}.{f}: {getattr(b, f)!r} -> {getattr(a, f)!r}")
-    assert not diffs, "\n".join(diffs[:10])
+    assert not diffs, "\n".join(diffs)


 def test_title_round_trips_from_the_heading(migrated):
@@ -107,13 +106,14 @@ def test_title_round_trips_from_the_heading(migrated):
        for k, (b, _) in before.items()
        if b.heading != after[k][0].heading
    ]
-    assert not diffs, "\n".join(diffs[:10])
+    assert not diffs, "\n".join(diffs)


 def test_rationale_bodies_are_byte_identical(migrated):
+    """Including interior blank lines and backticks — the bytes are the payload."""
    before, after, _ = migrated
    diffs = [k for k, (_, body) in before.items() if body != after[k][1]]
-    assert not diffs, f"{len(diffs)} rationale bodies changed, e.g. {diffs[:5]}"
+    assert not diffs, f"{len(diffs)} rationale bodies changed: {diffs}"


 def test_path_matches_key(migrated):
@@ -123,7 +123,7 @@ def test_path_matches_key(migrated):
        area, _, topic = key.partition(".")
        if rec.source.parent.name != area or rec.source.stem != topic:
            bad.append(f"{key} -> {rec.source}")
-    assert not bad, "\n".join(bad[:10])
+    assert not bad, "\n".join(bad)


 def test_archived_records_land_in_the_archive_wing(migrated):
@@ -137,17 +137,13 @@ def test_archived_records_land_in_the_archive_wing(migrated):


 def test_source_files_are_kept_and_retain_their_narrative(migrated):
-    """The ~500 lines of preamble are the only copy of that prose — they must survive."""
+    """Preamble prose is often the only copy — it must survive, not be deleted with the records."""
    _, _, tmp = migrated
    kept = tmp / "docs" / "decisions.md"
-    assert kept.exists(), "decisions.md was deleted — its lifecycle schema is the only copy"
-    text = kept.read_text()
-    assert "Lifecycle, not append-only" in text
-    # NB: do NOT string-match on "## " here — the schema header quotes an illustrative
-    # "## 2026-07-17 — …" example in prose. Whether real records remain is a PARSING question,
-    # covered by test_no_records_remain_in_the_legacy_files.
-    wp = tmp / "docs" / "decisions" / "workflow-process.md"
-    assert wp.exists() and "extracted from" in wp.read_text()
+    assert kept.exists(), "decisions.md was deleted"
+    assert "the only copy of it" in kept.read_text()
+    topic = tmp / "docs" / "decisions" / "topic.md"
+    assert topic.exists() and "the only copy of this narrative" in topic.read_text()


 def test_no_records_remain_in_the_legacy_files(migrated):
@@ -160,3 +156,18 @@ def test_no_records_remain_in_the_legacy_files(migrated):
        if keyed:
            leftovers[f.name] = keyed
    assert not leftovers, leftovers
+
+
+def test_same_file_anchor_bullets_are_dropped(migrated):
+    """The split makes them WRONG, not merely stale — they point at headings that left the file."""
+    _, _, tmp = migrated
+    assert "](#" not in (tmp / "docs" / "decisions.md").read_text(), "a same-file anchor now dangles"
+
+
+def test_moved_index_resolves_to_the_new_paths(migrated):
+    """A date-based pointer from another doc must still resolve after the split."""
+    _, _, tmp = migrated
+    text = (tmp / "docs" / "decisions.md").read_text()
+    assert mig._MOVED_HEADING in text
+    assert "[`alpha.first`](records/alpha/first.md)" in text
+    assert (tmp / "docs" / "decisions" / "records" / "alpha" / "first.md").exists()