docs(520,521): design spec — decision lifecycle + retrieval-efficient startup
Design for the decision-knowledge lifecycle schema/catalog/validator (#521) and the #237-retirement / parallel-startup kickoff rewrite (#520), as one arc. Refs #520 #521 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,205 @@
|
||||
# Decision-lifecycle + retrieval-efficient startup — design (ersatztv #520 + #521)
|
||||
|
||||
Date: 2026-07-21
|
||||
Issues: [#521](http://192.168.1.95:3000/timothy/ersatztv/issues/521) (decision knowledge lifecycle),
|
||||
[#520](http://192.168.1.95:3000/timothy/ersatztv/issues/520) (retire #237 from startup; parallel orientation + selection).
|
||||
Both originate from a bounded knowledge-retrieval audit in `timothy/adversarial-reviewer`.
|
||||
|
||||
## Goal
|
||||
|
||||
Two coupled problems from the same audit:
|
||||
|
||||
1. **#521** — `docs/decisions.md` (~2,468 lines; ~3,469 incl. topic files) is optimized for *append
|
||||
safety*, not *current-state retrieval*. Status lives in prose, supersession is inferred at
|
||||
release time, and fresh agents are told to read the corpus broadly before the task is known.
|
||||
2. **#520** — Startup still tells agents (and the selector fallback) to read the closed arc tracker
|
||||
#237 and its stale forward-looking comments, and preloads the whole doc corpus before the task is
|
||||
known. `scripts/select-queue.sh` already does live mechanical selection correctly.
|
||||
|
||||
The fix is one arc: **give decisions a stable identity + explicit lifecycle + a compact generated
|
||||
active view (#521), then point startup at that compact view and run orientation ‖ selection in
|
||||
parallel (#520).** Keep Git-backed Markdown authoritative; MemPalace stays candidate-discovery only.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Making MemPalace the source of truth; storing live queue state in decision docs.
|
||||
- Reopening #237 or creating a new manual session-log issue / Markdown ledger.
|
||||
- Implementing the MemPalace/Gitea exporter (`server-management#642`).
|
||||
- Changing `select-queue.sh`'s dependency/tier/priority algorithm (only its comments/output text),
|
||||
unless a test exposes a bug.
|
||||
|
||||
## Delegation model (agreed)
|
||||
|
||||
- **Judgment (orchestrator / escalated):** schema, validator, catalog generator, kickoff + docs
|
||||
rewrite, and reconciling the prose-only reversals (#390→#406; #411 measurement obsolescence).
|
||||
- **Mechanical (balanced/cheap subagents, fanned out by topic batch):** reformatting existing
|
||||
entries into the keyed schema. Each subagent gets exact schema rules + the batch's entries + a
|
||||
worktree/branch of its own; results merge back. No two committing agents share a worktree.
|
||||
|
||||
## Guard rework (the crux)
|
||||
|
||||
The current append-only guard blocks **any** modified/deleted line in `decisions.md` unless the
|
||||
commit carries `[decisions-edit]` (`.claude/hooks/decisions-guard.sh`, wired to Husky `commit-msg`
|
||||
and the CI `decisions-guard` job). That line-level mechanic is **incompatible** with a lifecycle
|
||||
model where migration reformats every entry, supersession *edits* a predecessor's `superseded-by:`
|
||||
line, and retiring *moves* a record to archive.
|
||||
|
||||
**Decision:** replace the line-level mechanic with a **lifecycle validator** that enforces the same
|
||||
*spirit* — rationale is never silently rewritten or deleted; every history touch is deliberate and
|
||||
reviewable. The `[decisions-edit]` token is **retired** (removed from the Husky hook, the CI job,
|
||||
and docs/memory references); routine lifecycle writes are expected and are proven well-formed by the
|
||||
validator instead of gated by a token.
|
||||
|
||||
Spirit preserved by these validator invariants (see Validator below): ≤1 active record per key; no
|
||||
record block may vanish (if it leaves the active set it must reappear under `docs/decisions/archive/`);
|
||||
supersession links are reciprocal and resolvable; status only moves active → superseded/retired.
|
||||
|
||||
## Information model (#521)
|
||||
|
||||
Each decision record carries a compact, **visible**, deterministically-parseable metadata block
|
||||
immediately under its `##` heading (front-loaded so MemPalace indexes key/status/rule first):
|
||||
|
||||
```
|
||||
## 2026-07-17 — No persistent compiler servers in CI … (#406)
|
||||
`key: ci.runner-placement` · `status: active` · `since: 2026-07-17` · `supersedes: none` · `superseded-by: none`
|
||||
**Rule:** Every CI `services:` container gets an explicit CPU/mem cap; no shared persistent compiler daemon.
|
||||
**Signals:** ci, runners, services, memory-cap · paths: .gitea/workflows/docker-build.yml · issues: #390 #406
|
||||
**Mechanics:** docs/ci-cd.md → CI lanes
|
||||
<existing rationale prose, incl. rejected alternatives — unchanged>
|
||||
```
|
||||
|
||||
Fields (issue-required):
|
||||
|
||||
- `key` — stable subject identity, dotted lowercase (`^[a-z0-9]+(\.[a-z0-9-]+)+$`), e.g.
|
||||
`ci.runner-placement`. A superseding record reuses the predecessor's key.
|
||||
- `status` — `active` | `superseded` | `retired` | `legacy-unmigrated` (the last is a transitional
|
||||
marker; see Migration). "Relocated"/"consolidated" are maintenance actions, not statuses.
|
||||
- `since` — effective date (YYYY-MM-DD) or version.
|
||||
- `supersedes` / `superseded-by` — predecessor/successor key+date, or `none`.
|
||||
- `**Rule:**` — one-line current rule, before the long rationale.
|
||||
- `**Signals:**` — concepts, `paths:`, `issues:` for task-driven retrieval.
|
||||
- `**Mechanics:**` — link(s) to the current convention/source/test anchors.
|
||||
- Rationale prose (incl. rejected alternatives) follows, unchanged from today where migrated.
|
||||
|
||||
Rejected alternatives stay *inside* their accepted record as rationale; they are never active
|
||||
records.
|
||||
|
||||
## Document architecture (#521)
|
||||
|
||||
- `docs/decisions.md` — stays the chronological record store for in-file (non-topic) decisions, now
|
||||
in schema form. Header rewritten: append-only-log framing → lifecycle framing.
|
||||
- `docs/decisions/*.md` — existing topic files, entries migrated to schema form.
|
||||
- **`docs/decisions/README.md` — NEW generated active catalog / task router.** A compact table
|
||||
(key · one-line rule · status · record link · signals) built by
|
||||
`scripts/build-decisions-catalog.py` from all `active` records across `decisions.md` + topic files.
|
||||
Regenerated like `endpoint-index.md`; drift fails CI. This is the compact "active view" startup
|
||||
points at.
|
||||
- **`docs/decisions/archive/` — NEW.** `superseded`/`retired` records relocated here, out of the
|
||||
default startup path, with forward/back links preserved. Not scanned by the catalog generator.
|
||||
|
||||
## Validator (#521)
|
||||
|
||||
`scripts/decisions-validate.py`, run locally (a Husky hook replacing the append-only commit-msg
|
||||
check) and in the CI `decisions-guard` job (renamed → lifecycle guard). Checks:
|
||||
|
||||
1. Required metadata present + well-formed on every structured record.
|
||||
2. Valid `status` enum; valid `key` format.
|
||||
3. **≤1 `active` record per key.**
|
||||
4. Reciprocal, resolvable `supersedes`/`superseded-by` links (successor exists; predecessor points
|
||||
back).
|
||||
5. **No-vanish (diff-aware):** any record block removed from the active files (`decisions.md` +
|
||||
topic files) between base and HEAD must be present under `docs/decisions/archive/`. Fail-open on
|
||||
missing refs, like the current guard.
|
||||
6. Active-catalog completeness + freshness (regenerate to a temp file, diff — like `check:api`).
|
||||
7. Archive/active placement consistency (no `active` record in archive; no `superseded`/`retired`
|
||||
in the active set once migration of its key is complete).
|
||||
8. **Aggregate active-corpus budget** — sum of lines across `decisions.md` + topic files + catalog
|
||||
(excluding archive). Warn over a threshold (start at the current 1800 applied to the aggregate;
|
||||
tune during migration). Report the `legacy-unmigrated` remainder count; it must be visible and
|
||||
trend to zero.
|
||||
9. Best-effort broken-link check on `Mechanics:`/archive links where practical.
|
||||
|
||||
Fail-open on tooling trouble, matching the current guard's philosophy.
|
||||
|
||||
## Startup / kickoff rewrite (#520)
|
||||
|
||||
- **`docs/handoffs/chicorytv-issue-queue.md`** — retire #237 from normal startup. Two concurrent
|
||||
tracks when no issue is named: **orientation** (`AGENTS.md`, `CLAUDE.md`, compact `docs/README.md`
|
||||
map + the active catalog) ‖ **selection** (`scripts/select-queue.sh 5`). After both: resolve only
|
||||
`CLAIM?`/`UMBRELLA?` flags + the winner, recheck live state, claim, then build a focused
|
||||
task-specific knowledge packet. Named-issue path skips selection → focused retrieval directly.
|
||||
Archive the contradictory *pre-script* selector lore (the "re-derive the whole contested tier by
|
||||
hand" bullets) into a clearly-labeled historical section. #237 appears only as labeled archival
|
||||
history.
|
||||
- **`docs/README.md`** — mandatory 1–10 reading order → compact **task-signal → minimal sections**
|
||||
authority/task map, plus a pointer to `docs/decisions/README.md` as the decision active view.
|
||||
- **`CLAUDE.md`** — docs-first guidance requires the map + relevant sections, not the whole corpus.
|
||||
Task-completion protocol gains the structured `## Closing record` template.
|
||||
- **`scripts/select-queue.sh`** — comments/output: active tiers lead with open
|
||||
milestones/review/priorities; drop "read #237" phrasing.
|
||||
- **Structured closing record** — documented template (Outcome / Root cause / Decisions-conventions
|
||||
changed / Reusable knowledge / Verification / Deferred-follow-ups / Docs updated) as the future
|
||||
per-issue history record. No second manual ledger.
|
||||
|
||||
### Regression check (#520 "cannot regress" box)
|
||||
|
||||
`scripts/check-kickoff-guard.sh` (shares the lifecycle CI job): fails if active kickoff/README/
|
||||
CLAUDE text reintroduces "#237 is the queue / source of truth / read #237 for current state"
|
||||
patterns. Allow-list the explicitly-archival mentions.
|
||||
|
||||
## Migration plan (#521 — full, via subagents)
|
||||
|
||||
1. Land schema rules + validator + catalog generator + archive skeleton first (validator tolerant of
|
||||
`legacy-unmigrated`).
|
||||
2. Inventory legacy entries by stable subject (not just chronology); assign keys.
|
||||
3. Reconcile known prose-only reversals/retirements first (#390→#406 supersession; #411
|
||||
obsolescence) — orchestrator/escalated, not a cheap subagent.
|
||||
4. Fan out mechanical reformatting in bounded topic batches (each subagent: exact schema + its
|
||||
entries + its own branch). Merge back.
|
||||
5. Move superseded/retired records to `docs/decisions/archive/` with links.
|
||||
6. Regenerate the catalog; flip aggregate budget check on; drive `legacy-unmigrated` to zero.
|
||||
7. Preserve an auditable mapping from every legacy heading to its active record or archive location
|
||||
(a migration map committed alongside).
|
||||
|
||||
Safety valve: if mechanical migration balloons, land complete machinery + a partial migration with a
|
||||
visible remainder count + a tracked follow-up issue (issue done-when explicitly allows a bounded
|
||||
remainder with no ambiguous active rules).
|
||||
|
||||
## Retrieval evaluation (#521)
|
||||
|
||||
A bounded question bank (committed, e.g. `docs/decisions/retrieval-eval.md` or a test fixture)
|
||||
covering: paraphrased task→decision discovery; exact code/path lookup; active-vs-superseded; retired
|
||||
features; rationale/rejected-alternatives; and at least one "convention already implemented"
|
||||
question that must prevent reimplementation. Success = correct **active-record selection + citation**,
|
||||
not merely a semantically-related passage. Executed as the cold-agent step below (deterministic
|
||||
grading where the answer is a specific key/citation).
|
||||
|
||||
## Verification
|
||||
|
||||
- **Deterministic:** validator + regression check + catalog freshness all green locally before push.
|
||||
- **Behavioral (cold-agent sim, #520 done-when):** a fresh subagent handed only the new
|
||||
kickoff/README produces the startup flow — orientation ‖ selector, resolve flags, recheck+claim
|
||||
winner, focused retrieval — and **never reads #237**. Second cold agent runs a sample of the
|
||||
retrieval question bank against the new active view.
|
||||
- **Independent adversarial review** of the whole diff (mandatory: >150 lines, touches CI) before
|
||||
push; cross-model if available, else a cold review-only agent.
|
||||
- Docs updated in the same PR (README index, decisions header, ci-cd release ritual, CLAUDE.md).
|
||||
|
||||
## CI / release-ritual changes
|
||||
|
||||
- Rename/extend the `decisions-guard` job → run `decisions-validate.py` + `check-kickoff-guard.sh` +
|
||||
catalog freshness.
|
||||
- Release ritual (`docs/ci-cd.md` → Versioning & releases): change "discover superseded entries" →
|
||||
"validate lifecycle metadata + reciprocal links; archive already-classified history; refresh the
|
||||
active catalog; enforce the aggregate budget; report unresolved legacy records."
|
||||
- Remove `[decisions-edit]` from the Husky `commit-msg` hook and the CI job; update docs/memory that
|
||||
reference the token.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Conflict surface:** `decisions.md` conflicts on nearly every CI cycle here; this arc rewrites it
|
||||
wholesale. Mitigation: single PR, rebase-then-`merge_when_checks_succeed`, land fast.
|
||||
- **Regenerated-artifact merges:** on rebase conflicts in the generated catalog, regenerate (never
|
||||
hand-merge), same discipline as `v1.json`.
|
||||
- **Memory/lore references** to `[decisions-edit]` and "#237 is the queue" are spread across
|
||||
MEMORY.md and the handoff lore; sweep by subject.
|
||||
Reference in New Issue
Block a user