Evaluated OKF as a replacement for our decision-record system, rejected it, and adopted two of its optional metadata families additively. Rationale is in the new record (docs.decision-optional-provenance); the short version is that OKF's conformance rules are permissive exactly where ours are strict — broken cross-links, unknown types and missing fields must all be tolerated, deprecated points at no successor, and its stable identity is the file path, which the breadcrumb rule tells agents not to trust.
Three commits: the feature, the fixes from an adversarial review, and the backfill.
What changed
decisions_lib.py
parse stale-after off the metadata line, **Sources:** off the metadata block
decisions_validate.py
malformed stale-after → blocking error; past-due active record → non-blocking::notice:: via stale_records()
build_decisions_catalog.py
new Review due section, sorted soonest-first, rendering the date only
header schema, the new active record, and the backfill onto three ci.* records
docs/README.md
schema line kept accurate
Neither field is in REQUIRED_META. No format migration, no change to key/status/since/supersedes/superseded-by/Signals:.
The two design constraints that drove it
Both exist to avoid a self-inflicted CI outage:
Past-due is never blocking. A record going stale is the passage of time, not a defect in whatever commit happens to be under test. A blocking check would turn an unrelated PR red on a calendar boundary.
The generated catalog is clock-free. It's checked with --check in CI, so anything time-dependent in it would silently drift red with no commit touching the corpus. Evaluating today >= stale-after lives only on the validator's notice path; the catalog renders the date.
A malformed date is blocking, because it would otherwise silently never fire.
Adversarial review found four defects — all fixed in f72bcc9a
Two of them broke the very invariant the feature claims to enforce. Worth reading:
date.fromisoformat is not a YYYY-MM-DD validator. On Python ≥3.11 it also accepts ISO basic format (20270101) and week dates (2027-W01-1), so malformed-looking values passed the blocking check — and which forms parse depends on the interpreter, meaning the same corpus could validate differently on a dev machine than on the runner (pr-checks.yml pins only python-version: '3.x'). Knock-on: the Review-due section sorts on the raw string, so an accepted 20270101 sorted after2027-01-15 ('-' < '0'), contradicting the section's own "soonest-first" text. Fixed with a ^\d{4}-\d{2}-\d{2}$ gate, which closes both — a fixed-width zero-padded form is what makes string sort ≡ date sort.
An empty stale-after: passed as "absent".or None in the parser plus a truthiness guard in the validator collapsed "" into None, so a blank or truncated value produced no error and no notice, ever — precisely the silently-never-fires failure the blocking check exists to prevent.
test_catalog_is_date_independent was partly vacuous — with no date in either render, both sides were trivially equal after the .replace(). It did still catch an injected clock-derived marker, but it passed with the feature deleted.
The malformed-date check exempted archive records. Staleness is moot there, but a typo is still a typo.
Each has a regression test. Fixes verified by independently reproducing the reviewer's exploit cases against the patched code, not by re-running the author's own tests.
Backfill (738c0e25)
The mechanism shipped unused, so three records that encode measured host behavior now adopt both fields, staggered so they don't all come due at once:
Record
stale-after
Why it drifts
ci.runner-placement
2027-01-15
Encodes bumblebee's sizing (25 GiB / 12 cores) and container caps. Premise has already partly moved — transcoders left for jazz 2026-07-20, runners retuned since.
ci.infra-shaped-red-under-load
2027-02-15
A triage heuristic calibrated against load 243 on a since-retuned host.
ci.peak-anon-measurement
2027-03-15
Weakest of the three, but does assert memory.peak is cache-inflated — a cgroup fact.
Confirmed empirically that amending three existing records needs no[decisions-edit] token: both fields land inside the contiguous metadata block that _rationale() strips before the body-diff, consistent with Rule/Signals/Mechanics. --base main --head HEAD passes with zero tokens in the branch.
Verification
51 tests passing; ruff, ruff format and pyright clean on the touched files; decisions-validate: OK; catalog up to date. Beyond unit tests, a live probe against the real corpus confirmed past-due → exit 0 + notice, malformed → exit 1, and past-vs-future catalogs byte-identical modulo the date string.
Recording one process note: the first run of that probe reported two failures, both artifacts of the probe's own arrangement — I validated before regenerating the catalog (so the real error was catalog-drift, not staleness), and my "is today baked in?" check matched 2026-07-25, which is legitimately in the file as a record's since: value. A sloppier read would have sent me debugging code that was already correct.
Known / deliberate
Corpus budget: 5695 → ~5750 against a 5600 limit. Already over before this PR; non-blocking warning, deliberately not raised — that's what the warning is for. Consolidation belongs to its own issue.
OKF's verified: {by, at} trust tiers were considered and deferred, not rejected.
fixes #603
Evaluated OKF as a replacement for our decision-record system, **rejected it**, and adopted two of its optional metadata families additively. Rationale is in the new record (`docs.decision-optional-provenance`); the short version is that OKF's conformance rules are permissive exactly where ours are strict — broken cross-links, unknown types and missing fields must all be tolerated, `deprecated` points at no successor, and its stable identity is the file *path*, which the breadcrumb rule tells agents not to trust.
Three commits: the feature, the fixes from an adversarial review, and the backfill.
## What changed
| | |
|---|---|
| `decisions_lib.py` | parse `stale-after` off the metadata line, `**Sources:**` off the metadata block |
| `decisions_validate.py` | malformed `stale-after` → **blocking** error; past-due active record → **non-blocking** `::notice::` via `stale_records()` |
| `build_decisions_catalog.py` | new **Review due** section, sorted soonest-first, rendering the date only |
| `docs/decisions.md`, `docs/decisions/workflow-process.md` | header schema, the new active record, and the backfill onto three `ci.*` records |
| `docs/README.md` | schema line kept accurate |
Neither field is in `REQUIRED_META`. No format migration, no change to `key`/`status`/`since`/`supersedes`/`superseded-by`/`Signals:`.
## The two design constraints that drove it
Both exist to avoid a self-inflicted CI outage:
1. **Past-due is never blocking.** A record going stale is the passage of time, not a defect in whatever commit happens to be under test. A blocking check would turn an unrelated PR red on a calendar boundary.
2. **The generated catalog is clock-free.** It's checked with `--check` in CI, so anything time-dependent in it would silently drift red with *no commit touching the corpus*. Evaluating `today >= stale-after` lives only on the validator's notice path; the catalog renders the date.
A malformed date *is* blocking, because it would otherwise silently never fire.
## Adversarial review found four defects — all fixed in `f72bcc9a`
Two of them broke the very invariant the feature claims to enforce. Worth reading:
1. **`date.fromisoformat` is not a YYYY-MM-DD validator.** On Python ≥3.11 it also accepts ISO *basic* format (`20270101`) and week dates (`2027-W01-1`), so malformed-looking values passed the blocking check — and *which* forms parse depends on the interpreter, meaning the same corpus could validate differently on a dev machine than on the runner (`pr-checks.yml` pins only `python-version: '3.x'`). Knock-on: the Review-due section sorts on the raw **string**, so an accepted `20270101` sorted *after* `2027-01-15` (`'-' < '0'`), contradicting the section's own "soonest-first" text. Fixed with a `^\d{4}-\d{2}-\d{2}$` gate, which closes both — a fixed-width zero-padded form is what makes string sort ≡ date sort.
2. **An empty `stale-after:` passed as "absent".** `or None` in the parser plus a truthiness guard in the validator collapsed `""` into `None`, so a blank or truncated value produced no error and no notice, ever — precisely the silently-never-fires failure the blocking check exists to prevent.
3. **`test_catalog_is_date_independent` was partly vacuous** — with no date in either render, both sides were trivially equal after the `.replace()`. It did still catch an injected clock-derived marker, but it passed with the feature deleted.
4. **The malformed-date check exempted archive records.** Staleness is moot there, but a typo is still a typo.
Each has a regression test. Fixes verified by independently reproducing the reviewer's exploit cases against the patched code, not by re-running the author's own tests.
## Backfill (`738c0e25`)
The mechanism shipped unused, so three records that encode *measured host behavior* now adopt both fields, staggered so they don't all come due at once:
| Record | `stale-after` | Why it drifts |
|---|---|---|
| `ci.runner-placement` | 2027-01-15 | Encodes bumblebee's sizing (25 GiB / 12 cores) and container caps. Premise has already partly moved — transcoders left for jazz 2026-07-20, runners retuned since. |
| `ci.infra-shaped-red-under-load` | 2027-02-15 | A triage heuristic calibrated against load 243 on a since-retuned host. |
| `ci.peak-anon-measurement` | 2027-03-15 | Weakest of the three, but does assert `memory.peak` is cache-inflated — a cgroup fact. |
Confirmed empirically that amending three *existing* records needs **no** `[decisions-edit]` token: both fields land inside the contiguous metadata block that `_rationale()` strips before the body-diff, consistent with `Rule`/`Signals`/`Mechanics`. `--base main --head HEAD` passes with zero tokens in the branch.
## Verification
51 tests passing; ruff, ruff format and pyright clean on the touched files; `decisions-validate: OK`; `catalog up to date`. Beyond unit tests, a live probe against the real corpus confirmed past-due → exit 0 + notice, malformed → exit 1, and past-vs-future catalogs byte-identical modulo the date string.
Recording one process note: the *first* run of that probe reported two failures, both artifacts of the probe's own arrangement — I validated before regenerating the catalog (so the real error was catalog-drift, not staleness), and my "is today baked in?" check matched `2026-07-25`, which is legitimately in the file as a record's `since:` value. A sloppier read would have sent me debugging code that was already correct.
## Known / deliberate
- **Corpus budget**: 5695 → ~5750 against a 5600 limit. Already over before this PR; non-blocking warning, deliberately *not* raised — that's what the warning is for. Consolidation belongs to its own issue.
- OKF's `verified: {by, at}` trust tiers were considered and **deferred**, not rejected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Cold same-model review (Codex CLI is not installed in this environment, so no cross-family pass; stating the exemption explicitly).
Pass 1 (@ 1d3d2185) found four defects, two of which broke the invariant the feature claims to enforce:
date.fromisoformat is not a YYYY-MM-DD validator — accepts 20270101 and 2027-W01-1 on py≥3.11, making the blocking check interpreter-dependent, and (since the catalog sorts on the raw string) sorting 20270101after2027-01-15.
An empty stale-after: collapsed to "absent" — the silently-never-fires failure the check exists to prevent.
test_catalog_is_date_independent passed with the feature deleted.
Archive records exempt from the malformed-date check.
All fixed in 75139bbf. Pass 2 verified them airtight — brute-forced all 1,114,112 codepoints against the regex (unicode-digit forms pass the regex, then fail closed through except ValueError), and proved the de-vacuumed test by mutation.
Pass 2 (@ 738c0e25) also caught that the record's own prose claimed "no backfill was done here" while the backfill commit falsified it — fixed free, before it became surviving rationale prose.
Pass 3 (@ dd4f2c96) BLOCKED on a self-inflicted gate failure worth recording: my commit message explaining the edit token contained the bracketed literal, and decisions_validate matches it as a bare substring over the whole commit range. That armed it and suppressed all three rationale-rewrite comparisons — a green --base/--head run that was vacuous on the body-diff dimension, in the one PR that hand-resolved a conflict inside the corpus that guard protects. Amended in f04412a9; hardening filed as #609 with a required negative test.
Final pass (@ f04412a9) confirmed the guard is now live by independent method rather than trusting my positive control: 328 _rationale invocations — exactly 160 active pairs ×2 + 4 archive pairs ×2 + 0, so no comparison was skipped — plus an in-memory poisoning of ci.runner-placement to prove sensitivity, then clean on the real tree. Also confirmed not fail-open (base parsed 160 records, head 161).
Rebased onto origin/main (abeef63f); the append-vs-append conflict in docs/decisions.md was resolved keeping both records, verified as +1 H2 / +1 parsed record / zero records gone, with main's ci.ui-e2e-harness byte-identical to its origin copy.
51 tests, validator OK, catalog --check OK. Remaining budget warning (5846 vs 5600) is pre-existing — base abeef63f already measured 5782.
## Adversarial review — 3 passes
Cold same-model review (Codex CLI is not installed in this environment, so no cross-family pass; stating the exemption explicitly).
**Pass 1** (@ 1d3d2185) found four defects, two of which broke the invariant the feature claims to enforce:
1. `date.fromisoformat` is not a YYYY-MM-DD validator — accepts `20270101` and `2027-W01-1` on py≥3.11, making the blocking check interpreter-dependent, and (since the catalog sorts on the raw string) sorting `20270101` *after* `2027-01-15`.
2. An empty `stale-after:` collapsed to "absent" — the silently-never-fires failure the check exists to prevent.
3. `test_catalog_is_date_independent` passed with the feature deleted.
4. Archive records exempt from the malformed-date check.
All fixed in `75139bbf`. Pass 2 verified them airtight — brute-forced all 1,114,112 codepoints against the regex (unicode-digit forms pass the regex, then fail *closed* through `except ValueError`), and proved the de-vacuumed test by mutation.
**Pass 2** (@ 738c0e25) also caught that the record's own prose claimed "no backfill was done here" while the backfill commit falsified it — fixed free, before it became surviving rationale prose.
**Pass 3** (@ dd4f2c96) BLOCKED on a self-inflicted gate failure worth recording: my commit message *explaining* the edit token contained the bracketed literal, and `decisions_validate` matches it as a **bare substring over the whole commit range**. That armed it and suppressed all three rationale-rewrite comparisons — a green `--base/--head` run that was **vacuous on the body-diff dimension**, in the one PR that hand-resolved a conflict inside the corpus that guard protects. Amended in `f04412a9`; hardening filed as #609 with a required negative test.
**Final pass** (@ f04412a9) confirmed the guard is now live by independent method rather than trusting my positive control: 328 `_rationale` invocations — exactly 160 active pairs ×2 + 4 archive pairs ×2 + 0, so no comparison was skipped — plus an in-memory poisoning of `ci.runner-placement` to prove sensitivity, then clean on the real tree. Also confirmed not fail-open (base parsed 160 records, head 161).
Rebased onto `origin/main` (`abeef63f`); the append-vs-append conflict in `docs/decisions.md` was resolved keeping both records, verified as +1 H2 / +1 parsed record / zero records gone, with main's `ci.ui-e2e-harness` byte-identical to its origin copy.
51 tests, validator OK, catalog `--check` OK. Remaining budget warning (5846 vs 5600) is pre-existing — base `abeef63f` already measured 5782.
Review-verdict: MERGEABLE @ f04412a95b599d2f1ed1dd5c4ece15d5a2aead3a
timothy
scheduled this pull request to auto merge when all checks succeed 2026-07-25 15:33:49 +02:00
timothy
merged commit c3beee1629 into main2026-07-25 16:16:00 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
fixes #603
Evaluated OKF as a replacement for our decision-record system, rejected it, and adopted two of its optional metadata families additively. Rationale is in the new record (
docs.decision-optional-provenance); the short version is that OKF's conformance rules are permissive exactly where ours are strict — broken cross-links, unknown types and missing fields must all be tolerated,deprecatedpoints at no successor, and its stable identity is the file path, which the breadcrumb rule tells agents not to trust.Three commits: the feature, the fixes from an adversarial review, and the backfill.
What changed
decisions_lib.pystale-afteroff the metadata line,**Sources:**off the metadata blockdecisions_validate.pystale-after→ blocking error; past-due active record → non-blocking::notice::viastale_records()build_decisions_catalog.pydocs/decisions.md,docs/decisions/workflow-process.mdci.*recordsdocs/README.mdNeither field is in
REQUIRED_META. No format migration, no change tokey/status/since/supersedes/superseded-by/Signals:.The two design constraints that drove it
Both exist to avoid a self-inflicted CI outage:
--checkin CI, so anything time-dependent in it would silently drift red with no commit touching the corpus. Evaluatingtoday >= stale-afterlives only on the validator's notice path; the catalog renders the date.A malformed date is blocking, because it would otherwise silently never fire.
Adversarial review found four defects — all fixed in
f72bcc9aTwo of them broke the very invariant the feature claims to enforce. Worth reading:
date.fromisoformatis not a YYYY-MM-DD validator. On Python ≥3.11 it also accepts ISO basic format (20270101) and week dates (2027-W01-1), so malformed-looking values passed the blocking check — and which forms parse depends on the interpreter, meaning the same corpus could validate differently on a dev machine than on the runner (pr-checks.ymlpins onlypython-version: '3.x'). Knock-on: the Review-due section sorts on the raw string, so an accepted20270101sorted after2027-01-15('-' < '0'), contradicting the section's own "soonest-first" text. Fixed with a^\d{4}-\d{2}-\d{2}$gate, which closes both — a fixed-width zero-padded form is what makes string sort ≡ date sort.stale-after:passed as "absent".or Nonein the parser plus a truthiness guard in the validator collapsed""intoNone, so a blank or truncated value produced no error and no notice, ever — precisely the silently-never-fires failure the blocking check exists to prevent.test_catalog_is_date_independentwas partly vacuous — with no date in either render, both sides were trivially equal after the.replace(). It did still catch an injected clock-derived marker, but it passed with the feature deleted.Each has a regression test. Fixes verified by independently reproducing the reviewer's exploit cases against the patched code, not by re-running the author's own tests.
Backfill (
738c0e25)The mechanism shipped unused, so three records that encode measured host behavior now adopt both fields, staggered so they don't all come due at once:
stale-afterci.runner-placementci.infra-shaped-red-under-loadci.peak-anon-measurementmemory.peakis cache-inflated — a cgroup fact.Confirmed empirically that amending three existing records needs no
[decisions-edit]token: both fields land inside the contiguous metadata block that_rationale()strips before the body-diff, consistent withRule/Signals/Mechanics.--base main --head HEADpasses with zero tokens in the branch.Verification
51 tests passing; ruff, ruff format and pyright clean on the touched files;
decisions-validate: OK;catalog up to date. Beyond unit tests, a live probe against the real corpus confirmed past-due → exit 0 + notice, malformed → exit 1, and past-vs-future catalogs byte-identical modulo the date string.Recording one process note: the first run of that probe reported two failures, both artifacts of the probe's own arrangement — I validated before regenerating the catalog (so the real error was catalog-drift, not staleness), and my "is today baked in?" check matched
2026-07-25, which is legitimately in the file as a record'ssince:value. A sloppier read would have sent me debugging code that was already correct.Known / deliberate
verified: {by, at}trust tiers were considered and deferred, not rejected.🤖 Generated with Claude Code
738c0e2564todd4f2c96d9dd4f2c96d9tof04412a95bAdversarial review — 3 passes
Cold same-model review (Codex CLI is not installed in this environment, so no cross-family pass; stating the exemption explicitly).
Pass 1 (@ 1d3d2185) found four defects, two of which broke the invariant the feature claims to enforce:
date.fromisoformatis not a YYYY-MM-DD validator — accepts20270101and2027-W01-1on py≥3.11, making the blocking check interpreter-dependent, and (since the catalog sorts on the raw string) sorting20270101after2027-01-15.stale-after:collapsed to "absent" — the silently-never-fires failure the check exists to prevent.test_catalog_is_date_independentpassed with the feature deleted.All fixed in
75139bbf. Pass 2 verified them airtight — brute-forced all 1,114,112 codepoints against the regex (unicode-digit forms pass the regex, then fail closed throughexcept ValueError), and proved the de-vacuumed test by mutation.Pass 2 (@ 738c0e25) also caught that the record's own prose claimed "no backfill was done here" while the backfill commit falsified it — fixed free, before it became surviving rationale prose.
Pass 3 (@ dd4f2c96) BLOCKED on a self-inflicted gate failure worth recording: my commit message explaining the edit token contained the bracketed literal, and
decisions_validatematches it as a bare substring over the whole commit range. That armed it and suppressed all three rationale-rewrite comparisons — a green--base/--headrun that was vacuous on the body-diff dimension, in the one PR that hand-resolved a conflict inside the corpus that guard protects. Amended inf04412a9; hardening filed as #609 with a required negative test.Final pass (@
f04412a9) confirmed the guard is now live by independent method rather than trusting my positive control: 328_rationaleinvocations — exactly 160 active pairs ×2 + 4 archive pairs ×2 + 0, so no comparison was skipped — plus an in-memory poisoning ofci.runner-placementto prove sensitivity, then clean on the real tree. Also confirmed not fail-open (base parsed 160 records, head 161).Rebased onto
origin/main(abeef63f); the append-vs-append conflict indocs/decisions.mdwas resolved keeping both records, verified as +1 H2 / +1 parsed record / zero records gone, with main'sci.ui-e2e-harnessbyte-identical to its origin copy.51 tests, validator OK, catalog
--checkOK. Remaining budget warning (5846 vs 5600) is pre-existing — baseabeef63falready measured 5782.Review-verdict: MERGEABLE @
f04412a95b