timothyandClaude Opus 5
9d2b30dc3b
Two defects in scripts/decisions_validate.py, fixed together because they share the validator and its pytest suite. #674 — the validator reported OK on frontmatter PyYAML rejects. The hand-rolled reader is deliberately dependency-free (decisions-guard and the Husky hooks install nothing), so it cannot see a bare apostrophe closing a single-quoted scalar. Hit twice in one session by two independent agents. `pyyaml_frontmatter_faults()` now cross-checks the parse against PyYAML whenever PyYAML is importable, and is SKIPPED with a ::notice:: when it is not — the read path stays dependency-free. The two known hazards fail differently and the fix covers both: the apostrophe makes PyYAML reject the document, while an unquoted ` #` parses fine and silently TRUNCATES the value. So the check compares parsed results key by key rather than try/except-ing the load, which is also what makes it generalize past the two known characters. PyYAML wrote these files, so on disagreement it is authoritative and the file is the defect. The comparison has one implementation, called by the validator and by the existing test_decisions_lib agreement test, so the tool and the suite cannot drift. #688 — test_real_corpus_ceiling_sits_at_the_TAIL_BOUNDARY asserted p90 <= 60 <= p95 in the BLOCKING script-tests job. p90 sat exactly on the ceiling and the distribution above it is sparse, so one ordinary record moved p90 by twenty lines and reddened CI for whoever wrote it; it reproduced twice live (#672, #706) and both times the only in-scope remedy was trimming the new record to fit the constant. v5 splits the claim by robustness instead of hunting for a better single assertion. The blocking test now asserts only the coarse, non-ratcheting property (the ceiling flags a nonempty proper minority, 0 < fraction_over < 1/3); the fine tail-boundary claim is measured every run and REPORTED as a ::notice::, on the same reasoning stale_records already uses — a constant going out of date is the passage of corpus growth, not a defect in the commit under test. The fine property is still asserted, against synthetic distributions the test owns. The ceiling stays 60. Verification: 424 scripts/tests pass; ruff at baseline parity (47 before and after); the cross-check is clean on all 183 real records; a positive control pins that record_wing_faults alone still reports both hazard files as clean, so the new red cannot pass for the wrong reason; and a test demonstrates that appending #672's 62-line and #687's 107-line records to the real corpus does not red the blocking property. Docs: new record docs.frontmatter-pyyaml-crosscheck, docs.corpus-size-signal updated for the v5 split, catalog regenerated, docs/ci-cd.md updated for both. fixes #674 fixes #688 Decisions-Edit: yes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
91 lines
3.5 KiB
Python
91 lines
3.5 KiB
Python
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
import scripts.decisions_lib as dl
|
|
|
|
FIX = Path(__file__).parent / "fixtures" / "sample_decisions.md"
|
|
|
|
|
|
def test_parses_migrated_record():
|
|
recs = dl.parse_file(FIX)
|
|
migrated = [r for r in recs if r.key == "ci.runner-placement"]
|
|
assert len(migrated) == 1
|
|
r = migrated[0]
|
|
assert r.status == "active"
|
|
assert r.since == "2026-07-17"
|
|
assert r.supersedes == "none"
|
|
assert r.superseded_by == "none"
|
|
assert r.rule == "Every CI services container gets an explicit cap."
|
|
assert r.signals is not None
|
|
assert "issues: #390 #406" in r.signals
|
|
assert r.mechanics is not None
|
|
assert r.mechanics.startswith("docs/ci-cd.md")
|
|
|
|
|
|
def test_legacy_record_is_unmigrated():
|
|
recs = dl.parse_file(FIX)
|
|
legacy = [r for r in recs if r.heading.endswith("(#231)")]
|
|
assert len(legacy) == 1
|
|
assert legacy[0].key is None
|
|
assert legacy[0].status == "legacy-unmigrated"
|
|
|
|
|
|
def test_index_section_parses_as_heading():
|
|
recs = dl.parse_file(FIX)
|
|
assert any(r.heading == "Index" for r in recs)
|
|
|
|
|
|
def test_parses_optional_stale_after_and_sources():
|
|
recs = dl.parse_file(FIX)
|
|
r = next(r for r in recs if r.key == "ci.peak-anon-measurement")
|
|
assert r.stale_after == "2027-01-15"
|
|
assert r.sources is not None
|
|
assert "gitea run 4471" in r.sources
|
|
|
|
|
|
def test_optional_fields_default_to_none_when_absent():
|
|
recs = dl.parse_file(FIX)
|
|
r = next(r for r in recs if r.key == "ci.runner-placement")
|
|
assert r.stale_after is None
|
|
assert r.sources is None
|
|
|
|
|
|
def test_empty_stale_after_parses_as_empty_string_not_none():
|
|
"""Absent vs present-but-empty must stay distinguishable for the validator."""
|
|
text = (
|
|
"## H\n"
|
|
"`key: a.b` · `status: active` · `since: 2026-01-01` · `stale-after:` "
|
|
"· `supersedes: none` · `superseded-by: none`\n"
|
|
"**Rule:** r\n"
|
|
)
|
|
r = dl.parse_text(text, Path("fake.md"))[0]
|
|
assert r.stale_after == ""
|
|
|
|
|
|
def test_frontmatter_reader_matches_pyyaml_on_every_real_record():
|
|
"""The dependency-free reader must agree with PyYAML on the whole real corpus.
|
|
|
|
The read path cannot import PyYAML — it runs in CI's `decisions lifecycle` job, the Husky
|
|
pre-commit hook, and on every contributor's machine, none of which install it. (Requiring it
|
|
made the validator crash with ModuleNotFoundError once the corpus was migrated.) A hand parser
|
|
is only safe if it provably matches the library that WROTE the files, so this compares the two
|
|
across every record rather than on a sample.
|
|
|
|
Since #674 the comparison itself lives in `decisions_validate.pyyaml_frontmatter_faults`, which
|
|
the VALIDATOR now runs too — before that it existed only here, so `decisions_validate.py`
|
|
happily reported OK on a record PyYAML rejects. This test delegates to that one implementation
|
|
rather than keeping a second copy of the comparison, so the suite and the validator cannot
|
|
drift apart and agree on what "matches PyYAML" means.
|
|
"""
|
|
pytest.importorskip("yaml")
|
|
import scripts.decisions_validate as dv
|
|
|
|
files = [p for p in dl.RECORDS_DIR.rglob("*.md")] + [p for p in dl.ARCHIVE_DIR.rglob("*.md")]
|
|
files = [f for f in files if dl.has_frontmatter(f.read_text(encoding="utf-8"))]
|
|
assert len(files) > 100, f"only {len(files)} frontmatter files found — test would be near-vacuous"
|
|
|
|
faults, ran = dv.pyyaml_frontmatter_faults(files)
|
|
assert ran, "PyYAML is importable here, so the comparison must have actually run"
|
|
assert not faults, f"{len(faults)} field(s) differ from PyYAML:\n" + "\n".join(faults[:5])
|