decisions_validate.py stays green on frontmatter PyYAML rejects — an apostrophe in a single-quoted scalar (hit twice in one session) #674

Closed
opened 2026-07-27 01:12:45 +02:00 by timothy · 2 comments
Owner

Hit twice in one session, independently, by two different agents working unrelated branches (#578 and #651). That frequency is the reason to file it: it is not an exotic edge case, it is what happens when anyone writes ordinary English prose into a rule: field.

The gap

docs/decisions/records/** frontmatter values are single-quoted YAML scalars. A bare apostrophe inside one — the old type's space, SQLite's LOWER() — terminates the scalar early. PyYAML rejects the file. scripts/decisions_validate.py reports OK.

The validator uses a hand-rolled frontmatter parser, so it cannot see the defect. Only scripts/tests (which parses with PyYAML) catches it.

This is the same class as the already-documented unquoted- # trap, with a different character. The existing unquoted-hash-in-decisions-frontmatter knowledge says the validator "CAN'T see it — run scripts/tests". That is accurate but it is a workaround for a tool that reports success on input it should reject.

Why the hand parser exists — do not just import PyYAML

The decisions read path is deliberately dependency-free: decisions-guard and the git hooks install nothing, and reading has to work everywhere while writing runs in one place. Swapping the parser for PyYAML would break that contract. Any fix has to preserve it.

Plausible shapes:

  • Teach the hand parser to detect the specific structural hazards (a lone apostrophe inside a single-quoted scalar, an unquoted #) and fail loudly, without becoming a general YAML parser.
  • Or have the validator assert its own parse matches PyYAML's when PyYAML happens to be importable, and stay silent when it isn't — so a developer machine catches it while the dependency-free path still runs.

The second is attractive because it generalizes: it catches the next character class too, rather than enumerating hazards one at a time — which is precisely the mistake this session kept making elsewhere.

Why CI already mostly covers it

script-tests runs scripts/tests on every PR, so a broken record does not reach main. The exposure is local: the validator is the tool a developer or agent reaches for directly, and it says OK.

Done-when

  • decisions_validate.py fails on a record whose frontmatter PyYAML rejects — including the apostrophe case and the existing unquoted- # case
  • The read path stays dependency-free: no hard PyYAML import on any path decisions-guard or the git hooks execute
  • The fix generalizes beyond the two known characters, or the decision to enumerate instead is recorded with its reasoning
  • A test in scripts/tests pins both known hazards and asserts the validator itself now rejects them (today it would pass a file the suite rejects — the two disagree)
  • The unquoted-hash-in-decisions-frontmatter knowledge is updated to reflect the resolved state
  • Adversarial review passed

Context: surfaced during #578 and #651.

Hit **twice in one session, independently, by two different agents** working unrelated branches (#578 and #651). That frequency is the reason to file it: it is not an exotic edge case, it is what happens when anyone writes ordinary English prose into a `rule:` field. ## The gap `docs/decisions/records/**` frontmatter values are single-quoted YAML scalars. A bare apostrophe inside one — `the old type's space`, `SQLite's LOWER()` — terminates the scalar early. **PyYAML rejects the file. `scripts/decisions_validate.py` reports OK.** The validator uses a hand-rolled frontmatter parser, so it cannot see the defect. Only `scripts/tests` (which parses with PyYAML) catches it. This is the same class as the already-documented unquoted-` #` trap, with a different character. The existing `unquoted-hash-in-decisions-frontmatter` knowledge says the validator "CAN'T see it — run `scripts/tests`". That is accurate but it is a workaround for a tool that reports success on input it should reject. ## Why the hand parser exists — do not just import PyYAML The decisions **read** path is deliberately dependency-free: `decisions-guard` and the git hooks install nothing, and reading has to work everywhere while writing runs in one place. Swapping the parser for PyYAML would break that contract. Any fix has to preserve it. Plausible shapes: - Teach the hand parser to *detect* the specific structural hazards (a lone apostrophe inside a single-quoted scalar, an unquoted ` #`) and fail loudly, without becoming a general YAML parser. - Or have the validator assert its own parse matches PyYAML's **when PyYAML happens to be importable**, and stay silent when it isn't — so a developer machine catches it while the dependency-free path still runs. The second is attractive because it generalizes: it catches the *next* character class too, rather than enumerating hazards one at a time — which is precisely the mistake this session kept making elsewhere. ## Why CI already mostly covers it `script-tests` runs `scripts/tests` on every PR, so a broken record does not reach `main`. The exposure is local: the validator is the tool a developer or agent reaches for directly, and it says OK. ## Done-when - [x] `decisions_validate.py` fails on a record whose frontmatter PyYAML rejects — including the apostrophe case and the existing unquoted-` #` case - [x] The read path stays dependency-free: no hard PyYAML import on any path `decisions-guard` or the git hooks execute - [x] The fix generalizes beyond the two known characters, or the decision to enumerate instead is recorded with its reasoning - [x] A test in `scripts/tests` pins both known hazards **and** asserts the validator itself now rejects them (today it would pass a file the suite rejects — the two disagree) - [x] The `unquoted-hash-in-decisions-frontmatter` knowledge is updated to reflect the resolved state - [x] Adversarial review passed Context: surfaced during #578 and #651.
timothy added the ci-cdpriority: medium labels 2026-07-27 01:12:45 +02:00
timothy added the in-progress label 2026-08-04 20:00:10 +02:00
Author
Owner

Claiming this for a session working #674 + #688 together as a ci-cd bundle — both are defects in scripts/decisions_validate.py / its pytest suite, so they share a worktree and land as one reviewed change.

Pre-claim checks run and clear: no open PR references either number, no remote branch names them, no prior claiming comment (#688's 2026-07-28 comment explicitly states "No claim on this issue"), and origin/main re-fetched.

Claiming this for a session working #674 + #688 together as a `ci-cd` bundle — both are defects in `scripts/decisions_validate.py` / its pytest suite, so they share a worktree and land as one reviewed change. Pre-claim checks run and clear: no open PR references either number, no remote branch names them, no prior claiming comment (#688's 2026-07-28 comment explicitly states "No claim on this issue"), and `origin/main` re-fetched.
Author
Owner

Closing record

Outcome: Shipped in PR #725 (merged, efc34a3). decisions_validate.py now runs pyyaml_frontmatter_faults() over every record-wing file whenever PyYAML is importable, comparing its parse against the dependency-free reader and failing on either a PyYAML rejection or a value mismatch. Absent PyYAML the check is SKIPPED with a ::notice::, never silently — the read path stays dependency-free.

Root cause: dl._read_frontmatter is a hand parser written for machine-generated single-line scalars. It has no concept of quoting rules, so it accepted input the writer's own library rejects. Nothing compared the two outside the test suite, so the tool and its own tests could disagree about the same file and the tool reported OK.

Decisions/conventions changed: added docs.frontmatter-pyyaml-crosscheck. docs.corpus-size-signal also updated (see #688).

Reusable knowledge: the two known hazards fail DIFFERENTLY and a try/except catches only one — a bare apostrophe in a single-quoted scalar makes PyYAML reject the document, while an unquoted # parses fine and silently TRUNCATES the value (rule: use --flag #2 -> use --flag). Comparing parsed results rather than catching exceptions is what generalises past the two known characters. Also: PyYAML raises a bare ValueError (not YAMLError) on a well-shaped but impossible date such as stale-after: 2026-06-31, and returns TYPED mapping keys, so sorted(set | set) needs key=str.

Honest scope: in CI this changes little. decisions-guard installs no PyYAML so the cross-check always skips there, and script-tests already went red on both hazards. Branch protection requires exactly Build & test (.NET), EF migration integrity and review-verdict/h10 — neither script-tests nor decisions-guard. The real fix is the LOCAL loop, where the validator is the tool an agent reaches for and it printed OK on a corrupt file, plus the tool/suite disagreement now being impossible (one implementation, two callers).

Verification: 432 scripts/tests pass; CI 10/10 green on efc34a3; ruff at baseline parity (47); cross-check clean on all 195 wing files (190 with frontmatter) — no false positives. A positive control pins that record_wing_faults ALONE still reports both hazard files as clean, so the new red cannot pass for the wrong reason. Mutation-verified: stubbing the cross-check, feeding it [], dropping key=str, narrowing the except, or unwiring it from main() each fail a test.

Deferred: none.

Docs updated: docs/decisions/records/docs/frontmatter-pyyaml-crosscheck.md (new), docs/ci-cd.md, docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Shipped in PR #725 (merged, `efc34a3`). `decisions_validate.py` now runs `pyyaml_frontmatter_faults()` over every record-wing file whenever PyYAML is importable, comparing its parse against the dependency-free reader and failing on either a PyYAML rejection or a value mismatch. Absent PyYAML the check is SKIPPED with a `::notice::`, never silently — the read path stays dependency-free. **Root cause:** `dl._read_frontmatter` is a hand parser written for machine-generated single-line scalars. It has no concept of quoting rules, so it accepted input the writer's own library rejects. Nothing compared the two outside the test suite, so the tool and its own tests could disagree about the same file and the tool reported OK. **Decisions/conventions changed:** added `docs.frontmatter-pyyaml-crosscheck`. `docs.corpus-size-signal` also updated (see #688). **Reusable knowledge:** the two known hazards fail DIFFERENTLY and a try/except catches only one — a bare apostrophe in a single-quoted scalar makes PyYAML *reject* the document, while an unquoted ` #` parses fine and silently TRUNCATES the value (`rule: use --flag #2` -> `use --flag`). Comparing parsed *results* rather than catching exceptions is what generalises past the two known characters. Also: PyYAML raises a bare `ValueError` (not `YAMLError`) on a well-shaped but impossible date such as `stale-after: 2026-06-31`, and returns TYPED mapping keys, so `sorted(set | set)` needs `key=str`. **Honest scope:** in CI this changes little. `decisions-guard` installs no PyYAML so the cross-check always skips there, and `script-tests` already went red on both hazards. Branch protection requires exactly `Build & test (.NET)`, `EF migration integrity` and `review-verdict/h10` — neither `script-tests` nor `decisions-guard`. The real fix is the LOCAL loop, where the validator is the tool an agent reaches for and it printed OK on a corrupt file, plus the tool/suite disagreement now being impossible (one implementation, two callers). **Verification:** 432 `scripts/tests` pass; CI 10/10 green on `efc34a3`; ruff at baseline parity (47); cross-check clean on all 195 wing files (190 with frontmatter) — no false positives. A positive control pins that `record_wing_faults` ALONE still reports both hazard files as clean, so the new red cannot pass for the wrong reason. Mutation-verified: stubbing the cross-check, feeding it `[]`, dropping `key=str`, narrowing the except, or unwiring it from `main()` each fail a test. **Deferred:** none. **Docs updated:** `docs/decisions/records/docs/frontmatter-pyyaml-crosscheck.md` (new), `docs/ci-cd.md`, `docs/decisions/README.md` (regenerated).
timothy removed the in-progress label 2026-08-04 23:45:26 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#674