decisions-validate: [decisions-edit] is a bare substring match — a commit that merely *describes* the token disarms the guard #609

Closed
opened 2026-07-25 15:31:22 +02:00 by timothy · 2 comments
Owner

The defect

scripts/decisions_validate.py (~L264) arms the edit token with a bare substring test over the whole commit range:

if EDIT_TOKEN.lower() in (_run(["git", "log", "--format=%B", f"{mb}..{head}"]) or "").lower():

So any commit message containing the literal string anywhere — including prose about the mechanism — sets token = True and suppresses the entire if not token: block: all three rationale-rewrite comparisons (active survivors, active→archive laundering, archive survivors). removed and demoted still run; only rewritten is silenced.

How it was found

Live, in PR #605. A commit message explaining why no token was needed contained the bracketed token, which armed it. The result was a green --base/--head run that was vacuous on the body-diff dimension — in the one PR that hand-resolved a merge conflict inside the corpus that guard exists to police. The safety net was off during exactly the operation it protects against.

No content was actually damaged (verified by re-running the guard with token forced False → clean, and afterwards by a positive control confirming the guard fires on a real prose edit). The bug is that the gate can no-op while reporting success.

This is the "a gate can merge green yet be a no-op" class — a CI-behavior change has to be verified by measuring the effect, not by observing a green check.

Suggested fix

Require the token with some discipline rather than as a free-floating substring. Options, roughly in order of preference:

  1. Trailer form — Decisions-Edit: yes as a git trailer, parsed with git log --format=%(trailers:key=Decisions-Edit). Unambiguous, conventional, and impossible to trip in prose.
  2. Own-line match — require the token alone on a line (^\s*\[decisions-edit\]\s*$ per commit message line).
  3. At minimum, a word-boundary regex plus a check that it isn't inside a quoted/indented block.

Whichever is chosen, keep the fail-open posture on git errors, and add a regression test that a message mentioning the token in prose does not arm it — plus a positive control that the guard genuinely fires on a rationale-prose edit when the token is absent.

Related

  • Found during #603 / PR #605 (that PR works around it by avoiding the literal string; it does not fix the matcher).
  • Same family as the stale-after findings in #603: an optional/soft check that silently never fires is worse than no check, because it reads as coverage.

Done-when

  • Token matching is disciplined (trailer or own-line), not a bare substring — option 1, the git trailer, chosen over own-line because this repo's docs commits routinely quote example commit messages (this fix's own commits do), and an own-line matcher arms on the example
  • Negative test: a commit message describing the token in prose does NOT arm it — plus three more negative controls found in review (quoted mid-body trailer, folded no+ yes, forge-composed merge body), all mutation-verified red against the unfixed matcher
  • Positive test: the guard still fires on a rationale-prose edit with no token present
  • Verified by measuring the guard's effect, not by a green check alone — measured on this PR's own range: with the trailer → OK; with the marker forced off → it flags the record whose rationale this PR edits
  • docs/decisions.md header text updated if the token's spelling/form changes — header, new record ci.decisions-edit-trailer, regenerated catalog, ci-cd.md, pr-checks.yml, hook shim
  • Adversarial review passed — cross-family cold review, 4 rounds, Review-verdict: MERGEABLE @ f49556b6

PR: #618

## The defect `scripts/decisions_validate.py` (~L264) arms the edit token with a bare substring test over the whole commit range: ```python if EDIT_TOKEN.lower() in (_run(["git", "log", "--format=%B", f"{mb}..{head}"]) or "").lower(): ``` So **any** commit message containing the literal string anywhere — including prose *about* the mechanism — sets `token = True` and suppresses the entire `if not token:` block: all three rationale-rewrite comparisons (active survivors, active→archive laundering, archive survivors). `removed` and `demoted` still run; only `rewritten` is silenced. ## How it was found Live, in PR #605. A commit message explaining *why no token was needed* contained the bracketed token, which armed it. The result was a **green `--base/--head` run that was vacuous on the body-diff dimension** — in the one PR that hand-resolved a merge conflict inside the corpus that guard exists to police. The safety net was off during exactly the operation it protects against. No content was actually damaged (verified by re-running the guard with `token` forced `False` → clean, and afterwards by a positive control confirming the guard fires on a real prose edit). The bug is that the gate can no-op while reporting success. This is the "a gate can merge green yet be a no-op" class — a CI-behavior change has to be verified by *measuring the effect*, not by observing a green check. ## Suggested fix Require the token with some discipline rather than as a free-floating substring. Options, roughly in order of preference: 1. **Trailer form** — `Decisions-Edit: yes` as a git trailer, parsed with `git log --format=%(trailers:key=Decisions-Edit)`. Unambiguous, conventional, and impossible to trip in prose. 2. **Own-line match** — require the token alone on a line (`^\s*\[decisions-edit\]\s*$` per commit message line). 3. At minimum, a word-boundary regex plus a check that it isn't inside a quoted/indented block. Whichever is chosen, keep the fail-open posture on git errors, and add a regression test that a message *mentioning* the token in prose does **not** arm it — plus a positive control that the guard genuinely fires on a rationale-prose edit when the token is absent. ## Related - Found during #603 / PR #605 (that PR works around it by avoiding the literal string; it does not fix the matcher). - Same family as the `stale-after` findings in #603: an optional/soft check that silently never fires is worse than no check, because it reads as coverage. ## Done-when - [x] Token matching is disciplined (trailer or own-line), not a bare substring — **option 1, the git trailer**, chosen over own-line because this repo's docs commits routinely quote example commit messages (this fix's own commits do), and an own-line matcher arms on the example - [x] Negative test: a commit message describing the token in prose does NOT arm it — plus three more negative controls found in review (quoted mid-body trailer, folded `no`+` yes`, forge-composed merge body), all mutation-verified red against the unfixed matcher - [x] Positive test: the guard still fires on a rationale-prose edit with no token present - [x] Verified by measuring the guard's effect, not by a green check alone — measured on this PR's own range: with the trailer → OK; with the marker forced off → it flags the record whose rationale this PR edits - [x] `docs/decisions.md` header text updated if the token's spelling/form changes — header, new record `ci.decisions-edit-trailer`, regenerated catalog, `ci-cd.md`, `pr-checks.yml`, hook shim - [x] Adversarial review passed — cross-family cold review, 4 rounds, `Review-verdict: MERGEABLE @ f49556b6` PR: #618
timothy added the bugci-cdpriority: high labels 2026-07-25 17:16:52 +02:00
timothy added the in-progress label 2026-07-25 18:15:14 +02:00
Author
Owner

Claiming this — working in fix/609-decisions-edit-token off origin/main.

Sequencing note: #610 rewrites decisions_validate.py / decisions_lib.py wholesale, so this lands first and #610 carries the disciplined matcher forward rather than the two colliding.

Claiming this — working in `fix/609-decisions-edit-token` off `origin/main`. Sequencing note: #610 rewrites `decisions_validate.py` / `decisions_lib.py` wholesale, so this lands first and #610 carries the disciplined matcher forward rather than the two colliding.
Author
Owner

Closing record

Outcome: Shipped in #618 (merged to main). The body-diff exemption is now an affirmative Decisions-Edit: git trailer (yes/true/1, case-insensitive), read with --no-merges and unfold, replacing the bare substring test over git log --format=%B. The retired [decisions-edit] substring arms nothing and gets a ::warning:: nudge so the old habit fails loudly rather than confusingly.

Root cause: The marker was tested as a substring anywhere in the commit range, so the guard could be disarmed by being described. Any commit message containing the literal string — including one explaining why no marker was needed — set token = True and suppressed all three rationale comparisons (active survivors, active→archive laundering, archive survivors) while the job still reported success. The deeper cause is that the exemption was expressed in a medium with no grammar: free commit-message text, where a mention and an instruction are indistinguishable. A git trailer has a grammar, so the two are structurally different things.

Decisions/conventions changed: Added ci.decisions-edit-trailer (active). docs.decision-lifecycle prose amended to point at it — the marker's scope is unchanged (still only rationale-prose edits), only its form.

Reusable knowledge:

  • Git recognises a trailer only in the final paragraph of a message. A Decisions-Edit: yes line quoted mid-body followed by more prose parses as nothing — which is why the trailer, not an own-line match, is the right fix: this repo's docs commits routinely quote example commit messages, and an own-line matcher arms on the example. Verified against git 2.55.
  • %(trailers:key=X,valueonly) splits folded values into separate lines, so Decisions-Edit: no + continuation yes lets the continuation arm the exemption on its own — inverting the author's explicit refusal. Always pass unfold.
  • prepush-rebase-check.sh does not forbid merging main into a PR branch. It refuses a branch that is behind origin/main, and a merge makes origin/main an ancestor, so the push is allowed. Merging main in is discouraged by convention only. (I asserted the opposite in a code comment and review caught it.)
  • A CI monitor that breaks on a guessed context count reports a confident terminal verdict while having stopped watching. Gate on the combined status endpoint and enumerate non-green contexts before exiting — this bit during this very session.

Verification: 59 python tests. Every negative control mutation-verified red against the unfixed matcher (and specifically against the matcher without unfold / without --no-merges) — a green suite that cannot fail proves nothing. Guard measured on the PR's own range: with the trailer → OK; with the marker forced off → it flags the docs.decision-lifecycle record whose rationale the PR edits, i.e. the body-diff dimension is genuinely armed here, the inverse of the vacuous #605 run. CI on f49556b6: 8 green, 1 skipped (image push, PR-irrelevant), combined success — including decisions lifecycle, the job whose behavior changed. ruff/format clean on touched files. No .cs touched → BOM/format gate n/a; no write-path handler → live-E2E not required.

Deferred: None blocking. Two residuals are documented in the record rather than fixed, both deliberate: (a) a quoted example that is the final paragraph of a non-merge commit is a trailer by git's grammar and will arm the guard — inherent to any trailer scheme; (b) --no-merges costs a narrow false negative if an author marks ONLY a conflict-resolving merge commit — a loud failure costing one extra commit, preferred over the silent guard-disabling the alternative would reintroduce.

Docs updated: docs/decisions.md (header contract + new ci.decisions-edit-trailer record + docs.decision-lifecycle prose), docs/decisions/README.md (regenerated catalog), docs/ci-cd.md (3 mentions), .gitea/workflows/pr-checks.yml, .claude/hooks/decisions-guard.sh.

Note for #610: that issue rewrites decisions_validate.py / decisions_lib.py wholesale. It must carry the disciplined matcher forward — re-deriving a substring test would silently reintroduce this bug.

## Closing record **Outcome:** Shipped in #618 (merged to `main`). The body-diff exemption is now an affirmative `Decisions-Edit:` git trailer (`yes`/`true`/`1`, case-insensitive), read with `--no-merges` and `unfold`, replacing the bare substring test over `git log --format=%B`. The retired `[decisions-edit]` substring arms nothing and gets a `::warning::` nudge so the old habit fails loudly rather than confusingly. **Root cause:** The marker was tested as a substring anywhere in the commit range, so the guard could be disarmed *by being described*. Any commit message containing the literal string — including one explaining why no marker was needed — set `token = True` and suppressed all three rationale comparisons (active survivors, active→archive laundering, archive survivors) while the job still reported success. The deeper cause is that the exemption was expressed in a medium with no grammar: free commit-message text, where a mention and an instruction are indistinguishable. A git trailer has a grammar, so the two are structurally different things. **Decisions/conventions changed:** Added `ci.decisions-edit-trailer` (active). `docs.decision-lifecycle` prose amended to point at it — the marker's *scope* is unchanged (still only rationale-prose edits), only its *form*. **Reusable knowledge:** - Git recognises a trailer **only in the final paragraph** of a message. A `Decisions-Edit: yes` line quoted mid-body followed by more prose parses as nothing — which is why the trailer, not an own-line match, is the right fix: this repo's docs commits routinely quote example commit messages, and an own-line matcher arms on the example. Verified against git 2.55. - `%(trailers:key=X,valueonly)` **splits folded values into separate lines**, so `Decisions-Edit: no` + continuation ` yes` lets the continuation arm the exemption on its own — inverting the author's explicit refusal. Always pass `unfold`. - `prepush-rebase-check.sh` does **not** forbid merging main into a PR branch. It refuses a branch that is *behind* `origin/main`, and a merge makes `origin/main` an ancestor, so the push is allowed. Merging main in is discouraged by convention only. (I asserted the opposite in a code comment and review caught it.) - A CI monitor that breaks on a guessed context count reports a confident terminal verdict while having stopped watching. Gate on the **combined** status endpoint and enumerate non-green contexts before exiting — this bit during this very session. **Verification:** 59 python tests. Every negative control **mutation-verified red** against the unfixed matcher (and specifically against the matcher without `unfold` / without `--no-merges`) — a green suite that cannot fail proves nothing. Guard measured on the PR's own range: with the trailer → `OK`; with the marker forced off → it flags the `docs.decision-lifecycle` record whose rationale the PR edits, i.e. the body-diff dimension is genuinely armed here, the inverse of the vacuous #605 run. CI on `f49556b6`: 8 green, 1 skipped (image push, PR-irrelevant), combined success — including `decisions lifecycle`, the job whose behavior changed. ruff/format clean on touched files. No `.cs` touched → BOM/format gate n/a; no write-path handler → live-E2E not required. **Deferred:** None blocking. Two residuals are **documented in the record rather than fixed**, both deliberate: (a) a quoted example that is the *final paragraph of a non-merge commit* is a trailer by git's grammar and will arm the guard — inherent to any trailer scheme; (b) `--no-merges` costs a narrow false negative if an author marks ONLY a conflict-resolving merge commit — a loud failure costing one extra commit, preferred over the silent guard-disabling the alternative would reintroduce. **Docs updated:** `docs/decisions.md` (header contract + new `ci.decisions-edit-trailer` record + `docs.decision-lifecycle` prose), `docs/decisions/README.md` (regenerated catalog), `docs/ci-cd.md` (3 mentions), `.gitea/workflows/pr-checks.yml`, `.claude/hooks/decisions-guard.sh`. **Note for #610:** that issue rewrites `decisions_validate.py` / `decisions_lib.py` wholesale. It must carry the disciplined matcher forward — re-deriving a substring test would silently reintroduce this bug.
timothy removed the in-progress label 2026-07-25 19:08:28 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#609