From 3bbe4f0635e26d210f02e2ad17dfe69df7d3ff35 Mon Sep 17 00:00:00 2001 From: Timothy Date: Sun, 30 Aug 2026 03:07:38 +0200 Subject: [PATCH] fix(845): "14 of 36" was a grep LINE count, and alignment ADDS a named failure rather than replacing them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seventh cold review. Three findings, all in prose I wrote correcting earlier prose. HIGH — "14 of 36 did" is not a failure count. `grep -c` over pytest output counts LINES, and `--tb=line` prints each failure twice, so 14 was 7 failures doubled; only 5 distinct tests are involved, and with truncation disabled the figure moves the other way (31 of 36). Measured: raw grep -c 14 of which '^E ' rows 7 of which '^/' tb=line rows 7 DISTINCT failing tests 5 A number that changes with pytest's display options is not a fact about the code, so no figure is given at all now — which is what the same comment already said it was doing about counts. The trap itself is recorded so one is not added back. MEDIUM — "ONE named guard failure INSTEAD OF half the suite reddening" asserts a substitution that does not happen, and contradicted the sentence directly above it. Measured on the aligned tree: the guard failure appears AND the ~36 unrelated reds still appear, in the same job. Alignment adds a diagnosis; it replaces nothing. Now "ALONGSIDE". MEDIUM — the `set -u` sweep was incomplete, and the previous commit message's "gone repo-wide" was false. Four files still asserted the refuted mechanism. The sharpest was `scripts/check-review-verdict.sh`, whose comment said "see the sentinel comment in the library" — and following that pointer landed on the retraction saying the opposite. Corrected there, in `docs/ci-cd.md` and in `test_review_verdict_vocabulary.py`, all three already touched by this branch. Every remaining mention in code, tests and docs is now a quotation INSIDE a retraction; the only standing assertion left is the decision record, which is its own change and stays with ersatztv#886 (now narrowed to exactly that). CORRECTION to the previous commit message: it said the retracted wording was "gone repo-wide". That was true only of the two phrases the round-6 reviewer checked, not of the mechanism. It is true now for code, tests and docs. Also: dropped a "for the same reason" back-reference that resolved to the wrong reason (drift, where the actual reason is falsity), and removed the session narrative from the figure notes — "a first attempt said 14" is autobiography; the durable form is the trap and its consequence. Verified again on this tree: full suite 1278 passed / 2 skipped, catalog in sync, decisions-validate OK, doc-narrative clean. fixes #845 refs #886 Decisions-Edit: yes Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019zUmJZHhVP7kXg5DV237TW --- docs/ci-cd.md | 16 +++++++++---- .../records/ci/exemption-provenance.md | 17 +++++++++----- scripts/check-review-verdict.sh | 18 +++++++++++---- scripts/lib/h10-reviewers.sh | 20 +++++++++------- scripts/lib/review-verdict-vocabulary.sh | 7 +++--- scripts/tests/test_pr_changed_files.py | 12 ++++++---- .../tests/test_review_verdict_vocabulary.py | 23 +++++++++++-------- 7 files changed, 71 insertions(+), 42 deletions(-) diff --git a/docs/ci-cd.md b/docs/ci-cd.md index f781a70cf..90ac02714 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -1515,11 +1515,17 @@ nothing. Words are interpolated into a regex, so a `.*` in the positive list wou explicit `BLOCKED` as `positive`; validation restricts every word to `[a-z][a-z-]*`. **What enforces that is a data dependency, not the `if` around the validator** — worth knowing -before editing either script, because the obvious reading is wrong and the first version of #788 -shipped the hole. Under `set -u`, an unbound-variable error inside a function aborts the function -but not the script, and in an `if ! validate` form neither branch runs, so the `exit 2` is skipped. -The reader has no `set -e`, so unvalidated words reached the regex and an explicit `BLOCKED @ head` -classified `positive`, exit 0. Validation now sets a sentinel on its last line and the derived views +before editing either script, because the first version of #788 shipped the hole. The fail-open is +measured: with the negative list written as a scalar and `.*` in the positive list, unvalidated words +reached the regex and an explicit `BLOCKED @ head` classified `positive`, exit 0. + +The MECHANISM first recorded for it was wrong, corrected in ersatztv#845 and stated here so it is not +re-derived: it held that under `set -u` an unbound-variable error inside a function aborts the +function but not the script, so an `if ! validate` form takes neither branch and the `exit 2` is +skipped. Measured on bash 3.2.57 and 5.3.15, under `set -u` alone as well as `set -eu` — the +distinction that matters, since the reader deliberately runs without `set -e` — the shell EXITS +outright, so nothing continues past the gate. That reading holds only inside a command substitution. +What the sentinel actually covers is the caller that never runs validation in the first place. Validation now sets a sentinel on its last line and the derived views refuse without it. Likewise a top-level `exit` in a sourced library terminates the sourcing script where it stands, so both consumers probe-source the library in a subshell and require an end-of-file marker before trusting it. diff --git a/docs/decisions/records/ci/exemption-provenance.md b/docs/decisions/records/ci/exemption-provenance.md index 6af2013a6..c4c01dd8d 100644 --- a/docs/decisions/records/ci/exemption-provenance.md +++ b/docs/decisions/records/ci/exemption-provenance.md @@ -252,15 +252,20 @@ same literal with the same anchors OVER THE SAME POPULATION — the whole file reddens CI on the PR that makes it. The population is part of the rule, not an implementation detail: that guard used to scope itself to the classify step's `run:` body, and a second anchored assignment in any other `run:` block then left it green while making the writer refuse every verdict. -Measured before it was aligned: the guard green, and about half the writer suite red. Precisely what -was missing is worth stating, since the obvious phrasing is wrong: those failures DO carry the -library's refusal, which names the count and the variable — 14 of 36 did. What was missing is a -failure POINTING AT THE WORKFLOW, because the reds surface under unrelated test names and the cause -has to be inferred from a stack of them. Such an edit reaching `main` WOULD be a repo-wide merge outage — no verdict could be +Measured before it was aligned: the guard green, and about half the writer suite red. What was +missing is worth stating precisely, because the obvious phrasing is wrong twice over. Those failures +DO carry the library's refusal, which names the count and the variable. And alignment ADDS a named +guard failure rather than replacing them. What was missing is a failure POINTING AT THE WORKFLOW, +since the reds surface under unrelated test names and the cause had to be inferred from a stack of +them. No figure is given for how many name it, and the trap is worth recording so one is not added +back: a `grep -c` over pytest output counts LINES rather than failures and `--tb=line` prints each +failure twice, so such a count overstates by 2x, while truncation moves it the other way (7 under the +default display, 31 under `--tb=long -vv` on the same run). A number that changes with display +options is not a fact about the code. Such an edit reaching `main` WOULD be a repo-wide merge outage — no verdict could be posted and `review-verdict/h10` is required — but the guard is not the only thing standing in the way, and saying so would overstate it: the writer suite reddens too, in the ADVISORY `script-tests` job (the required contexts, measured 2026-08-30, are `Build & test`, `EF migration integrity` and -`review-verdict/h10`). What alignment buys is ONE named guard failure pointing at the workflow, instead of half the suite reddening under unrelated +`review-verdict/h10`). What alignment buys is ONE named guard failure pointing at the workflow, ALONGSIDE the half-suite of unrelated failures. The route by which the outage is actually reachable is the snapshot boundary below — a verdict posted from a checkout predating the edit — which is why the two paragraphs belong together. diff --git a/scripts/check-review-verdict.sh b/scripts/check-review-verdict.sh index 4893b84da..090988281 100755 --- a/scripts/check-review-verdict.sh +++ b/scripts/check-review-verdict.sh @@ -84,11 +84,19 @@ esac # verdict word is; the one thing it must never do is fall through to `absent`, which the hook reads # as "convention not adopted" and downgrades into an `ask`. # -# THE GATE IS THE SENTINEL, NOT THIS `if`. Stated because the obvious reading is wrong: under -# `set -u` an unbound-variable error inside a function aborts the function and NOT the script, and -# in an `if ! validate` form neither branch is then taken, so the `exit 2` is skipped and execution -# continues. That is a measured fail-open, not a hypothetical — see the sentinel comment in the -# library. `etv_verdict_alternation` refusing without the sentinel is what actually holds. +# THE GATE IS THE SENTINEL, NOT THIS `if`. The fail-open this guards against is measured and real: +# with the negative list written as a scalar and `.*` in the positive list, an explicit +# `BLOCKED @ ` classified `positive`, exit 0, with only a stderr line the calling hook +# discards. +# +# THE MECHANISM ORIGINALLY GIVEN HERE WAS WRONG and is corrected rather than repeated (ersatztv#845). +# It said an unbound-variable error inside a function aborts the function and NOT the script, so an +# `if ! validate` form takes neither branch and execution continues past the gate. Measured on bash +# 3.2.57 and 5.3.15, and with `set -u` ALONE as well as `set -eu` — which matters here, since this +# reader deliberately runs without `set -e` — the shell EXITS outright: no branch runs and nothing +# after the gate runs. That reading holds only inside a command substitution, which is not this +# shape. `etv_verdict_alternation` refusing without the sentinel is what actually holds, and what it +# covers is the caller that never runs validation at all. _here=$(CDPATH='' cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) || { printf 'check-review-verdict: could not resolve this script own directory to locate the shared verdict vocabulary\n' >&2; exit 2; } VOCABULARY_LIB="$_here/lib/review-verdict-vocabulary.sh" diff --git a/scripts/lib/h10-reviewers.sh b/scripts/lib/h10-reviewers.sh index 4650d863c..949aca9a1 100644 --- a/scripts/lib/h10-reviewers.sh +++ b/scripts/lib/h10-reviewers.sh @@ -40,15 +40,19 @@ # That guard used to scope itself to the classify step's `run:` body, which is NOT the same tree this # file walks: a second anchored assignment in any other `run:` block left it green while making this # refuse every verdict. Measured before the fix: the guard green, and about half the writer suite red. -# BE PRECISE ABOUT WHAT WAS MISSING, because the obvious phrasing is wrong: those failures DO carry -# this library's refusal, which names the count and the variable — 14 of 36 did. What was missing is -# a failure that POINTS AT THE WORKFLOW: the reds surface under unrelated test names -# (`test_positive_verdict_posts_success_on_the_resolved_head`, …), so the cause has to be inferred -# from a stack of them. Aligning the populations buys ONE named guard failure instead. Counts are -# avoided elsewhere in this comment because they drift as tests are added. +# BE PRECISE ABOUT WHAT WAS MISSING, because the obvious phrasing is wrong twice over. Those failures +# DO carry this library's refusal, which names the count and the variable. And alignment does not +# REPLACE them — it ADDS a named guard failure alongside them. What was missing is a failure that +# POINTS AT THE WORKFLOW: the reds surface under unrelated test names +# (`test_positive_verdict_posts_success_on_the_resolved_head`, …), so the cause had to be inferred +# from a stack of them. # -# Stated as "about half" rather than "the whole suite" for the same reason — 36 of 75 at the time of -# writing, so "whole" would be false. +# NO FIGURE IS GIVEN FOR HOW MANY NAME IT, deliberately, and the trap is worth stating so nobody +# adds one back. A `grep -c` over pytest output counts LINES, not failures, and `--tb=line` prints +# each failure twice — so that count overstates by 2x. Truncation moves it again: the same run shows +# 7 under the default display and 31 under `--tb=long -vv`. A figure that changes with display +# options is not a fact about the code. "About half" survives because it is coarse enough to be +# robust. # # BE EXACT ABOUT THE CONSEQUENCE, because the obvious phrasing overstates it. Such an edit landing on # `main` WOULD be a repo-wide merge outage — no verdict could be posted, and `review-verdict/h10` is diff --git a/scripts/lib/review-verdict-vocabulary.sh b/scripts/lib/review-verdict-vocabulary.sh index e8e77f409..9d085dd1f 100644 --- a/scripts/lib/review-verdict-vocabulary.sh +++ b/scripts/lib/review-verdict-vocabulary.sh @@ -77,10 +77,9 @@ ETV_VERDICT_NEGATIVE_WORDS=(blocked not-mergeable) # validation in the first place: the words are unobtainable unless it ran all the way to its final # line, so a derived view cannot be reached by a path that simply did not call the validator. # -# The same refuted mechanism still appears in `release.verdict-vocabulary-shared` and in -# `test_review_verdict_vocabulary.py`; correcting a decision record is its own change, tracked as -# ersatztv#886. The fail-open that record documents is real and is not in question — only the -# mechanism it attributes. +# The same refuted mechanism still stands in `release.verdict-vocabulary-shared`; correcting a +# decision record is its own change, tracked as ersatztv#886. The fail-open that record documents is +# real and is not in question — only the mechanism it attributes. ETV_VERDICT_VOCABULARY_OK=0 # --- Validation. ------------------------------------------------------------------------------- diff --git a/scripts/tests/test_pr_changed_files.py b/scripts/tests/test_pr_changed_files.py index de1cd3625..37e5f9c42 100644 --- a/scripts/tests/test_pr_changed_files.py +++ b/scripts/tests/test_pr_changed_files.py @@ -2388,14 +2388,16 @@ def _h10_reviewers_literal(): # classify step therefore does not walk the same tree as its subject: a second anchored # assignment in ANY OTHER `run:` block is invisible here while making the writer refuse every # verdict. Measured on this file: injecting one into the last step left this test green and - # reddened about half the writer suite. Those failures DO carry the library's refusal, which - # names the count and the variable (14 of 36 did) — what they do NOT do is point at the workflow, - # since they surface under unrelated test names and the cause must be inferred from a stack of - # them. "About half" rather than "the whole suite" because 36 of 75 is not all of it. Such an edit + # reddened about half the writer suite (36 of 75). Those failures DO carry the library's + # refusal, which names the count and the variable — what they do NOT do is point at the + # workflow, since they surface under unrelated test names. No figure is given for how many name + # it: `grep -c` over pytest output counts LINES rather than failures, and `--tb=line` prints each + # failure twice, so such a count overstates by 2x; truncation moves it the other way. Such an edit # reaching `main` would be a repo-wide merge outage — no verdict could be posted and # `review-verdict/h10` is required — but the honest claim for THIS guard is narrower: the writer # suite already reddens, in the ADVISORY `script-tests` job, so what aligning the populations - # buys is ONE named guard failure pointing at the workflow, rather than half the suite reddening. + # buys is ONE named guard failure pointing at the workflow, ALONGSIDE those unrelated reds — + # it adds a diagnosis, it does not replace them. # The classify step remains what the GATE runs, and # `_classify_step()` is still the right scope for every other assertion about that body. body = WORKFLOW.read_text() diff --git a/scripts/tests/test_review_verdict_vocabulary.py b/scripts/tests/test_review_verdict_vocabulary.py index 731cfe598..badd4800d 100644 --- a/scripts/tests/test_review_verdict_vocabulary.py +++ b/scripts/tests/test_review_verdict_vocabulary.py @@ -318,18 +318,23 @@ def test_the_write_side_maps_the_shipped_words_onto_the_same_states(sandbox, tok assert sandbox.posted_status_state() == state -# --- A control-flow gate can be SKIPPED; the sentinel is what actually holds. -------------------- +# --- The sentinel, not the `if`, is what holds. --------------------------------------------------- # -# Under `set -u`, an unbound-variable error inside a FUNCTION aborts the function but not the -# script, and in an `if ! validate` form NEITHER branch is then taken — so the `exit 2` never runs -# and execution continues past the gate. `check-review-verdict.sh` runs without `set -e`, so the -# words reached the regex unvalidated: with `.*` in the positive list and the negative list written -# as a scalar, an explicit `BLOCKED @ ` classified `positive`, exit 0. The calling hook -# discards stderr, so the only signal was invisible. +# The fail-open is measured and real: with `.*` in the positive list and the negative list written as +# a scalar, unvalidated words reached the regex and an explicit `BLOCKED @ ` classified +# `positive`, exit 0. The calling hook discards stderr, so the only signal was invisible. +# +# THE MECHANISM ORIGINALLY RECORDED FOR IT WAS WRONG, corrected in ersatztv#845 and restated here +# rather than repeated: it held that under `set -u` an unbound-variable error inside a FUNCTION +# aborts the function but not the script, so an `if ! validate` form takes NEITHER branch and the +# `exit 2` never runs. Measured on bash 3.2.57 and 5.3.15, under `set -u` ALONE as well as `set -eu` +# — the case that matters, since `check-review-verdict.sh` deliberately runs without `set -e` — the +# shell EXITS outright, so nothing continues past the gate. That reading holds only inside a command +# substitution. # # The fix is a DATA dependency rather than a control-flow one: validation sets -# `ETV_VERDICT_VOCABULARY_OK=1` on its last line, and the derived views refuse without it. These -# tests pin the malformation classes that produce the abort. +# `ETV_VERDICT_VOCABULARY_OK=1` on its last line, and the derived views refuse without it. What that +# covers is the caller that never runs validation at all. These tests pin the malformation classes. def test_a_SCALAR_word_list_cannot_reach_the_regex_unvalidated(sandbox):