Build ErsatzTV Image / CI toolchain image resolves (pull_request) Successful in 35s
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 57s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 37s
PR Gates / Docs update reminder (pull_request) Successful in 1m0s
PR Gates / decisions lifecycle (pull_request) Successful in 20s
PR Gates / Fix proofs (Proves trailers) (pull_request) Successful in 17s
review-verdict/h10 Review-verdict: MERGEABLE @ a7d91bf (base: main)
Review verdict / Set review-verdict status (pull_request_target) Successful in 45s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m25s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 6m17s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
PR Gates / Script lint and tests (ruff + pytest) (pull_request) Successful in 19m27s
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m4s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 8s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 7s
`docs.no-session-narrative` reaches every durable artifact, but its detector scanned only `docs/**/*.md` and root markdown, and nothing had ever swept the rest. The issue named four sites from one grep and called them a floor. Deriving the population instead — a whitespace-joined sweep over every tracked file outside the detector, for the detector's own phrasings plus the attribution and review-round class #812 found — gave 453 sites in 108 files at `fb5592971`, and a second pass for phrasings the first list missed (hyphenated `round-N`, "an earlier version", "the reviewer proved") added residuals in the same files. Every site was classified with #812's three dispositions (CUT / SEVER / KEEP with its sub-kind) under the who-benefits test; the per-site manifests are on the PR. The rejected designs, tested-and-rejected fixtures, measurements and traps stay; the attribution of who found them and the round in which they were found go. The detector's population grows to `.claude/`, `.gitea/`, `.husky/` and `scripts/` regardless of extension, minus the detector and its own test (whose fixtures ARE the phrasings) and minus `scripts/tests/fixtures/` (test data, including decision-record copies — the same reasoning as the records' own exemption, and what keeps the record's depth measurement true), and `--all` lists tracked REGULAR files only — a symlink's content is its target and a gitlink has none. The #812 argument for leaving `docs/superpowers/**` in the population runs the other way here: `--diff` sees only ADDED lines, and 287 of the 453 sites were under 30 days old — this corpus is where narrative is being added, so the advisory nudge has reach. Density agrees: 56 line-mode hits over the 113 regular files the predicate admits, against 9 over 66 docs files before #812. `web/` and C# stay out on the same measurement (3 of 74 PATTERNS-matching sites, ~4,600 files). The predicate did not grow: PATTERNS matched 74 of 453 sites, and widening the word list to the attribution class is the treadmill the withdrawn parity test ran on. The population oracle is restated over segments with the new arms, the synthetic cross product gains the process heads and non-markdown extensions, a fixture witnesses that a tracked symlink is neither scanned nor counted, a `.py.bak` axis separates a by-name exemption from a `startswith` over the same tuple, and eight mutants (drop the process arm, drop the by-name exemption, exempt by `startswith`, drop or add a prefix, drop the fixtures exemption, list only markdown, drop the symlink filter, test the mode per row instead of per path) each redden it. A pre-existing silent drop in `--diff` goes with it: git tab-terminates a `+++` filename that contains a space, and the kept tab made `is_scanned_path` refuse the file with no notice — fixed, with a positive control and its own mutant. Code is unchanged by construction, measured per file type against `origin/main`: Python modules are AST-equal with docstrings stripped, except `#` lines inside the embedded fixture programs (string literals) of three test modules; workflows differ only in `#` lines inside `run:` block scalars; shell, C#, TypeScript and jq are equal with comment lines stripped. The stated exceptions: the detector and its test, 26 vitest titles that carried review-round or severity labels or a reviewer attribution (call sites whose title changed — every changed title line walked back to its `it(` / `it.each(...)(` anchor, so a `' + '` concatenation counts once), two registry note strings and the mutation manifest's prose fields. scripts/tests: 1565 passed. Web: lint, typecheck, 1319 tests green. Closes #876. Decisions-Edit: yes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PEcBoFw7ctrf3Nb7R7x7wk
7271 lines
389 KiB
Python
7271 lines
389 KiB
Python
"""Tests for `scripts/pr-changed-files.sh` — the SHARED PR file enumeration (ersatztv#649).
|
|
|
|
Why this file exists. The enumeration used to be written twice: once in
|
|
`.claude/hooks/pretooluse-merge-consent.sh` (ADVISORY — a failure produces a human prompt) and once
|
|
inline in `.gitea/workflows/review-verdict.yml` (ENFORCED — it writes the branch-protection-required
|
|
`review-verdict/h10` status). They drifted, and in the dangerous direction: four rounds of
|
|
ersatztv#643 hardening landed on the advisory copy and never reached the enforced one, so the copy
|
|
with real authority ended up strictly weaker than the copy without.
|
|
|
|
The specific thing this suite pins is the point of ersatztv#649's second Done-when box. The
|
|
enforced copy's fail-closed behaviour on a garbage response was INCIDENTAL, not designed: `n` came
|
|
back empty, `[ "$n" -lt 50 ]` errored to false, the loop ran to MAX_PAGES and left complete=no. The
|
|
right answer, reached through a bash arithmetic error that any refactor of the
|
|
loop could have silently flipped. Every failure-path test below therefore asserts a NON-ZERO exit
|
|
explicitly, so the behaviour is a contract rather than a coincidence.
|
|
|
|
Observable contract of the script:
|
|
exit 0 -> enumeration complete, with no head/base movement observable from inside it (ONE-WAY;
|
|
an A->B->A alias passes — #664/#803); stdout is the authoritative path set
|
|
exit 1 -> could not enumerate/verify; stdout meaningless, caller MUST withhold any exemption
|
|
exit 2 -> usage error
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from scripts.tests import hook_fire_isolation
|
|
from scripts.tests.tracked_files import tracked_paths
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
SCRIPT = REPO_ROOT / "scripts" / "pr-changed-files.sh"
|
|
HOOK = REPO_ROOT / ".claude" / "hooks" / "pretooluse-merge-consent.sh"
|
|
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "review-verdict.yml"
|
|
|
|
SHA = "a9e3e23abf337980ca4c05854f5b1e210099d08b"
|
|
OTHER_SHA = "b71c0d4e2f8a91b3c5d7e9f1a3b5c7d9e1f3a5b7"
|
|
|
|
# Serves paged `pulls/N/files`, plus the PR object the script re-reads to bind the enumeration.
|
|
CURL_SHIM = r"""#!/usr/bin/env python3
|
|
import json, os, sys, pathlib, urllib.parse
|
|
|
|
state = pathlib.Path(os.environ["STUB_DIR"])
|
|
args = sys.argv[1:]
|
|
url = [a for a in args if a.startswith("http")][-1]
|
|
|
|
if "/pulls/" in url and "/files" in url:
|
|
q = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
|
|
page = int(q.get("page", ["1"])[0])
|
|
pages = json.loads((state / "pages.json").read_text())
|
|
if page > len(pages):
|
|
print("[]"); sys.exit(0)
|
|
entry = pages[page - 1]
|
|
if entry == "ERROR": # transport failure on this page
|
|
sys.exit(22)
|
|
if entry == "GARBAGE": # 200 with a non-array body (proxy/error page)
|
|
print('{"message":"internal error"}'); sys.exit(0)
|
|
print(json.dumps(entry)); sys.exit(0)
|
|
|
|
if "/pulls/" in url:
|
|
# The PR object is read TWICE now: once before paging to bind the base, once after to bind both
|
|
# the head and the base (ersatztv#698). A counter distinguishes them, so a test can move either
|
|
# field at either end and the two guards can be isolated from each other.
|
|
counter = state / "pr_reads.txt"
|
|
n = int(counter.read_text()) if counter.exists() else 0
|
|
counter.write_text(str(n + 1))
|
|
|
|
sha = os.environ["STUB_SHA"]
|
|
alt = state / "pr_sha_after.txt"
|
|
if alt.exists() and n >= 1: # force-push landing between pagination round-trips
|
|
sha = alt.read_text().strip()
|
|
|
|
base = os.environ.get("STUB_BASE", "main")
|
|
base_before = state / "pr_base_before.txt"
|
|
base_after = state / "pr_base_after.txt"
|
|
if base_before.exists() and n == 0: # already retargeted when the job started
|
|
base = base_before.read_text().strip()
|
|
if base_after.exists() and n >= 1: # retargeted mid-enumeration
|
|
base = base_after.read_text().strip()
|
|
|
|
# The base's TIP sha (ersatztv#707) — independent of the branch NAME above. Defaults to a fixed
|
|
# value so an unmoved base is the default for every existing test that never touches this.
|
|
base_sha = os.environ.get("STUB_BASE_SHA", "aaaa000000000000000000000000000000000a")
|
|
base_sha_before = state / "pr_base_sha_before.txt"
|
|
base_sha_after = state / "pr_base_sha_after.txt"
|
|
if base_sha_before.exists() and n == 0:
|
|
base_sha = base_sha_before.read_text().strip()
|
|
if base_sha_after.exists() and n >= 1:
|
|
base_sha = base_sha_after.read_text().strip()
|
|
|
|
base_obj = {"ref": base}
|
|
if base_sha != "MISSING":
|
|
base_obj["sha"] = base_sha
|
|
|
|
print(json.dumps({"head": {"sha": sha}, "base": base_obj}))
|
|
sys.exit(0)
|
|
|
|
print("{}")
|
|
"""
|
|
|
|
|
|
def _rows(paths, status="modified"):
|
|
return [{"filename": p, "status": status} for p in paths]
|
|
|
|
|
|
@pytest.fixture
|
|
def enumerate_files(tmp_path):
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
shim = bindir / "curl"
|
|
shim.write_text(CURL_SHIM)
|
|
shim.chmod(0o755)
|
|
state = tmp_path / "state"
|
|
state.mkdir()
|
|
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["STUB_DIR"] = str(state)
|
|
env["STUB_SHA"] = SHA
|
|
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
|
env["ETV_GITEA_URL"] = "http://gitea.example"
|
|
env.pop("ETV_GITEA_BASICAUTH", None)
|
|
env.pop("GITEA_TOKEN", None)
|
|
|
|
class Handle:
|
|
def __init__(self, env_, state_):
|
|
self.env = env_
|
|
self.state = state_
|
|
|
|
def set_pages(self, *pages):
|
|
(state / "pages.json").write_text(json.dumps(list(pages)))
|
|
|
|
def head_moves_to(self, sha):
|
|
(state / "pr_sha_after.txt").write_text(sha)
|
|
|
|
def base_is_already(self, ref):
|
|
"""The PR targets `ref` before the first page is requested (retargeted pre-run)."""
|
|
(state / "pr_base_before.txt").write_text(ref)
|
|
|
|
def base_moves_to(self, ref):
|
|
"""The PR is retargeted to `ref` between the files pages and the binding re-read."""
|
|
(state / "pr_base_after.txt").write_text(ref)
|
|
|
|
def base_sha_is_already(self, sha):
|
|
"""The base tip sits at `sha` for the WHOLE run — i.e. it advanced before this
|
|
enumeration started, which is outside the window the #707 guard polices."""
|
|
(state / "pr_base_sha_before.txt").write_text(sha)
|
|
(state / "pr_base_sha_after.txt").write_text(sha)
|
|
|
|
def base_sha_moves_to(self, sha):
|
|
"""The base branch's TIP advances to `sha` between the files pages and the binding
|
|
re-read, with the branch NAME unchanged (ersatztv#707)."""
|
|
(state / "pr_base_sha_after.txt").write_text(sha)
|
|
|
|
def base_sha_missing_after(self):
|
|
"""The base tip sha is absent/unparseable on the post-paging re-read."""
|
|
(state / "pr_base_sha_after.txt").write_text("MISSING")
|
|
|
|
def run(self, expected_sha=SHA, args=("timothy", "ersatztv", "42"), expected_base="main"):
|
|
argv = ["bash", str(SCRIPT), *args, expected_sha]
|
|
if expected_base is not None:
|
|
argv.append(expected_base)
|
|
return subprocess.run(argv, env=self.env, capture_output=True, text=True)
|
|
|
|
def paths(self):
|
|
"""Assert success and return the enumerated path set."""
|
|
r = self.run()
|
|
assert r.returncode == 0, f"expected success, got {r.returncode}: {r.stderr}"
|
|
return [ln for ln in r.stdout.splitlines() if ln.strip()]
|
|
|
|
def fails_closed(self):
|
|
"""The whole point: a NON-ZERO exit, asserted, not inferred."""
|
|
r = self.run()
|
|
return r.returncode != 0
|
|
|
|
return Handle(env, state)
|
|
|
|
|
|
# --- The happy path, so the failure-path assertions below cannot pass vacuously ----------------
|
|
|
|
|
|
def test_complete_enumeration_returns_every_path(enumerate_files):
|
|
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs", "README.md"]))
|
|
assert enumerate_files.paths() == ["docs/a.md", "ErsatzTV/Program.cs", "README.md"]
|
|
|
|
|
|
def test_a_rename_contributes_BOTH_sides(enumerate_files):
|
|
"""One row, two paths — the `git mv` hole. Reading `.filename` alone hides the source."""
|
|
enumerate_files.set_pages(
|
|
[
|
|
{
|
|
"filename": "docs/innocuous-note.md",
|
|
"status": "renamed",
|
|
"previous_filename": ".gitea/workflows/renovate.yml",
|
|
}
|
|
]
|
|
)
|
|
assert sorted(enumerate_files.paths()) == [".gitea/workflows/renovate.yml", "docs/innocuous-note.md"]
|
|
|
|
|
|
def test_paths_on_a_later_page_are_included(enumerate_files):
|
|
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["scripts/decisions_lib.py"]))
|
|
assert "scripts/decisions_lib.py" in enumerate_files.paths()
|
|
|
|
|
|
# --- Fail-closed contract: each of these MUST be non-zero, by design ---------------------------
|
|
|
|
|
|
def test_transport_failure_mid_pagination_fails_closed(enumerate_files):
|
|
"""The defect that started all of this: an errored page counted as zero rows and read as
|
|
'end of list', completing the enumeration over a PARTIAL list."""
|
|
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), "ERROR", _rows(["docs/tail.md"]))
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_non_array_body_fails_closed(enumerate_files):
|
|
enumerate_files.set_pages("GARBAGE")
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_an_OBJECT_OF_VALID_ROWS_isolates_the_top_level_array_check(enumerate_files):
|
|
"""`test_non_array_body_fails_closed` passes for the wrong reason, and the nearest fixes for
|
|
it fail for a THIRD reason — worth recording, because both near-misses look like coverage.
|
|
|
|
`jq`'s `all(.[]; …)` iterates an object's VALUES, so the top-level `type == "array"` check is
|
|
only load-bearing when those values would themselves validate:
|
|
|
|
* `{"message":"internal error"}` — values are strings, `.filename` on a string errors. Rejected
|
|
with the clause deleted, so it never isolated it.
|
|
* `{"filename":"docs/a.md","status":"modified"}` — a single row, but its values are still
|
|
strings. Same non-isolation, one level less obvious.
|
|
* `{"0": {"filename":"docs/a.md","status":"modified"}}` — values ARE valid rows. With the clause
|
|
deleted this validates, `length` is 1, the path is collected, and the next page ends the
|
|
enumeration cleanly: a non-array body enumerated as a complete docs-only list. That is the
|
|
fail-open, and only this shape exposes it.
|
|
"""
|
|
enumerate_files.set_pages({"0": {"filename": "docs/a.md", "status": "modified"}})
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_rows_without_filename_fail_closed(enumerate_files):
|
|
"""`[{}]` is a well-formed array that yields no paths — a partial list wearing a valid shape."""
|
|
enumerate_files.set_pages([{}, {}])
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_a_row_with_a_VALID_status_but_no_filename_isolates_the_filename_check(enumerate_files):
|
|
"""Same wrong-reason problem: `[{}]` is also rejected by the closed `.status` allow-list, since
|
|
an absent status is not in it. A row carrying a legitimate `status` and no `filename` removes
|
|
that second reason, leaving only the guard under test."""
|
|
enumerate_files.set_pages([{"status": "modified"}])
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_array_of_scalars_fails_closed(enumerate_files):
|
|
enumerate_files.set_pages(["docs/a.md", "docs/b.md"])
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
@pytest.mark.parametrize("evil", ["safe.md\ndocs/Program.cs", "safe.md\rdocs/Program.cs"])
|
|
def test_CRLF_in_filename_fails_closed(enumerate_files, evil):
|
|
"""A newline splits one path into two lines that are each allow-list-matched separately, so
|
|
`safe.md\\ndocs/Program.cs` reads as two exempt paths while the real path ends in .cs."""
|
|
enumerate_files.set_pages(_rows([evil]))
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_CRLF_in_previous_filename_on_a_NON_renamed_row_fails_closed(enumerate_files):
|
|
"""The hole one predicate wide: `previous_filename` is CONSUMED on every row, so it must be
|
|
VALIDATED on every row — not only where `.status == "renamed"` makes it semantically expected."""
|
|
enumerate_files.set_pages(
|
|
[{"filename": "docs/a.md", "status": "modified", "previous_filename": "safe.md\ndocs/Program.cs"}]
|
|
)
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_dotdot_path_component_fails_closed(enumerate_files):
|
|
"""The callers' allow-lists anchor `^docs/`, which `docs/../ErsatzTV/Program.cs` matches."""
|
|
enumerate_files.set_pages(_rows(["docs/../ErsatzTV/Program.cs"]))
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
@pytest.mark.parametrize("status", ["Renamed", "moved", ""])
|
|
def test_status_outside_the_closed_allow_list_fails_closed(enumerate_files, status):
|
|
"""Without a closed set, the `renamed => previous_filename REQUIRED` clause is dodgeable by any
|
|
other value, letting a `git mv` drop its source path and read as docs-only."""
|
|
enumerate_files.set_pages([{"filename": "docs/a.md", "status": status, "previous_filename": "ErsatzTV/Program.cs"}])
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_renamed_row_without_previous_filename_fails_closed(enumerate_files):
|
|
enumerate_files.set_pages([{"filename": "docs/a.md", "status": "renamed"}])
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_head_differing_from_the_expected_sha_fails_closed(enumerate_files):
|
|
"""Scoped to what this actually proves.
|
|
|
|
The stub serves the alternate sha from the BINDING read (the one after paging; since
|
|
ersatztv#698 the script also reads the PR object BEFORE paging, to bind the base). So this
|
|
exercises "the final head does not equal the expected sha", not movement *during* enumeration.
|
|
The distinction matters: an A->B->A force-push round trip would restore the expected sha and
|
|
pass this check while the pages came from two different states. That race is inherent to
|
|
enumerating a mutable list over several round-trips against an API with no commit-pinned files
|
|
endpoint, and is tracked separately rather than papered over with a test name that implies it is
|
|
covered.
|
|
"""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
enumerate_files.head_moves_to(OTHER_SHA)
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_missing_credentials_fails_closed(enumerate_files):
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
enumerate_files.env.pop("ETV_GITEA_TOKEN", None)
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
@pytest.mark.parametrize("args", [("timothy", "ersatztv"), ("timothy", "ersatztv", "")])
|
|
def test_usage_errors_exit_2(enumerate_files, args):
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
r = (
|
|
enumerate_files.run(args=args)
|
|
if len(args) == 3
|
|
else subprocess.run(["bash", str(SCRIPT), *args], env=enumerate_files.env, capture_output=True, text=True)
|
|
)
|
|
assert r.returncode == 2, r.stderr
|
|
|
|
|
|
# --- The base-ref binding (ersatztv#698 route 1) ------------------------------------------------
|
|
#
|
|
# `/pulls/{n}/files` diffs against the PR's LIVE base, so retargeting changes the answer without
|
|
# moving the head. Reproduced on the real instance as probe PR #703: opened into `main`, retargeted
|
|
# mid-run, enumerated docs-only, granted `review-verdict/h10=success` while its diff against `main`
|
|
# carried a C# file.
|
|
|
|
|
|
def test_a_FOUR_argument_call_is_a_usage_error_not_an_unbound_enumeration(enumerate_files):
|
|
"""The binding is REQUIRED, not optional.
|
|
|
|
This is the test that matters most for the shape of the fix. Had the base ref been added as an
|
|
OPTIONAL 5th argument, every existing caller would have kept compiling and kept running
|
|
unbound — an opt-out that is invisible at the call site, and the caller most likely to omit it
|
|
is the one that most needed it. Dropping the argument must be LOUD.
|
|
"""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
r = enumerate_files.run(expected_base=None)
|
|
assert r.returncode == 2, (
|
|
"a 4-argument call was accepted, so the base binding is effectively optional and any caller "
|
|
f"that forgets it silently enumerates against a mutable base (stdout={r.stdout!r})"
|
|
)
|
|
|
|
|
|
def test_an_EMPTY_base_ref_argument_fails_closed(enumerate_files):
|
|
"""The hook passes `.base.ref` straight from PR JSON; unparseable JSON yields an empty string."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
r = enumerate_files.run(expected_base="")
|
|
assert r.returncode == 2, r.stderr
|
|
|
|
|
|
def test_a_PR_already_retargeted_before_the_enumeration_fails_closed(enumerate_files):
|
|
"""The pre-paging read. Without it, a PR retargeted before the job started would enumerate
|
|
against the scratch base with every page agreeing with every other page — internally consistent
|
|
and entirely wrong."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
enumerate_files.base_is_already("probe/scratch-base")
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_a_RETARGET_DURING_the_enumeration_fails_closed(enumerate_files):
|
|
"""The post-paging read. The head never moves here, which is the entire point: the existing
|
|
head-sha binding cannot see a retarget, so removing the base half of the binding leaves this
|
|
case exempted."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
enumerate_files.base_moves_to("probe/scratch-base")
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_positive_control_an_unmoved_base_still_enumerates(enumerate_files):
|
|
"""Without this, the three tests above could pass because the 5-argument call is broken outright
|
|
rather than because the binding works."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs"]))
|
|
assert enumerate_files.paths() == ["docs/a.md", "ErsatzTV/Program.cs"]
|
|
|
|
|
|
def test_a_BASE_ADVANCE_DURING_enumeration_fails_closed(enumerate_files):
|
|
"""ersatztv#707. `/pulls/{n}/files` diffs EACH PAGE against the base's LIVE tip; if `main`
|
|
advances between the pre-paging read and the post-paging re-read, later pages can be diffed
|
|
against a base earlier pages never saw. Rows can drop out of the result entirely (a file `main`
|
|
no longer differs on) while later rows shift into offset ranges already consumed against the old
|
|
tip. `.base.ref` never changes here — that is the entire point of the defect and of this being an
|
|
ADDITIONAL guard layered on top of the existing retarget check, not a reclassification of it."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md", "ErsatzTV/Program.cs"]))
|
|
enumerate_files.base_sha_moves_to("cccc000000000000000000000000000000000c")
|
|
r = enumerate_files.run()
|
|
assert r.returncode != 0, f"expected fail-closed, got {r.returncode}: stdout={r.stdout!r}"
|
|
assert not r.stdout.strip(), f"stdout must be meaningless on a failed enumeration: {r.stdout!r}"
|
|
assert "base" in r.stderr and "advanced" in r.stderr, (
|
|
f"stderr should name the base-advance failure distinctly, got: {r.stderr!r}"
|
|
)
|
|
assert re.search(r"[0-9a-f]{7}", r.stderr), f"stderr should name both short shas involved, got: {r.stderr!r}"
|
|
|
|
|
|
def test_positive_control_an_UNMOVED_base_sha_still_enumerates_across_pages(enumerate_files):
|
|
"""Without this, the fail-closed tests around it could be passing only because the new guard
|
|
broke the ordinary path outright rather than because it correctly distinguishes movement from
|
|
no movement. Deliberately multi-page, to prove the guard survives several round trips."""
|
|
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), _rows(["ErsatzTV/Program.cs"]))
|
|
paths = enumerate_files.paths()
|
|
assert len(paths) == 51
|
|
assert "ErsatzTV/Program.cs" in paths
|
|
|
|
|
|
def test_the_707_WINDOW_base_ref_unchanged_but_TIP_advanced_still_fails(enumerate_files):
|
|
"""The exact #707 window, and the centerpiece of this suite. The base BRANCH NAME is unchanged
|
|
throughout (`main` -> `main`), so the pre-existing `.base.ref` retarget guard sees nothing to
|
|
object to — by construction, since a mere advance is not a retarget. Only the branch's TIP moved.
|
|
A test that passes here proves the NEW sha-based guard is what fired, not the old ref guard,
|
|
which cannot see this case at all."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]), _rows(["ErsatzTV/Program.cs"]))
|
|
enumerate_files.base_sha_moves_to("dddd000000000000000000000000000000000d")
|
|
r = enumerate_files.run() # expected_base="main" throughout; ref never moves
|
|
assert r.returncode != 0, (
|
|
"base ref stayed 'main' but its tip advanced mid-enumeration; must still fail closed "
|
|
f"(got {r.returncode}, stdout={r.stdout!r}, stderr={r.stderr!r})"
|
|
)
|
|
|
|
|
|
def test_a_base_that_ADVANCED_BEFORE_the_enumeration_STILL_SUCCEEDS(enumerate_files):
|
|
"""ersatztv#707, the deadlock this guard must NOT cause — and the reason the check is scoped to
|
|
the window rather than to the base being "current".
|
|
|
|
`main` advances constantly. If the guard compared the base tip against anything other than what
|
|
it observed at the START of THIS enumeration, every open PR would fail closed on every unrelated
|
|
merge to `main` — losing its exemption for reasons that have nothing to do with it. Here the base
|
|
already sits at a tip different from the suite's default before the first page is requested, and
|
|
then holds still: that is an ordinary, healthy PR and it must enumerate normally.
|
|
|
|
Note this is deliberately NOT the same test as the unmoved-base control: that one pins "nothing
|
|
happened", this one pins "something happened, but OUTSIDE the window, so it is none of our
|
|
business."
|
|
"""
|
|
enumerate_files.base_sha_is_already("eeee000000000000000000000000000000000e")
|
|
enumerate_files.set_pages(_rows(["docs/a.md", "docs/b.md"]), [])
|
|
assert enumerate_files.paths() == ["docs/a.md", "docs/b.md"], (
|
|
"a PR whose base advanced BEFORE this enumeration began was failed closed; the guard is "
|
|
"comparing against a stale expectation instead of the tip it actually started from"
|
|
)
|
|
|
|
|
|
def test_an_UNREADABLE_base_sha_on_the_AFTER_read_fails_closed(enumerate_files):
|
|
"""Consistent with every other guard in this script: an absent/unparseable field is never read
|
|
as 'no movement' — it is fail-closed, same as the head-sha and base-ref binding above."""
|
|
enumerate_files.set_pages(_rows(["docs/a.md"]))
|
|
enumerate_files.base_sha_missing_after()
|
|
assert enumerate_files.fails_closed()
|
|
|
|
|
|
def test_the_base_comparator_is_the_BRANCH_NAME_never_the_TIP_SHA():
|
|
"""`.base.ref` is compared, never `.base.sha` — matching `post-review-verdict.sh` (ersatztv#632).
|
|
|
|
Comparing tips would fail every enumeration on every unrelated merge to `main`: a self-inflicted
|
|
deadlock dressed as a security control.
|
|
|
|
This is a STRUCTURAL assertion on purpose. A behavioural version — `head_moves_to(SHA)`, the
|
|
sha that is ALREADY current — models no movement at all and is a duplicate positive control that
|
|
would pass just as happily against a script comparing tip shas. The stub serves only branch
|
|
names, so no behavioural test in this harness can distinguish the two comparators; say so and
|
|
assert the source instead.
|
|
"""
|
|
src = SCRIPT.read_text()
|
|
assert ".base.ref" in src, "the enumeration no longer reads .base.ref"
|
|
binding = [ln for ln in src.splitlines() if "base_before=" in ln or "base_after=" in ln]
|
|
assert binding, "no base binding assignments found"
|
|
for ln in binding:
|
|
assert ".base.ref" in ln and ".base.sha" not in ln, (
|
|
f"the base binding compares a tip sha, which deadlocks on ordinary churn: {ln.strip()!r}"
|
|
)
|
|
|
|
|
|
def test_a_SHORT_page_does_not_end_the_enumeration(enumerate_files):
|
|
"""Gitea caps `limit` at the server-wide MAX_RESPONSE_ITEMS and may return fewer than asked.
|
|
'Fewer than 50 rows means last page' would complete over a partial list without any transport
|
|
error — so termination requires a validated EMPTY page. A 30-row page followed by code must be
|
|
seen."""
|
|
enumerate_files.set_pages(_rows([f"docs/f{i}.md" for i in range(30)]), _rows(["ErsatzTV/Program.cs"]))
|
|
assert "ErsatzTV/Program.cs" in enumerate_files.paths()
|
|
|
|
|
|
# --- The same guard, under the runner's jq 1.6 ------------------------------------------------
|
|
#
|
|
# `test_transport_failure_mid_pagination_fails_closed` above does NOT isolate the explicit
|
|
# `if [ -z "${raw//[[:space:]]/}" ]` clause — mutation confirms it: deleting that clause leaves
|
|
# the whole suite green on a developer Mac, because jq 1.8 rejects empty
|
|
# input on its own. jq 1.6 does not, and the runner ships 1.6 — so the one environment where the
|
|
# clause is load-bearing was the one environment with no coverage. That is the #643/#647 failure
|
|
# class exactly, reproduced in the test suite meant to prevent it.
|
|
#
|
|
# The shim is IMPORTED, not copied. A second copy of a version-quirk emulator is the same
|
|
# two-implementations-drift problem this whole issue is about, one level down.
|
|
from scripts.tests.test_merge_consent_exemption import _JQ16_SHIM # noqa: E402
|
|
|
|
|
|
@pytest.fixture
|
|
def enumerate_files_jq16(enumerate_files, tmp_path):
|
|
"""The same harness, plus a jq shim reproducing jq 1.6's empty-input `-e` exit status."""
|
|
jq = tmp_path / "bin" / "jq"
|
|
jq.write_text(_JQ16_SHIM)
|
|
jq.chmod(0o755)
|
|
return enumerate_files
|
|
|
|
|
|
def test_jq16_shim_actually_reproduces_the_quirk(enumerate_files_jq16, tmp_path):
|
|
"""Verify the verifier. A shim that failed to install would make the test below pass vacuously,
|
|
reporting the guard safe on jq 1.6 without ever exercising the quirk."""
|
|
assert enumerate_files_jq16 is not None # the fixture is what installs the shim
|
|
jq = str(tmp_path / "bin" / "jq")
|
|
empty = subprocess.run([jq, "-e", "."], input="", capture_output=True, text=True)
|
|
assert empty.returncode == 0, "the shim does not reproduce jq 1.6's empty-input exit 0"
|
|
real = subprocess.run([jq, "-e", ".a"], input='{"a":1}', capture_output=True, text=True)
|
|
assert real.returncode == 0 and real.stdout.strip() == "1", "the shim broke ordinary jq"
|
|
false = subprocess.run([jq, "-e", ".a"], input='{"a":false}', capture_output=True, text=True)
|
|
assert false.returncode == 1, "the shim broke jq's real -e semantics for a false result"
|
|
|
|
|
|
def test_transport_failure_mid_pagination_fails_closed_on_jq16(enumerate_files_jq16):
|
|
"""The property, asserted on the interpreter that actually runs it in CI."""
|
|
enumerate_files_jq16.set_pages(_rows([f"docs/f{i}.md" for i in range(50)]), "ERROR", _rows(["docs/tail.md"]))
|
|
assert enumerate_files_jq16.fails_closed()
|
|
|
|
|
|
def test_a_docs_only_pr_still_enumerates_cleanly_on_jq16(enumerate_files_jq16):
|
|
"""Positive control: the shim must not make everything fail, or the test above proves nothing."""
|
|
enumerate_files_jq16.set_pages(_rows(["docs/a.md", "docs/b.md"]))
|
|
assert enumerate_files_jq16.paths() == ["docs/a.md", "docs/b.md"]
|
|
|
|
|
|
# --- The CALLER contract: a non-zero exit must withhold the exemption, on its own -------------
|
|
#
|
|
# This is the single line the whole extraction rests on, and it was the one guard nothing pinned. A
|
|
# review mutated the hook's `if files=$(...)` into `files=$(...) || true; files_complete=yes` — i.e.
|
|
# ignore the exit status entirely — and the ENTIRE suite still passed.
|
|
#
|
|
# It survives today only by REDUNDANCY: the script writes stdout once, immediately before `exit 0`,
|
|
# so every failure path also happens to yield empty stdout, and the hook's independent
|
|
# `[ -n "$files" ]` check catches it. That is exactly the shape this PR criticises elsewhere — safe
|
|
# by accident rather than by assertion. Any future change that streams pages, or prints a partial
|
|
# list before failing, turns it into a live false exemption.
|
|
#
|
|
# So the stub below FAILS *while emitting a perfectly docs-only list*, which is the one combination
|
|
# the redundancy cannot absorb. The positive control immediately after it proves the harness can
|
|
# actually observe the difference, rather than reporting "not exempt" for some unrelated reason.
|
|
|
|
_HOOK_STUB_CURL_TEMPLATE = r"""#!/usr/bin/env python3
|
|
import json, sys
|
|
url = [a for a in sys.argv[1:] if a.startswith("http")][-1]
|
|
if "/pulls/" in url and "/files" not in url:
|
|
print(json.dumps({"head": {"sha": "%s"}, "body": "no linked issue"}))
|
|
else:
|
|
print("{}")
|
|
"""
|
|
|
|
# Percent formatting is required: the template is Python source containing literal `{}`,
|
|
# which .format() would eat.
|
|
HOOK_STUB_CURL = _HOOK_STUB_CURL_TEMPLATE % SHA
|
|
|
|
|
|
def _mirror_tree(tmp_path, stub_body):
|
|
"""Lay out a minimal repo mirror so the hook resolves OUR stub as the shared script.
|
|
|
|
The hook finds the script via `${BASH_SOURCE[0]}/../..`, so the mirror must reproduce the real
|
|
`.claude/hooks/` + `scripts/` shape rather than just dropping the stub anywhere.
|
|
"""
|
|
hooks = tmp_path / ".claude" / "hooks"
|
|
hooks.mkdir(parents=True)
|
|
(hooks / "pretooluse-merge-consent.sh").write_text(HOOK.read_text())
|
|
scripts = tmp_path / "scripts"
|
|
scripts.mkdir()
|
|
stub = scripts / "pr-changed-files.sh"
|
|
stub.write_text(stub_body)
|
|
stub.chmod(0o755)
|
|
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
curl = bindir / "curl"
|
|
curl.write_text(HOOK_STUB_CURL)
|
|
curl.chmod(0o755)
|
|
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
|
env["ETV_GITEA_URL"] = "http://gitea.example"
|
|
env.pop("ETV_GITEA_BASICAUTH", None)
|
|
|
|
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
|
|
r = subprocess.run(
|
|
["bash", str(hooks / "pretooluse-merge-consent.sh")],
|
|
input=json.dumps(payload),
|
|
env=env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
# Exempt == passthrough == exit 0 with no decision JSON on stdout.
|
|
return r.stdout.strip() == ""
|
|
|
|
|
|
DOCS_ONLY_OUTPUT = 'printf "docs/a.md\\ndocs/b.md\\n"\n'
|
|
|
|
|
|
def test_a_FAILING_script_withholds_the_exemption_even_when_stdout_looks_docs_only(tmp_path):
|
|
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "exit 1\n")
|
|
assert exempt is False, (
|
|
"the hook granted a docs-only exemption from the stdout of a script that FAILED — the exit "
|
|
"status is not being checked"
|
|
)
|
|
|
|
|
|
def test_positive_control_the_same_output_with_exit_0_DOES_exempt(tmp_path):
|
|
"""Without this, the test above could pass for any unrelated reason and prove nothing."""
|
|
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "exit 0\n")
|
|
assert exempt is True, (
|
|
"the positive control failed, so the negative test above cannot be trusted to be measuring "
|
|
"the exit status at all"
|
|
)
|
|
|
|
|
|
def test_a_script_that_crashes_also_withholds_the_exemption(tmp_path):
|
|
"""Not every failure is a clean `exit 1` — a crash must not read as success either."""
|
|
exempt = _mirror_tree(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY_OUTPUT + "kill -TERM $$\n")
|
|
assert exempt is False
|
|
|
|
|
|
def test_a_MISSING_script_withholds_the_exemption(tmp_path):
|
|
"""The transition window: the shared checkout has no such script until this lands."""
|
|
hooks = tmp_path / ".claude" / "hooks"
|
|
hooks.mkdir(parents=True)
|
|
(hooks / "pretooluse-merge-consent.sh").write_text(HOOK.read_text())
|
|
(tmp_path / "scripts").mkdir()
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
curl = bindir / "curl"
|
|
curl.write_text(HOOK_STUB_CURL)
|
|
curl.chmod(0o755)
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["ETV_GITEA_TOKEN"] = "stub" # noqa: S105 - deliberately fake; the real credential comes from the environment
|
|
env["ETV_GITEA_URL"] = "http://gitea.example"
|
|
payload = {"tool_input": {"method": "merge", "owner": "timothy", "repo": "ersatztv", "pull_number": 42}}
|
|
r = subprocess.run(
|
|
["bash", str(hooks / "pretooluse-merge-consent.sh")],
|
|
input=json.dumps(payload),
|
|
env=env,
|
|
capture_output=True,
|
|
text=True,
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert r.stdout.strip() != "", "a missing shared script must not grant an exemption"
|
|
|
|
|
|
# --- Drift guard: the reason this file is worth having at all ----------------------------------
|
|
#
|
|
# Scope is BOTH callers. Until the enforced workflow was rewired (the follow-up half of ersatztv#649)
|
|
# this guard could only assert the hook, which left the copy with real authority — the one that
|
|
# writes the branch-protection-required `review-verdict/h10` status — unpinned. That asymmetry was
|
|
# the entire subject of the issue, so a guard that covered only the advisory side would have been
|
|
# the same mistake one level up.
|
|
|
|
CALLERS = pytest.mark.parametrize("caller", [HOOK, WORKFLOW], ids=["advisory-hook", "enforced-workflow"])
|
|
|
|
|
|
def _code_lines(path: Path) -> str:
|
|
"""Strip comment lines.
|
|
|
|
Both callers' comments legitimately discuss the `pulls/N/files` endpoint, and a future comment
|
|
writing `pulls/$pr/files?limit=100` as an example of what NOT to do would redden these tests —
|
|
which, since `script-tests` red blocks merges via the combined status, would block the repo over
|
|
a piece of prose. `#`-prefixed works for both files: YAML comments and the shell comments inside
|
|
the workflow's `run:` block share the marker.
|
|
"""
|
|
return "\n".join(ln for ln in path.read_text().splitlines() if not ln.lstrip().startswith("#"))
|
|
|
|
|
|
@CALLERS
|
|
def test_the_caller_uses_the_shared_script(caller):
|
|
assert "scripts/pr-changed-files.sh" in _code_lines(caller), (
|
|
f"{caller.relative_to(REPO_ROOT)} no longer calls the shared enumeration"
|
|
)
|
|
|
|
|
|
@CALLERS
|
|
def test_the_caller_passes_the_BASE_REF_argument(caller):
|
|
"""Both callers must invoke the 5-argument form (ersatztv#698 route 1).
|
|
|
|
The runtime tests prove the SCRIPT rejects a 4-argument call. They cannot prove a caller still
|
|
makes a 5-argument one, and the failure is quiet in opposite directions at the two sites: the
|
|
workflow would die on `set -u` (fail-closed, but it takes every PR with it), while the hook would
|
|
pass an empty base and lose every docs-only exemption. Pin the call site itself.
|
|
|
|
Matching is deliberately narrow. Keying on "any line mentioning the script name" matches the
|
|
enum_error MESSAGE string, failing for a reason that has nothing to do with the call.
|
|
The workflow also invokes through `"$ENUM"` rather than the literal path, so the alias is resolved
|
|
here and asserted to point at the shared script — otherwise this test could be satisfied while
|
|
`ENUM` pointed somewhere else entirely.
|
|
"""
|
|
code = _code_lines(caller)
|
|
if '"$ENUM"' in code:
|
|
assert re.search(r"^\s*ENUM=\S*scripts/pr-changed-files\.sh\s*$", code, re.M), (
|
|
f'{caller.relative_to(REPO_ROOT)} invokes "$ENUM" but ENUM is not assigned the shared enumeration script'
|
|
)
|
|
invocations = [ln for ln in code.splitlines() if re.search(r'\$\(\s*"(\$ENUM|[^"]*pr-changed-files\.sh)"', ln)]
|
|
assert invocations, f"{caller.relative_to(REPO_ROOT)} has no executable call to the shared enumeration"
|
|
for ln in invocations:
|
|
after = ln.split('"', 2)[2] if '"$ENUM"' in ln else ln.split("pr-changed-files.sh", 1)[1]
|
|
args = re.findall(r'"[^"]*\$[^"]*"', after)
|
|
assert len(args) >= 5, (
|
|
f"{caller.relative_to(REPO_ROOT)} calls the enumeration with {len(args)} quoted "
|
|
f"arguments, expected 5 including the expected base ref: {ln.strip()!r}"
|
|
)
|
|
# Counting five arguments is not enough — passing `"$SHA"` twice satisfies the count
|
|
# while stalling every real exemption. Name the fifth.
|
|
assert re.search(r"(?i)base", args[4]), (
|
|
f"{caller.relative_to(REPO_ROOT)} passes {args[4]} as the 5th argument; it must be the "
|
|
f"expected BASE ref: {ln.strip()!r}"
|
|
)
|
|
|
|
|
|
@CALLERS
|
|
def test_the_caller_does_not_reimplement_the_enumeration(caller):
|
|
"""Structural rather than behavioural on purpose. Behavioural equivalence tests would still pass
|
|
if someone pasted the loop back inline and kept it correct *that day* — which is exactly how the
|
|
drift happened the first time. What must be prevented is a SECOND implementation existing.
|
|
"""
|
|
# An inline `pulls/<n>/files?` fetch is the signature of a re-inlined copy. Match on the endpoint
|
|
# alone, NOT on `?limit=` — anchoring the query string lets a copy written as
|
|
# `files?page=1&limit=50` walk straight past a guard that exists to stop exactly
|
|
# that. Still evadable by a copy that builds the URL without a literal `?`, so this narrows the
|
|
# gap rather than closing it.
|
|
assert not re.search(r"pulls/\$?\{?\w+\}?/files\?", _code_lines(caller)), (
|
|
f"{caller.relative_to(REPO_ROOT)} appears to enumerate PR files inline again — "
|
|
"that is the duplication ersatztv#649 removed"
|
|
)
|
|
|
|
|
|
# --- The ENFORCED caller's own preconditions ---------------------------------------------------
|
|
|
|
|
|
def _workflow_steps():
|
|
import yaml
|
|
|
|
wf = yaml.safe_load(WORKFLOW.read_text())
|
|
return wf["jobs"]["set-verdict-status"]["steps"]
|
|
|
|
|
|
def _classify_step():
|
|
for step in _workflow_steps():
|
|
if "review-verdict/h10" in (step.get("run") or ""):
|
|
return step
|
|
raise AssertionError("no step in review-verdict.yml posts review-verdict/h10")
|
|
|
|
|
|
def _sentinel(name: str) -> str:
|
|
"""A sentinel description, read from the SHIPPED step body rather than copied into this file.
|
|
|
|
Both sentinels are FIXED POINTS: the classification recognises its own previous output and
|
|
refuses to grant an exemption over it. A test carrying its own copy of the literal would keep
|
|
passing after the workflow's copy was reworded, while the real chain silently broke — the exact
|
|
shape of the defect the fixed-point test exists to catch. Reading it from the body binds
|
|
the two, and the count assertion means a renamed or duplicated assignment is a loud failure
|
|
rather than a wrong string.
|
|
"""
|
|
body = _classify_step()["run"]
|
|
found = re.findall(rf'^\s*{name}="([^"]*)"$', body, flags=re.MULTILINE)
|
|
assert len(found) == 1, f"expected exactly one {name} assignment in the step body, found {len(found)}"
|
|
return found[0]
|
|
|
|
|
|
REPAIR_DESC = _sentinel("REPAIR_DESC")
|
|
UNVERIFIED_DESC = _sentinel("UNVERIFIED_DESC")
|
|
|
|
|
|
def _workflow_triggers():
|
|
"""The `on:` block, tolerating YAML 1.1's `on` -> True coercion.
|
|
|
|
`yaml.safe_load` parses the bare key `on` as the BOOLEAN True, not the string "on" — so the
|
|
obvious `wf["on"]` raises KeyError against a perfectly valid workflow. Looking up both is not
|
|
defensive padding: a test that died on a KeyError here would read as "the trigger assertion is
|
|
broken" rather than "the trigger changed", which is the wrong failure to hand a maintainer.
|
|
"""
|
|
import yaml
|
|
|
|
wf = yaml.safe_load(WORKFLOW.read_text())
|
|
on = wf.get("on", wf.get(True))
|
|
assert isinstance(on, dict), f"review-verdict.yml has no parseable `on:` mapping (got {on!r})"
|
|
return on
|
|
|
|
|
|
def test_the_workflow_trigger_is_pull_request_TARGET_scoped_to_main():
|
|
"""The definition-rewrite hole (ersatztv#672) — sibling of the base-ref checkout below.
|
|
|
|
That checkout binds the SCRIPTS this job runs to the base. It cannot bind the job DEFINITION:
|
|
Gitea resolves a `pull_request` workflow definition from the PR's own head, so a PR editing
|
|
review-verdict.yml ran its own rewritten copy and could post `review-verdict/h10=success` for
|
|
itself. Branch protection does not care who posted the context, and carries
|
|
`required_approvals: 0`.
|
|
|
|
Both halves are asserted because either alone closes nothing:
|
|
|
|
- `pull_request_target` resolves the definition from the base.
|
|
- `branches: [main]` keeps "the base" from being an attacker-pushed branch. Base resolution
|
|
without it merely moves the rewrite from the head to a scratch base — and since a commit
|
|
status is repo-global per sha (ersatztv#663), a success forged there is inherited by a later
|
|
real PR into `main` with the same head.
|
|
|
|
Parsed, not substring-matched, for the reason the checkout test gives — and here the text-level
|
|
version is actively broken rather than merely weak: `pull_request` is a PREFIX of
|
|
`pull_request_target`, so `"pull_request" in text` cannot tell the safe trigger from the
|
|
vulnerable one, and `"pull_request_target" in text` stays green when a plain `pull_request`
|
|
trigger is ADDED back alongside it.
|
|
"""
|
|
on = _workflow_triggers()
|
|
# EXACT SET, not "target present and plain absent". The weaker pair of assertions stays green
|
|
# after ADDING `workflow_dispatch:` or `push:` alongside the safe trigger —
|
|
# both are ref-resolved and both get secrets, so either one restores an equivalent
|
|
# self-supplied-definition path while the test reports clean. Enumerating the two known-bad
|
|
# extra triggers would have the same hole one trigger later; pinning the whole set does not.
|
|
assert set(on) == {"pull_request_target"}, (
|
|
f"review-verdict.yml must trigger on `pull_request_target` and NOTHING else; got "
|
|
f"{sorted(map(str, on))}. Plain `pull_request` takes the workflow DEFINITION from the PR "
|
|
"head (ersatztv#672), and `push`/`workflow_dispatch` resolve it from an arbitrary ref — "
|
|
"any of them, ADDED ALONGSIDE rather than replacing, reopens the hole"
|
|
)
|
|
|
|
branches = (on["pull_request_target"] or {}).get("branches")
|
|
assert branches == ["main"], (
|
|
f"`pull_request_target.branches` is {branches!r}; it must be exactly ['main']. Base "
|
|
"resolution means the BASE branch supplies the gate, so an unfiltered trigger lets a PR "
|
|
"into an attacker-pushed base run that branch's rewritten copy (ersatztv#672)"
|
|
)
|
|
|
|
|
|
def test_no_OTHER_workflow_writes_the_review_verdict_status():
|
|
"""`review-verdict.yml` is the only workflow allowed to write `review-verdict/h10`.
|
|
|
|
The trigger test above hardens ONE file, and by construction cannot see the more general
|
|
problem: the gate is forgeable by *any* head-resolved workflow holding credentials that can POST
|
|
a commit status, not only by the gate's own definition. `docker-build.yml` is exactly that — it
|
|
triggers on `pull_request` (correctly; it builds the PR's code) and carries `ETV_STATUS_AUTH`.
|
|
A probe once confirmed those credentials could WRITE statuses, not merely read them
|
|
(ersatztv#697); a follow-up probe confirmed `REGISTRY_PASSWORD` is now scoped off
|
|
`write:repository` and the same POST returns 403 (see `ci.actions-credential-scoping`). That
|
|
closes the registry-credential route, not the general problem — `GITEA_TOKEN`, `RENOVATE_TOKEN`,
|
|
and a collaborator's own token can all still POST a status, and none of that is closed by this
|
|
test.
|
|
|
|
What this test does close is the cheap regression: a second workflow quietly starting to write
|
|
the context. It is a guard against drift, not a security boundary — a workflow can still write
|
|
the status through an indirection this scan cannot see (a variable, a script it calls). Said
|
|
plainly so nobody reads a green here as "only one writer is possible".
|
|
"""
|
|
offenders = []
|
|
for wf in _workflow_files():
|
|
if wf.name == WORKFLOW.name:
|
|
continue
|
|
# CODE lines only — `docs`-style commentary in a workflow header legitimately discusses the
|
|
# context by name, and a raw-text scan would red on the prose explaining this very rule.
|
|
if "review-verdict/h10" in _code_lines(wf):
|
|
offenders.append(wf.name)
|
|
assert not offenders, (
|
|
f"{offenders} reference `review-verdict/h10` in executable lines. Only review-verdict.yml "
|
|
"may write the gate's own status; another workflow doing so is a forgery path (see "
|
|
"ersatztv#697) or, at best, a second implementation of the gate that will drift"
|
|
)
|
|
|
|
|
|
def test_the_workflow_checks_out_the_BASE_ref_never_the_head():
|
|
"""The self-rewrite hole. This workflow judges the PR, so the PR must not supply the code that
|
|
judges it: checking out `head.sha` would let a PR rewrite `scripts/pr-changed-files.sh` to
|
|
return an empty list and exempt itself.
|
|
|
|
Asserted through a YAML parse rather than a substring search, because the dangerous edit is
|
|
subtle — `head.sha` for `base.sha` is a nine-character diff in a `with:` block — and a
|
|
text-level `"base.sha" in text` check would still pass if a second checkout step took the head
|
|
afterwards and won.
|
|
"""
|
|
checkouts = [s for s in _workflow_steps() if "actions/checkout" in (s.get("uses") or "")]
|
|
assert len(checkouts) == 1, (
|
|
f"expected exactly one checkout step, found {len(checkouts)} — a second checkout can "
|
|
"silently overwrite the base ref with the PR head"
|
|
)
|
|
with_ = checkouts[0].get("with") or {}
|
|
ref = str(with_.get("ref", ""))
|
|
assert "pull_request.base.sha" in ref, (
|
|
f"the checkout ref is {ref!r}; it must be the PR's BASE sha, so a PR cannot rewrite the gate that judges it"
|
|
)
|
|
assert "head" not in ref, f"the checkout ref {ref!r} references the PR head"
|
|
assert with_.get("persist-credentials") is False, (
|
|
"persist-credentials must be false — nothing here pushes, and a token left in .git/config "
|
|
"is handed to every script the job runs"
|
|
)
|
|
|
|
|
|
def test_the_workflow_runs_the_jq_preflight_in_FLOOR_mode_only():
|
|
"""`--expect` pins an exact jq version and fails when it drifts. That is right for the advisory
|
|
`script-tests` job and catastrophic here: this workflow writes `review-verdict/h10`, a REQUIRED
|
|
check on `main`, so a pin would turn any jq upgrade on the runner into a repo-wide merge
|
|
deadlock — a required gate failing because an upstream package manager did its job.
|
|
"""
|
|
# CODE only, for the reason `_code_lines` documents: reading the raw text goes red on the
|
|
# workflow's own comment explaining why `--expect` is banned here.
|
|
code = _code_lines(WORKFLOW)
|
|
# A bare `"jq-preflight.sh" in code` is NOT enough: the path also appears in the
|
|
# `if [ -x ./scripts/jq-preflight.sh ]` presence guard, so deleting the
|
|
# actual invocation would leave that substring behind and the assertion green. Require a line
|
|
# that INVOKES it.
|
|
steps = [s for s in _workflow_steps() if "jq-preflight.sh" in (s.get("run") or "")]
|
|
assert steps, (
|
|
"review-verdict.yml no longer runs the jq preflight, so the version its shell gates run "
|
|
"under is unobservable again (ersatztv#648)"
|
|
)
|
|
invocations = [
|
|
ln.strip() for ln in steps[0]["run"].splitlines() if re.match(r"^\s*(\./)?scripts/jq-preflight\.sh(\s|$)", ln)
|
|
]
|
|
assert invocations, "the jq preflight is referenced but never actually invoked"
|
|
assert "--expect" not in code, (
|
|
"review-verdict.yml must run jq-preflight.sh in floor-only mode; --expect here deadlocks "
|
|
"every merge on `main` the day the runner's jq changes"
|
|
)
|
|
assert all("--expect" not in ln for ln in invocations)
|
|
|
|
|
|
# --- The ENFORCED caller's contract, EXECUTED ---------------------------------------------------
|
|
#
|
|
# The structural guards above prove the workflow *calls* the shared script. They cannot prove it
|
|
# reacts correctly when the script FAILS — and that is precisely the mutation that survived the last
|
|
# round on the hook side: making the caller ignore the exit status left the entire suite green,
|
|
# because every failure path also happened to produce empty stdout. So the stub below FAILS while
|
|
# emitting a perfectly docs-only list, the one combination that redundancy cannot absorb.
|
|
#
|
|
# The step's `run:` block is extracted from the YAML and executed directly. That is a real
|
|
# behavioural test of the shipped text — not a paraphrase of it — at the cost of not exercising the
|
|
# runner's step wiring, which no local test can reach anyway.
|
|
|
|
WORKFLOW_STUB_CURL = r"""#!/usr/bin/env python3
|
|
import json, os, pathlib, sys
|
|
|
|
args = sys.argv[1:]
|
|
url = [a for a in args if a.startswith("http")][-1]
|
|
out = pathlib.Path(os.environ["STUB_DIR"])
|
|
|
|
if "-X" in args and args[args.index("-X") + 1] == "POST":
|
|
if os.environ.get("STUB_POST_FAILS") == "after-first":
|
|
# THE FIRST POST SUCCEEDS AND EVERY LATER ONE FAILS. That is the only arrangement that
|
|
# reaches a REPAIR write's failure handling: with every POST failing, the job dies on the
|
|
# first one and never gets there, so the two behaviours are indistinguishable.
|
|
ctr = out / "post_attempts.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
sys.exit(22)
|
|
if os.environ.get("STUB_POST_FAILS") == "first":
|
|
# ONLY THE FIRST ATTEMPT, so a writer that retries succeeds and one that does not fails.
|
|
# Against a stub that fails EVERY attempt the two are indistinguishable, which is how a
|
|
# retry loop ends up shipped and unexercised.
|
|
ctr = out / "post_attempts.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen == 0:
|
|
sys.exit(22)
|
|
if os.environ.get("STUB_POST_FAILS") == "1":
|
|
# EVERY POST FAILS, retries included. `gh` is `curl -sf`, so an HTTP error is exit 22 with
|
|
# empty stdout. Without this the write helpers' failure arms are unreachable — and one of
|
|
# them reported SUCCESS after both attempts failed, which is what let a caller exit 0
|
|
# believing the head had been marked.
|
|
sys.exit(22)
|
|
payload = args[args.index("-d") + 1]
|
|
(out / "posted.json").write_text(payload)
|
|
(out / "posted_url.txt").write_text(url)
|
|
# EVERY post is also appended (ersatztv#706 race 2): the repair path POSTs a second time, and a
|
|
# test that only ever saw the last write could not tell "posted success, then repaired it to
|
|
# pending" from "posted pending once".
|
|
#
|
|
# Re-serialized COMPACT rather than appended verbatim: `jq -n` pretty-prints by default, so the
|
|
# payloads arrive spanning several lines and a raw append would produce a file that is not
|
|
# JSONL at all.
|
|
with (out / "posted_all.jsonl").open("a") as fh:
|
|
fh.write(json.dumps(json.loads(payload)) + "\n")
|
|
print("{}")
|
|
sys.exit(0)
|
|
|
|
# The PR timeline, which the retarget fence counts (ersatztv#706 race 1). Checked BEFORE the status
|
|
# branch: the fence's URL does not contain "/status", but keeping the order explicit means a later
|
|
# edit to either pattern cannot silently reroute one endpoint into the other's handler.
|
|
#
|
|
# Real shapes, measured on this instance and deliberately mirrored, because the job's guards are
|
|
# type-sensitive: a NON-EMPTY page is a bare array of events, each with a `type`, and a retarget is
|
|
# `change_target_branch` (confirmed on PR #703, the route-1 reproduction, which carries exactly two;
|
|
# and on PR #717, never retargeted, which carries none). A page PAST THE END is the JSON value `null`
|
|
# — NOT an empty array (measured again at 1.27.1 on PR #752, ersatztv#751). A terminator printing
|
|
# `[]` instead leaves the fence's type gate unexercised, which is how it shipped rejecting every
|
|
# real timeline.
|
|
if "/timeline" in url:
|
|
mode = os.environ.get("STUB_TIMELINE_MODE", "none")
|
|
page = 1
|
|
for part in url.split("?", 1)[-1].split("&"):
|
|
if part.startswith("page="):
|
|
page = int(part.split("=", 1)[1])
|
|
flaky = os.environ.get("STUB_TIMELINE_FLAKY_FIRST", "")
|
|
if flaky:
|
|
# ONE TRANSIENT BLIP, on the very first timeline request of the run (ersatztv#870 review).
|
|
# This is the only arrangement that separates a walk that retries from one that does not:
|
|
# `transport-error` and `unreadable` fail EVERY attempt, so a retry changes nothing and both
|
|
# implementations refuse identically. Counting REQUESTS, not pages, because the retry is
|
|
# per-request — `timeline_reads.txt` bumps once per walk and cannot see a second attempt.
|
|
#
|
|
# TWO BLIP SHAPES, because they reach DIFFERENT arms of the retry and only one was pinned.
|
|
# `transport` exits non-zero with no body, so `raw` is empty and `kind` is never computed.
|
|
# `body` returns 200 with a well-formed JSON OBJECT — which is what this Gitea really sends
|
|
# on an error, measured: `?since=NOTATIME` returns an object, not an array — so `jq -r type`
|
|
# SUCCEEDS and sets `kind=object`.
|
|
#
|
|
# That difference is the whole point of the second shape. An unreadable body (a 502 HTML
|
|
# page) would NOT do: jq fails on it, `|| kind=""` fires, and the walk takes the same path
|
|
# as `transport`. Only a parseable non-array body distinguishes "break out of the retry on
|
|
# anything readable" from "break only on a shape the `case` below accepts", and that
|
|
# narrowing is a real fail-open — an error object would be read as the walk's terminator
|
|
# verdict instead of being retried. With the 502 page as the fixture the mutant passes.
|
|
rctr = out / "timeline_requests.txt"
|
|
rseen = int(rctr.read_text()) if rctr.exists() else 0
|
|
rctr.write_text(str(rseen + 1))
|
|
if rseen == 0:
|
|
if flaky == "body":
|
|
print(json.dumps({"message": "internal server error", "url": "http://gitea"}))
|
|
sys.exit(0)
|
|
sys.exit(22)
|
|
if mode == "empty-first-page":
|
|
# A terminator on PAGE 1 — no real page ever read (ersatztv#751). The job must NOT certify a
|
|
# zero retarget count from this, so the exemption is withheld.
|
|
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
|
|
sys.exit(0)
|
|
if mode == "unreadable":
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if mode == "unreadable-after-post":
|
|
# THE THIRD WALK ONLY (ersatztv#849 route 3). The job walks this endpoint three times on the
|
|
# exemption path — before classifying, at the pre-POST fence, and again after the POST — and
|
|
# this mode has to leave the first two trusted or the pre-POST fence refuses and there is no
|
|
# POST to re-check.
|
|
#
|
|
# THE THRESHOLD IS DERIVED FROM WHERE THE COUNTER MOVES, not picked. `timeline_reads.txt` is
|
|
# written on the LAST-REAL-PAGE branch below, i.e. once per walk at page 1, so the value READ
|
|
# at the top of this handler runs 0,1 (walk 1), 1,2 (walk 2), 2,3 (walk 3). `>= 3` is
|
|
# therefore first true at walk 3's terminator request and at no earlier one. `>= 2` — which is
|
|
# what `trusted-then-unreadable` uses for its own two-walk purpose — would break walk 2.
|
|
ctr0 = out / "timeline_reads.txt"
|
|
seen0 = int(ctr0.read_text()) if ctr0.exists() else 0
|
|
if seen0 >= 3:
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if mode == "trusted-then-unreadable":
|
|
# The FIRST count succeeds, the SECOND cannot be established. `timeline_reads.txt` is
|
|
# incremented only on the LAST-REAL-PAGE branch below, after the `page > pages` and
|
|
# `page < pages` early exits — so it counts REAL PAGES SERVED, not requests: it holds 1
|
|
# after the before-count, and `>= 2` first becomes true partway through the after-count.
|
|
#
|
|
# This is the only arrangement that separates the head arm from its trust guard: with both
|
|
# counts trusted the guard is a no-op, and with both untrusted `pushes_before` is 0 too, so
|
|
# the arm cannot fire either way. Only trusted-then-untrusted makes the two disagree.
|
|
ctr0 = out / "timeline_reads.txt"
|
|
seen0 = int(ctr0.read_text()) if ctr0.exists() else 0
|
|
if seen0 >= 2:
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if mode == "untyped-rows":
|
|
# Well-formed JSON, well-formed array, rows the walk cannot classify: no `.type` at all on
|
|
# one row and a non-string one on another. Both are invisible to a `select(.type == ...)`.
|
|
#
|
|
# THIS MODE MUST STILL TERMINATE, and getting that wrong made the test measure nothing.
|
|
# Serving these rows on EVERY page ran the walk into its 20-page cap, which withholds the
|
|
# exemption on its own — so the test passed identically with the row validation deleted.
|
|
# Page 2 therefore terminates normally: without the guard the walk completes and certifies a
|
|
# zero count (exemption GRANTED), with it the page is refused (exemption withheld). Only
|
|
# then does the assertion discriminate.
|
|
if page > 1:
|
|
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
|
|
else:
|
|
print(json.dumps([{"id": 1}, {"id": 2, "type": 7}, {"id": 3, "type": "comment"}]))
|
|
sys.exit(0)
|
|
if mode == "transport-error":
|
|
sys.exit(22)
|
|
# MULTI-PAGE TIMELINES (ersatztv#803 review). Until this existed the double always terminated
|
|
# after page 1 and put every event on it, so nothing in the suite exercised the walk ACCUMULATING
|
|
# across pages — a walk that read page 1 and stopped passed every fence test. `STUB_TIMELINE_PAGES`
|
|
# says how many REAL pages there are; page P+1 is the terminator. Events go on the LAST real page,
|
|
# which is where the server would put them: timeline rows are ordered ASCENDING, so the newest
|
|
# events — the ones a fence is looking for — are the furthest from page 1. Earlier pages carry
|
|
# filler `comment` rows so they are full-length and cannot be mistaken for the end.
|
|
#
|
|
# PAST THE LAST REAL PAGE THE DEFAULT IS `null`, NOT `[]`, because that is what this endpoint
|
|
# really returns past the end — measured at Gitea 1.27.1 (ersatztv#751). This stub printed `[]`
|
|
# for two rounds of #706 work while its comment claimed it mirrored measured reality, so every
|
|
# fence test was green against a shape the server never produces, and the fence's `array`-only
|
|
# type gate — which reads `null` as unreadable — was never exercised. On the real instance that
|
|
# made `rt_ok` false for EVERY pr, so the fence withheld every exemption. Parameterised rather
|
|
# than simply corrected: `/issues/{n}/comments` really does return `[]` when empty, so both
|
|
# shapes are live on this server and the job must accept either.
|
|
pages = int(os.environ.get("STUB_TIMELINE_PAGES", "1"))
|
|
# A FILTERED INTERMEDIATE PAGE (ersatztv#870). This is the shape the double could not produce
|
|
# before, and it is not "a page that happens to be empty": the endpoint pages at the DATABASE
|
|
# level and filters AFTERWARDS, so a page whose 50 rows are all `CommentTypeCode` serializes as
|
|
# the SAME `null` a page past the end does, while LATER pages still hold events. Serving it from
|
|
# the terminator env var rather than a literal is deliberate — the two shapes are
|
|
# indistinguishable to the walk on the wire, which is the entire defect, so a double that made
|
|
# them distinguishable would test a server we do not have.
|
|
#
|
|
# It exits BEFORE the last-real-page branch, so it does not touch `timeline_reads.txt`. Be exact
|
|
# about what that counter is, because the comment above it is not: it increments ONLY on the
|
|
# last-real-page branch, so it counts WALKS (one bump each, at the walk's last real page), not
|
|
# pages served. A filtered page must not bump it either way, or the `moves:`/`stable:` sequences
|
|
# would advance a walk early and address the wrong element.
|
|
filtered = {
|
|
int(x) for x in os.environ.get("STUB_TIMELINE_FILTERED_PAGES", "").split(",") if x.strip()
|
|
}
|
|
if page in filtered:
|
|
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
|
|
sys.exit(0)
|
|
if page > pages:
|
|
print(os.environ.get("STUB_TIMELINE_TERMINATOR", "null"))
|
|
sys.exit(0)
|
|
if page < pages:
|
|
print(json.dumps([{"id": 100 + i, "type": "comment"} for i in range(50)]))
|
|
sys.exit(0)
|
|
|
|
# ONE VALUE PER WALK, CLAMPED — not a fixed before/after pair. The job walks this endpoint a
|
|
# THIRD time after the POST on the exemption path (ersatztv#849 route 3), so a two-element knob
|
|
# can no longer address every count the job takes. "moves:A,B" keeps its old meaning exactly
|
|
# (walk 3 clamps to B, and the two-walk fixtures that use it abstain before walk 3 anyway);
|
|
# "moves:A,B,C" reaches the post-POST walk, which is the arrangement that separates "retargeted
|
|
# while classifying" — caught by the pre-POST fence — from "retargeted after the write landed".
|
|
n_seq = [0]
|
|
if mode.startswith("stable:"):
|
|
n_seq = [int(mode.split(":", 1)[1])]
|
|
elif mode.startswith("moves:"):
|
|
n_seq = [int(x) for x in mode.split(":", 1)[1].split(",")]
|
|
# THE PUSH AXIS IS INDEPENDENT OF THE RETARGET AXIS (ersatztv#803/#664), and it has to be, or the
|
|
# head-fence tests could not distinguish which fence fired. Both counts come off the SAME page —
|
|
# the job makes one walk and tallies two `.type` values — so the stub serves them from one
|
|
# response, but the two `moves:`/`stable:` knobs are separate.
|
|
pmode = os.environ.get("STUB_PUSH_MODE", "none")
|
|
p_seq = [0]
|
|
if pmode.startswith("stable:"):
|
|
p_seq = [int(pmode.split(":", 1)[1])]
|
|
elif pmode.startswith("moves:"):
|
|
p_seq = [int(x) for x in pmode.split(":", 1)[1].split(",")]
|
|
ctr = out / "timeline_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
n = n_seq[min(seen, len(n_seq) - 1)]
|
|
np = p_seq[min(seen, len(p_seq) - 1)]
|
|
# `is_force_push` is emitted because the real endpoint emits it (measured on PR #761), NOT
|
|
# because the job reads it — the fence counts rows by `.type` and never parses the body. A stub
|
|
# that omitted it would leave a reader thinking the field is unused by contract rather than by
|
|
# choice; one that made the count DEPEND on it would test a job we did not write.
|
|
#
|
|
# IT ALTERNATES, STARTING FALSE. Every row carried `true` until 2026-08-28, which made the
|
|
# documented invariant untestable: narrowing the tally to
|
|
# `select(.type == "pull_push" and (.body | fromjson | .is_force_push))` passed the ENTIRE fence
|
|
# suite. That narrowing is wrong precisely where it matters — an ordinary push also invalidates a
|
|
# mid-flight enumeration, and an ABA's restoring push can be non-forced when H1 is an ancestor —
|
|
# so the double has to contain at least one row the narrowed predicate would drop.
|
|
print(json.dumps([{"id": 1000 + i, "type": "change_target_branch",
|
|
"old_ref": "main", "new_ref": "scratch"} for i in range(n)]
|
|
+ [{"id": 1500 + i, "type": "pull_push",
|
|
"body": json.dumps({"is_force_push": bool(i % 2),
|
|
"commit_ids": ["a" * 40, "b" * 40]})}
|
|
for i in range(np)]
|
|
+ [{"id": 900, "type": "comment"}]))
|
|
sys.exit(0)
|
|
|
|
# The per-POST status HISTORY (`/statuses/{sha}`), read by the race-2 post-write verification. It is
|
|
# a BARE ARRAY, one row per POST — a different shape AND a different meaning from the combined
|
|
# `/commits/{sha}/status` below, which returns an object carrying the LATEST row per context. Both
|
|
# shapes were measured on the same live head (24 rows vs 12). Modelling them identically would have
|
|
# made the post-write check untestable, since the thing it looks for is precisely a row the combined
|
|
# endpoint no longer shows.
|
|
if "/statuses/" in url:
|
|
mode = os.environ.get("STUB_HISTORY_MODE", "none")
|
|
|
|
# PAGES FAITHFULLY (ersatztv#763). The job walks this endpoint to a validated empty page instead
|
|
# of reading one clamped page, so a stub describing page 1 only no longer describes anything the
|
|
# job does. The snapshot is a LIST OF PAGES, not a flat list, because two modes below need a page
|
|
# boundary the uniform 50-row slicing cannot express.
|
|
#
|
|
# 1. PAGE SIZE AND TERMINATOR match the real endpoint. Measured at 1.27.1 on 2026-08-28 against
|
|
# PR #761's 114-row head: pages 1 and 2 return 50, page 3 returns 14, page 4 is `[]`. So the
|
|
# slice width is 50 and past the end is `[]` — NOT `null`, which is what
|
|
# `/issues/{n}/timeline` returns and what `/commits/{sha}/status` spells `{"statuses": null}`.
|
|
# Three distinct empty shapes on one server; each stub owes its own measurement.
|
|
#
|
|
# 2. ONE LOGICAL READ IS ONE SNAPSHOT. The counter modes resolve their rows from how many times
|
|
# the job has LOOKED, and the job now looks two or three times per logical read. Recomputing
|
|
# on page 2 from an advanced counter would describe a different history than page 1 and
|
|
# surface `human-after-post`'s verdict halfway through the PRE-write read, inverting the race
|
|
# being modelled. So the pages are resolved once, on page 1, and cached.
|
|
#
|
|
# This is a MODELLING CHOICE, not a fidelity claim, and the difference matters. These are
|
|
# independent offset-paginated GETs with no snapshot token, so the REAL history can change
|
|
# between page requests — overlapping rows and shifting offsets are all reachable live. The
|
|
# cache deliberately suppresses that, because the tests here are about the job's paging
|
|
# logic, not about mid-walk mutation. Mid-walk mutation is untested, and saying so is the
|
|
# honest form of the claim.
|
|
#
|
|
# DELIBERATELY ORDERING-BLIND, and this bounds what the paging tests prove. The real endpoint
|
|
# serves `created_unix DESC`, so a row created DURING the write window is among the NEWEST and
|
|
# lands on page 1 — a genuinely raced verdict would not sit on page 2 at all. These pages are
|
|
# served in insertion order regardless, so a fixture can place a row beyond page 1 and observe
|
|
# whether the walk reaches it.
|
|
#
|
|
# So `verdict-on-page-2` and `verdict-after-short-page` prove WALK COMPLETENESS — that the job
|
|
# reads past page 1, and past a short page, to a validated empty one. They do NOT prove that a
|
|
# real raced verdict would otherwise be missed; under the server's ordering it would not be. That
|
|
# is the honest scope, and the workflow comment says the same thing: the walk exists so the one
|
|
# fail-toward-SUCCESS path stops depending on an undocumented ordering, not because the ordering
|
|
# is currently wrong.
|
|
#
|
|
# Counter branches are confined to the page-1 arm below, so a later page can never advance them.
|
|
HIST_PAGE_SIZE = 50
|
|
# ORDER-FAITHFUL MODES HONOUR `sort`. Most modes here are ordering-blind on purpose (see above):
|
|
# they test walk COMPLETENESS, which is an order-independent property, and serving insertion order
|
|
# lets a fixture place a row beyond page 1. But the partial-mark fallback's safety is a claim ABOUT
|
|
# the ordering, so the mode that pins it serves the real arrangement — DESC by default, ASC when
|
|
# the request asks for `sort=highestindex`. That is what lets re-adding the withdrawn parameter be
|
|
# caught by BEHAVIOUR rather than only by a structural assertion on the request line.
|
|
want_asc = "sort=highestindex" in url
|
|
order_faithful = False
|
|
hist_page = 1
|
|
for part in url.split("?", 1)[-1].split("&"):
|
|
if part.startswith("page="):
|
|
try:
|
|
hist_page = int(part.split("=", 1)[1])
|
|
except ValueError:
|
|
hist_page = 1
|
|
|
|
ordinary = [{"id": 7000 + i, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"} for i in range(60)]
|
|
raced_row = {"id": 9000, "context": "review-verdict/h10", "status": "failure",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}
|
|
|
|
snapshot = out / "history_snapshot.json"
|
|
logical = out / "history_logical_reads.txt"
|
|
if hist_page == 1:
|
|
n_logical = (int(logical.read_text()) if logical.exists() else 0) + 1
|
|
logical.write_text(str(n_logical))
|
|
rows = []
|
|
pages = None
|
|
if mode.startswith("human-after-post"):
|
|
# The raced verdict: absent when the high-water mark is taken, present afterwards. Its id
|
|
# is ABOVE the mark, which is what makes it detectable.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
# Configurable creator (ersatztv#742). The RACED test must stay broader than the
|
|
# INHERITANCE test: a verdict from an account off `$H10_REVIEWERS` still counts as
|
|
# "something human landed while we were writing", because narrowing it here would
|
|
# leave the exemption green over that row instead of repairing to pending.
|
|
rows = [dict(raced_row, id=5000,
|
|
creator={"login": os.environ.get("STUB_HISTORY_CREATOR", "timothy")})]
|
|
elif mode == "sentinel-after-post":
|
|
# Another overlapping run repaired mid-flight: its SENTINEL lands above this run's mark,
|
|
# while the human row it records sits BELOW the mark and is therefore invisible here.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
rows = [{"id": 6000, "context": "review-verdict/h10", "status": "pending",
|
|
"creator": None,
|
|
"description": "Human verdict raced this exemption write — re-post the verdict"}]
|
|
elif mode == "stale-human-already-present":
|
|
# A human verdict that was ALREADY in the history before this run — e.g. one whose
|
|
# recorded base did not match, which `read_existing_verdict` deliberately declines to
|
|
# honour. It must NOT be mistaken for a raced write, or every later run of that PR would
|
|
# repair its own exemption to pending forever.
|
|
rows = [{"id": 10, "context": "review-verdict/h10", "status": "success",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: other)"}]
|
|
elif mode in ("second-page", "verdict-on-page-2", "premark-page2-error",
|
|
"flaky-page2", "premark-page1-error"):
|
|
# A history that genuinely runs past one page: 60 ORDINARY rows, no verdict and no
|
|
# sentinel. Under the pre-#763 page-2 probe the mere existence of these rows forced a
|
|
# repair; now they are simply read, and `second-page` asserts the exemption STANDS.
|
|
rows = list(ordinary)
|
|
if mode in ("verdict-on-page-2", "premark-page2-error"):
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
# THE RACED VERDICT, PLACED BEYOND THE FIRST PAGE. Its id is above every ordinary
|
|
# row, so it is above the high-water mark too. A job that reads only page 1 cannot
|
|
# see it — the fail-toward-SUCCESS hole #763 closes.
|
|
rows.insert(55, dict(raced_row))
|
|
elif mode == "premark-page2-error-desc":
|
|
# THE PARTIAL-MARK FALLBACK, MEASURED UNDER THE REAL ORDERING. 60 rows, one of them a
|
|
# PRE-EXISTING base-mismatched human verdict at id 7055 — genuinely older than the newest
|
|
# row and therefore not something that raced this write.
|
|
#
|
|
# Under the server default (DESC) page 1 carries 7059..7010, so a walk that fails on page 2
|
|
# still saw the true maximum: the mark is 7059, the verdict at 7055 is below it, and the
|
|
# exemption correctly STANDS. Under ASC page 1 carries 7000..7049, the salvaged mark is
|
|
# 7049, and that same pre-existing verdict tests as NEWER than the mark — a spurious sticky
|
|
# repair on a head nothing raced, which is the #761 failure. This fixture is what makes
|
|
# that difference observable.
|
|
order_faithful = True
|
|
rows = [{"id": 7000 + i, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"}
|
|
for i in range(60) if 7000 + i != 7055]
|
|
rows.append({"id": 7055, "context": "review-verdict/h10", "status": "success",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: other)"})
|
|
rows.sort(key=lambda r: r["id"])
|
|
elif mode == "null-page1-after-post":
|
|
# The post-write read answers "no statuses exist" for a sha this job has just POSTed to.
|
|
# That is impossible — the endpoint returns one row per POST — but it is well-formed, so
|
|
# nothing retries it and the walk reports SUCCESS over it.
|
|
#
|
|
# NO RACED VERDICT IS PLANTED, deliberately. Appending one to dramatise the stakes is
|
|
# inert twice over: unconditionally it also joins the PRE-write
|
|
# read, which sets the mark to its OWN id — the strict `> $since` then excludes it — and
|
|
# gated to the post-write read it is never served at all, because page 1 answers `null`
|
|
# before any row reaches the wire. A row the test
|
|
# cannot observe is decoration that reads as coverage. What is actually under test is
|
|
# narrower and sufficient: a response asserting an empty history for a sha this job wrote
|
|
# to must not be accepted as proof that nothing raced.
|
|
rows = [{"id": 7000 + i, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"} for i in range(3)]
|
|
elif mode == "null-terminator":
|
|
# This endpoint returns `[]` past the end today. `null` is what `/issues/{n}/timeline`
|
|
# returns, and an array-only gate on THAT endpoint was ersatztv#751 — the walk never
|
|
# reached a validated empty page and every exemption was withheld. Same shape, second
|
|
# endpoint, worse blast radius (the sticky sentinel), so `null` is tolerated and this
|
|
# fixture proves it rather than asserting it.
|
|
rows = [{"id": 7000 + i, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"} for i in range(60)]
|
|
elif mode == "string-id-preexisting-row":
|
|
# THE TWIN of `string-id-inflates-mark`, on the COMPARISON rather than the maximum. A
|
|
# PRE-EXISTING row (a base-mismatched verdict `read_existing_verdict` declines to honour)
|
|
# carries a string id. jq orders strings above every number, so `.id > $since` reads it as
|
|
# newer than any mark and counts it as having raced this write — a sticky sentinel and a
|
|
# false "was overwritten" on every later run of that PR, forever.
|
|
rows = [{"id": 9999, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"},
|
|
{"id": "3", "context": "review-verdict/h10", "status": "success",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: other)"}]
|
|
elif mode == "no-numeric-ids":
|
|
# A NON-EMPTY history in which no row carries a numeric id. Distinct from an EMPTY history,
|
|
# whose mark of 0 is correct; collapsing this case to 0 too would be a schema we cannot
|
|
# read reported as one we can.
|
|
rows = [{"id": "abc", "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"}]
|
|
elif mode == "over-cap":
|
|
# Longer than the 20-page cap (1050 rows), so the walk can never reach a validated empty
|
|
# page. That is "could not establish", not exhaustion.
|
|
rows = [{"id": 20000 + i, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"} for i in range(1050)]
|
|
elif mode == "string-id-inflates-mark":
|
|
# A STRING ID. jq orders strings above every number, so `max` over raw ids returns
|
|
# `"99999"` — which then passes the numeric gate as a plain `99999` and sets a high-water
|
|
# mark far above anything real. Every later row looks OLDER than the mark, so the raced
|
|
# verdict below is invisible and the exemption stands over it. One corrupt row is enough.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
rows = [{"id": "99999", "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"},
|
|
{"id": 10, "context": "ci/other", "status": "success",
|
|
"creator": None, "description": "unrelated"}]
|
|
if seen > 0:
|
|
rows.append(dict(raced_row))
|
|
elif mode == "malformed-creator-beside-verdict":
|
|
# A SCHEMA-CORRUPT ROW NEXT TO A REAL ONE. `creator` is a number, so `.creator.login`
|
|
# hard-errors in jq ("Cannot index number with string"), jq exits 5, and an unguarded
|
|
# `raced=$(...)` takes the whole step down under `set -e` — after the exemption `success`
|
|
# is posted and with the repair never attempted. The genuine verdict beside it is what
|
|
# makes the consequence visible: the correct behaviour is to drop the malformed row and
|
|
# still repair.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
rows = [{"id": 8500, "context": "review-verdict/h10", "status": "failure",
|
|
"creator": 7, "description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"},
|
|
dict(raced_row)]
|
|
elif mode == "malformed-description-beside-verdict":
|
|
# THE TWIN OF `malformed-creator-beside-verdict`, on the other field the filter reads.
|
|
# `description` is a NUMBER, which `(.description // "") | startswith(...)` does NOT
|
|
# protect against — `//` replaces only `null` and `false` — so `startswith` hard-errors on
|
|
# it and jq exits 5. The genuine verdict beside it is what makes the consequence visible:
|
|
# the whole count dies, so a real raced verdict is reported as uncertainty instead.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen > 0:
|
|
rows = [{"id": 8600, "context": "review-verdict/h10", "status": "failure",
|
|
"creator": {"login": "timothy"}, "description": 7},
|
|
dict(raced_row)]
|
|
elif mode == "verdict-after-short-page":
|
|
# DELIBERATELY UNFAITHFUL, and that is the point. Page 2 is SHORT (10 rows) and yet page 3
|
|
# still carries rows — a shape the measured server does not produce, but exactly what a
|
|
# truncated or partially-served response looks like. It is the only way to observe the
|
|
# rule "terminate ONLY on a validated EMPTY page, never on a short one": against a
|
|
# faithful double a short page is always the last one, so an implementation that stops
|
|
# there is indistinguishable from a correct one.
|
|
ctr = out / "history_reads.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
pages = [ordinary[:50], ordinary[50:], [dict(raced_row)] if seen > 0 else []]
|
|
# SEEDED ROWS (ersatztv#849). The reconciliation may only clear the sentinel over a
|
|
# history that CONTAINS it, which is the one shape the mode-driven fixtures above cannot
|
|
# express: they describe a head as it was BEFORE this job ever wrote to it, and the sentinel
|
|
# is by definition a row an EARLIER run already POSTed. Seeding is how a fixture says "this
|
|
# head has been written to before", which is exactly the precondition `ex_unverified` states.
|
|
extra = os.environ.get("STUB_HISTORY_EXTRA", "")
|
|
if extra:
|
|
rows = list(rows) + json.loads(extra)
|
|
if order_faithful and not want_asc:
|
|
rows = list(reversed(rows))
|
|
if pages is None:
|
|
pages = [rows[i:i + HIST_PAGE_SIZE] for i in range(0, len(rows), HIST_PAGE_SIZE)] or [[]]
|
|
|
|
# THE JOB'S OWN WRITES APPEAR IN THE HISTORY. `/statuses/{sha}` returns one row per POST, so
|
|
# once this job has posted its exemption the next read MUST show it. The stub did not model
|
|
# that at all — the history was whatever the mode described, before and after the write alike,
|
|
# so every ordinary run looked like a head nothing had ever been posted to. `creator: null` is
|
|
# measured (an Actions-token POST records no creator), which is also what keeps our own row
|
|
# out of the raced count; the id is one above everything present, mirroring the real
|
|
# endpoint's per-head monotonic ids (verified live: 114 rows, ids strictly increasing with
|
|
# `created_at`, no duplicates).
|
|
posted_log = out / "posted_all.jsonl"
|
|
if posted_log.exists():
|
|
for line in posted_log.read_text().splitlines():
|
|
if not line.strip():
|
|
continue
|
|
body = json.loads(line)
|
|
# NUMERIC IDS ONLY when computing the next one. A fixture may deliberately carry a
|
|
# schema-corrupt id (see `string-id-inflates-mark`), and `max()` over a str beside an
|
|
# int raises TypeError — which fails the whole stub request, makes the walk look
|
|
# unreadable, and produces a repair for a reason the test was not asking about. That
|
|
# is a test passing for the wrong reason: the string-id mutation stayed green because
|
|
# the double crashed rather than because the mark was computed correctly.
|
|
nxt = max([r.get("id") for pg in pages for r in pg
|
|
if isinstance(r.get("id"), int)] or [0]) + 1
|
|
own = {"id": nxt, "context": body.get("context"),
|
|
"status": body.get("state"), "creator": None,
|
|
"description": body.get("description", "")}
|
|
# WHERE the newest row goes depends on the served ordering. Under DESC it is FIRST;
|
|
# appending it last would put the job's own POST on the OLDEST page, the arrangement
|
|
# the order-faithful fixture exists to rule out. Ordering-blind modes keep insertion
|
|
# order, which is what lets them place a row beyond page 1.
|
|
if order_faithful and not want_asc:
|
|
pages[0].insert(0, own)
|
|
else:
|
|
pages[-1].append(own)
|
|
snapshot.write_text(json.dumps(pages))
|
|
else:
|
|
n_logical = int(logical.read_text()) if logical.exists() else 1
|
|
pages = json.loads(snapshot.read_text()) if snapshot.exists() else [[]]
|
|
|
|
if hist_page > 1 and mode == "reconcile-page2-error" and n_logical == 1:
|
|
# PAGE 2 FAILS ON THE FIRST LOGICAL READ — the reconciliation walk — while page 1, which
|
|
# carries the seeded sentinel, is served. That separates the two halves of the trust
|
|
# condition: `ph_ok` is `no` and `witness` is 1, so a mutant that drops only the completeness
|
|
# operand still clears, and a mutant that drops only the witness operand does not. Against a
|
|
# fixture where BOTH are false, `if false` disarms two guards at once and isolates neither.
|
|
sys.exit(22)
|
|
if hist_page == 1 and mode == "postwrite-page1-error":
|
|
# FAILS THE POST-WRITE WALK ONLY (ersatztv#849). `premark-page1-error` fails the
|
|
# FIRST logical read, which is the mark; the repair floor needs a run that got its mark, made
|
|
# its write, and THEN could not read the history back. Both attempts of the second logical
|
|
# read fail, so the retry cannot rescue it.
|
|
# THE THRESHOLD IS DERIVED FROM WHERE THE REQUESTS FALL, not picked: `page_statuses` asks
|
|
# for page 1 exactly once per successful walk, so the mark's walk is request 0 and the
|
|
# post-write walk is requests 1 and 2 (the second being its retry). `>= 1` therefore lets the
|
|
# mark be established and fails the post-write read outright, which is the arrangement the
|
|
# repair floor needs and the one `premark-page1-error` cannot produce.
|
|
ctr = out / "p1_attempts_pw.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen >= 1:
|
|
sys.exit(22)
|
|
if hist_page == 1 and mode == "premark-page1-error":
|
|
# PAGE 1 ITSELF FAILS, for the whole first logical read (both attempts), so the walk returns
|
|
# nothing at all and the mark must be abandoned. Counted in its own file because the logical
|
|
# read counter below counts page-1 REQUESTS, retries included — not logical reads.
|
|
ctr = out / "p1_attempts.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen < 2:
|
|
sys.exit(22)
|
|
if hist_page > 1 and mode == "flaky-page2":
|
|
# TRANSIENT: fails the first attempt of each logical read and succeeds on the retry. This is
|
|
# the only fixture that exercises the retry at all — the other error modes fail every attempt,
|
|
# so against them a one-shot walk and a retrying walk are indistinguishable.
|
|
ctr = out / "page2_attempts.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen % 2 == 0:
|
|
sys.exit(22)
|
|
if hist_page > 1 and mode == "second-page-garbage":
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if hist_page > 1 and mode == "second-page-error":
|
|
sys.exit(22)
|
|
if hist_page == 1 and mode == "null-page1-after-post" and n_logical >= 2:
|
|
print("null")
|
|
sys.exit(0)
|
|
if hist_page > 1 and mode == "null-terminator" and hist_page > len(pages):
|
|
print("null")
|
|
sys.exit(0)
|
|
if hist_page > 1 and mode in ("premark-page2-error", "premark-page2-error-desc") and n_logical == 1:
|
|
# Fails ONLY on the PRE-write read, so the high-water mark must be salvaged from the partial
|
|
# list; the post-write read then pages cleanly and must still catch the raced verdict.
|
|
# `n_logical` is incremented on every page-1 REQUEST, including retried ones — which is fine
|
|
# here because page 1 never fails in this mode, but it counts requests, not logical reads.
|
|
sys.exit(22)
|
|
|
|
print(json.dumps(pages[hist_page - 1] if hist_page - 1 < len(pages) else []))
|
|
sys.exit(0)
|
|
|
|
if "/status" in url:
|
|
# Configurable. Hardcoding "no verdict yet" left the ersatztv#647 emptiness guard and the
|
|
# never-overwrite short-circuit unreachable: neither could be made to fire, so mutations
|
|
# deleting them survived the whole suite.
|
|
mode = os.environ.get("STUB_STATUS_MODE", "none")
|
|
|
|
# PAGE, honoured — the job now reads page 2 to justify "no verdict exists" (ersatztv#751). The
|
|
# real endpoint pages properly: measured at 1.27.1 on a 6-context head, `?limit=3&page=2` returns
|
|
# 3 more rows and `page=9` returns the `statuses: null` terminator. Every mode below describes
|
|
# page 1 only, so page 2+ must terminate, or the job would read its own page-1 rows again and
|
|
# conclude the list is longer than it is. `twopage` is the one mode with a real second page.
|
|
status_page = 1
|
|
for part in url.split("?", 1)[-1].split("&"):
|
|
if part.startswith("page="):
|
|
try:
|
|
status_page = int(part.split("=", 1)[1])
|
|
except ValueError:
|
|
status_page = 1
|
|
|
|
# THE EMPTY SHAPE IS `statuses: null`, NOT `[]` (ersatztv#751). Measured on this instance: a head
|
|
# with no statuses returns `{"state":"pending","total_count":0,"statuses":null}` — PR #739's head
|
|
# 5fa672e2. This stub printed `{"statuses": []}` at all three no-verdict sites, which is a shape
|
|
# the server does not produce for that case, so the job's `.statuses | type == "array"` gate was
|
|
# never exercised against reality and its `exit 1` branch — posting nothing at all — was
|
|
# unreachable in the suite. Same class as the timeline terminator, one function over.
|
|
# Parameterised, not merely corrected: a head that HAS statuses really does return an array, so
|
|
# both shapes are live and the job must read either.
|
|
def empty_statuses():
|
|
if os.environ.get("STUB_STATUS_EMPTY_SHAPE", "null") == "array":
|
|
return json.dumps({"state": "pending", "total_count": 0, "statuses": []})
|
|
return json.dumps({"state": "pending", "total_count": 0, "statuses": None})
|
|
|
|
# BEFORE ANY READ-COUNTING MODE. The `appears-on-read:N` modes count how many times the job has
|
|
# LOOKED at the combined status, and the page-2 completeness probe is part of the same look, not a
|
|
# further one — letting it increment those counters shifted "the verdict appears on read N" by one
|
|
# and broke three mid-run-race tests. Page 2 also has to terminate here for every mode that
|
|
# describes page 1 only, or the job would re-read page 1's rows as a second page and conclude the
|
|
# list is longer than it is.
|
|
if status_page > 1 and mode == "page2-garbage":
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if status_page > 1 and mode == "page2-error":
|
|
sys.exit(22)
|
|
if status_page > 1 and mode != "twopage":
|
|
print(empty_statuses())
|
|
sys.exit(0)
|
|
if mode.startswith("appears-on-read:"):
|
|
# A human verdict that does NOT exist at the first read and DOES exist at the re-read made
|
|
# immediately before the POST (ersatztv#706). Models a reviewer posting BLOCKED while the job
|
|
# is still enumerating — the window the first read structurally cannot see.
|
|
nth = int(mode.split(":", 1)[1])
|
|
ctr = out / "status_reads.txt"
|
|
n = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(n + 1))
|
|
if n + 1 < nth:
|
|
# STUB_PRE_ROW makes the earlier reads return a PRE-EXISTING row instead of an empty
|
|
# head (ersatztv#742). Without it every mid-run test starts from "no status at all", so
|
|
# only the ARRIVAL case is exercised and a presence-only abstain (`[ -z "$pre_state" ]`)
|
|
# is indistinguishable from a changed-row abstain. The REPLACEMENT case is the one that
|
|
# co-occurs with #742's own scenario: the head already carries an off-list verdict — that
|
|
# is WHY the job is re-deriving — and a real reviewer posts over it mid-run.
|
|
pre = os.environ.get("STUB_PRE_ROW", "")
|
|
if pre:
|
|
pre_creator, pre_state, pre_desc = pre.split("|", 2)
|
|
# DECOY CONTEXT + `total_count`, matching the shape the `existing:` modes model and the
|
|
# server actually returns. Without a decoy a dropped `select(.context == $c)` is
|
|
# invisible; without `total_count` the emptiness cross-check is never exercised.
|
|
rows = [
|
|
{"context": "ci/decoy", "status": "pending", "creator": None,
|
|
"description": "unrelated"},
|
|
{"context": "review-verdict/h10", "status": pre_state,
|
|
"creator": ({"login": pre_creator} if pre_creator else None),
|
|
"description": pre_desc},
|
|
]
|
|
print(json.dumps({"state": "pending", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
print(empty_statuses())
|
|
sys.exit(0)
|
|
# Configurable creator (ersatztv#742). This site asks "did anything human land while we were
|
|
# classifying", NOT "may this be inherited", so it must stay broad: an off-allow-list account
|
|
# posting BLOCKED here still has to make the job abstain.
|
|
# STUB_MIDRUN_ROW overrides the whole row, so a test can make the mid-run row differ from
|
|
# STUB_PRE_ROW in exactly ONE field of the state/creator/description triple. Without that,
|
|
# every mid-run test moves all three at once and no single clause of the comparison is pinned.
|
|
mid = os.environ.get("STUB_MIDRUN_ROW", "")
|
|
if mid:
|
|
mid_creator, mid_state, mid_desc = mid.split("|", 2)
|
|
else:
|
|
mid_creator = os.environ.get("STUB_MIDRUN_CREATOR", "timothy")
|
|
mid_state, mid_desc = "failure", "Review-verdict: BLOCKED @ a9e3e23 (base: main)"
|
|
mid_rows = [
|
|
{"context": "ci/decoy", "status": "pending", "creator": None, "description": "unrelated"},
|
|
{"context": "review-verdict/h10", "status": mid_state,
|
|
"creator": ({"login": mid_creator} if mid_creator else None),
|
|
"description": mid_desc},
|
|
]
|
|
print(json.dumps({"state": "pending", "total_count": len(mid_rows), "statuses": mid_rows}))
|
|
sys.exit(0)
|
|
if mode == "scalar-statuses-only":
|
|
# EVERY element unreadable and `total_count` agreeing, so both shape gates pass. This is the
|
|
# shape where dropping unreadable elements turns into "no verdict exists".
|
|
print(json.dumps({"state": "pending", "total_count": 1, "statuses": [7]}))
|
|
sys.exit(0)
|
|
if mode == "malformed-row-beside-verdict":
|
|
# A SCALAR IN `.statuses` BESIDE A REAL ROW. `.statuses` is an array and `total_count` agrees,
|
|
# so the response passes the shape gate; it is the ELEMENTS that cannot be read. An untyped
|
|
# `select(.context == $c)` hard-errors on the scalar, jq exits 5, and under `set -e` the
|
|
# assignment takes the whole step down — before any path that could mark the head, while the
|
|
# `success` below stays authoritative.
|
|
rows = [7, {"context": "review-verdict/h10", "status": "success",
|
|
"creator": {"login": "mallory"},
|
|
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: main)"}]
|
|
print(json.dumps({"state": "success", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
if mode == "flaky-combined":
|
|
# THE FIRST ATTEMPT ONLY. The read is retried once; without the retry this head reads as
|
|
# unreadable and takes the replacement path, which is a real behavioural difference and the
|
|
# only thing that distinguishes a retrying read from a one-shot one.
|
|
ctr = out / "combined_attempts.txt"
|
|
seen = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(seen + 1))
|
|
if seen == 0:
|
|
sys.exit(22)
|
|
print(empty_statuses())
|
|
sys.exit(0)
|
|
if mode.startswith("malformed-field:"):
|
|
# ONE CONSUMED FIELD AT A TIME. The `.creator` case had a proof and the other three did not,
|
|
# which is the per-FIELD gap this repo has a record for: a route written once for the field
|
|
# that produced it, and the siblings left to the reader's assumption.
|
|
field = mode.split(":", 1)[1]
|
|
row = {"id": 5, "context": "review-verdict/h10", "status": "failure",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}
|
|
# `.id` is a NUMBER when well-formed, so corrupting it needs a string; the other three
|
|
# are strings or an object, so a number corrupts them.
|
|
row[field] = "seven" if field == "id" else 7
|
|
print(json.dumps({"state": "failure", "total_count": 1, "statuses": [row]}))
|
|
sys.exit(0)
|
|
if mode.startswith("sentinel-with-id:"):
|
|
# THE CURRENT SENTINEL, AT A STABLE ID, on both reads. The reconciliation witness has to
|
|
# identify THIS row, so a fixture needs the combined endpoint to name an id that the seeded
|
|
# history may or may not contain — which is the whole discriminator between matching the row
|
|
# and matching its text. `:str` asks for a STRING id, for the numeric guard.
|
|
raw_id = mode.split(":", 1)[1]
|
|
row_id = raw_id[4:] if raw_id.startswith("str-") else int(raw_id)
|
|
rows = [
|
|
{"context": "ci/decoy", "status": "pending", "creator": None, "description": "unrelated"},
|
|
{"id": row_id, "context": "review-verdict/h10", "status": "pending",
|
|
"creator": None, "description": os.environ["STUB_UNVERIFIED_DESC"]},
|
|
]
|
|
print(json.dumps({"state": "pending", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
if mode == "malformed-creator-field":
|
|
# THE EXISTING h10 ROW WITH A CORRUPT `creator`. `.statuses` is an array of objects and
|
|
# `total_count` agrees, so the shape gates pass; it is a CONSUMED FIELD whose type the schema
|
|
# does not allow. Reading it as "no creator" makes the row unattributable, which is a licence
|
|
# to re-derive — over a human `failure`.
|
|
rows = [{"id": 5, "context": "review-verdict/h10", "status": "failure",
|
|
"creator": 7,
|
|
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)"}]
|
|
print(json.dumps({"state": "failure", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
if mode == "id-appears-on-second-read":
|
|
# THE SAME ROW, reported once WITHOUT `id` and once WITH it. Nothing else about it moves. An
|
|
# id comparison that does not require both sides to be present reads this as a replacement
|
|
# and makes the run abstain — over a row it had already declined to inherit.
|
|
ctr = out / "status_reads.txt"
|
|
n = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(n + 1))
|
|
row = {"context": "review-verdict/h10", "status": "success",
|
|
"creator": {"login": "mallory"},
|
|
"description": "Review-verdict: MERGEABLE @ a9e3e23 (base: main)"}
|
|
if n > 0:
|
|
row = dict(row, id=77)
|
|
rows = [{"context": "ci/decoy", "status": "pending", "creator": None,
|
|
"description": "unrelated"}, row]
|
|
print(json.dumps({"state": "pending", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
if mode == "sentinel-replaced-mid-run":
|
|
# THE SAME SENTINEL TEXT AT TWO DIFFERENT IDS (ersatztv#849). Both reads return an
|
|
# unverified sentinel whose description is byte-identical — which is what a fixed point IS —
|
|
# so only the row id distinguishes "the row I snapshotted" from "a row another run wrote
|
|
# while I classified". Ids are carried here and nowhere else in this stub because this is the
|
|
# only fixture whose outcome turns on them.
|
|
ctr = out / "status_reads.txt"
|
|
n = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(n + 1))
|
|
rows = [
|
|
{"context": "ci/decoy", "status": "pending", "creator": None, "description": "unrelated"},
|
|
{"id": 100 if n == 0 else 200, "context": "review-verdict/h10", "status": "pending",
|
|
"creator": None,
|
|
# FROM THE SHIPPED BODY, never a copy: the sentinel is a fixed point, so a stub carrying
|
|
# its own spelling would keep passing after the workflow reworded its own.
|
|
"description": os.environ["STUB_UNVERIFIED_DESC"]},
|
|
]
|
|
print(json.dumps({"state": "pending", "total_count": len(rows), "statuses": rows}))
|
|
sys.exit(0)
|
|
if mode.startswith("sentinel-appears-on-read:"):
|
|
# A repair sentinel written by ANOTHER, overlapping run between this job's first read and its
|
|
# last-moment re-read (ersatztv#706). Creator is null: the sentinel is machine-written.
|
|
nth = int(mode.split(":", 1)[1])
|
|
ctr = out / "status_reads.txt"
|
|
n = int(ctr.read_text()) if ctr.exists() else 0
|
|
ctr.write_text(str(n + 1))
|
|
if n + 1 < nth:
|
|
print(empty_statuses()); sys.exit(0)
|
|
print(json.dumps({"statuses": [
|
|
{"context": "review-verdict/h10", "status": "pending", "creator": None,
|
|
"description": "Human verdict raced this exemption write — re-post the verdict"}]}))
|
|
sys.exit(0)
|
|
if mode == "twopage":
|
|
# Page 1 AND page 2 both carry decoy contexts and NO `review-verdict/h10`: the list is longer
|
|
# than one page, so "no verdict exists" is not established and the job must refuse. Models a
|
|
# head with more contexts than the server-wide page cap (measured 50 here) — which is exactly
|
|
# why the guard cannot be written as a comparison against a hardcoded limit.
|
|
print(json.dumps({"state": "pending", "total_count": 3,
|
|
"statuses": [{"context": f"ci/p{status_page}x{i}", "status": "pending"}
|
|
for i in range(3)]}))
|
|
sys.exit(0)
|
|
if mode.startswith("rows:"):
|
|
# N decoy contexts and NO `review-verdict/h10`, to exercise the truncation guard. `total_count`
|
|
# deliberately MIRRORS the page length, because that is what the real endpoint does — it is the
|
|
# count for the page returned, not for the commit (measured at 1.27.1: `?limit=1` on a 6-context
|
|
# head returns len=1, total_count=1). A stub that reported the true total would make a
|
|
# length-vs-total guard look like it worked, which is the trap this models.
|
|
n = int(mode.split(":", 1)[1])
|
|
print(json.dumps({"state": "pending", "total_count": n,
|
|
"statuses": [{"context": f"ci/decoy{i}", "status": "pending"}
|
|
for i in range(n)]}))
|
|
sys.exit(0)
|
|
if mode == "total-count-string":
|
|
# A schema-corrupted `total_count` as a STRING alongside a null `.statuses`. `jq -r` renders 0
|
|
# and "0" identically, so a text compare would accept this.
|
|
print(json.dumps({"state": "pending", "total_count": "0", "statuses": None}))
|
|
sys.exit(0)
|
|
if mode == "transport-error":
|
|
# Real `gh()` is `curl -sf`: an HTTP error exits 22 with EMPTY stdout.
|
|
sys.exit(22)
|
|
if mode == "garbage":
|
|
print("<html>502 Bad Gateway</html>")
|
|
sys.exit(0)
|
|
if mode.startswith("existing:"):
|
|
# DECOY contexts either side, because real heads carry several (a live head showed 7 rows,
|
|
# the first an unrelated CI context). Their status is deliberately `pending`, NOT `success`:
|
|
# a `success` decoy triggers the very same short-circuit as a real verdict, so dropping
|
|
# `select(.context == $c)` produced an identical outcome and the mutation survived. With
|
|
# `pending` decoys, mis-selecting means no short-circuit — the job posts, and the test sees it.
|
|
#
|
|
# The row's PROVENANCE is configurable (ersatztv#698 route 3). A status POSTed by a user
|
|
# credential carries `.creator.login` and, for a real verdict, a `Review-verdict:`
|
|
# description; one POSTed by an Actions job carries `"creator": null`. Both were measured on
|
|
# the live combined endpoint. Defaults model a HUMAN verdict, so a test that does not opt out
|
|
# exercises the never-overwrite property.
|
|
creator = os.environ.get("STUB_STATUS_CREATOR", "timothy")
|
|
desc = os.environ.get("STUB_STATUS_DESC", "Review-verdict: MERGEABLE @ a9e3e23 (base: main)")
|
|
print(json.dumps({"statuses": [
|
|
{"context": "Build & test (.NET)", "status": "pending"},
|
|
{"context": "review-verdict/h10", "status": mode.split(":", 1)[1],
|
|
"creator": ({"login": creator} if creator else None), "description": desc},
|
|
{"context": "Functional E2E", "status": "pending"}]}))
|
|
sys.exit(0)
|
|
print(empty_statuses())
|
|
sys.exit(0)
|
|
|
|
print("{}")
|
|
"""
|
|
|
|
|
|
def _run_classify(
|
|
tmp_path,
|
|
enum_stub: str | None,
|
|
author: str = "timothy",
|
|
status_mode: str = "none",
|
|
jq16: bool = False,
|
|
status_creator: str | None = "timothy",
|
|
status_desc: str = "Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
timeline_mode: str = "none",
|
|
push_mode: str = "none",
|
|
timeline_pages: int = 1,
|
|
timeline_filtered_pages: str = "",
|
|
timeline_flaky_first: str = "",
|
|
history_mode: str = "none",
|
|
history_creator: str = "timothy",
|
|
history_extra: list | None = None,
|
|
midrun_creator: str = "timothy",
|
|
pre_row: str = "",
|
|
midrun_row: str = "",
|
|
reviewers: str | None = None,
|
|
timeline_terminator: str = "null",
|
|
status_empty_shape: str = "null",
|
|
mutate: tuple[str, str] | None = None,
|
|
post_fails: bool | str = False,
|
|
):
|
|
"""Execute the workflow's classify `run:` block with a stubbed enumeration script.
|
|
|
|
Returns the status payload the job POSTed, or None if it posted nothing.
|
|
"""
|
|
tmp_path.mkdir(parents=True, exist_ok=True) # the chained sentinel test passes sub-paths
|
|
bindir = tmp_path / "bin"
|
|
bindir.mkdir()
|
|
curl = bindir / "curl"
|
|
curl.write_text(WORKFLOW_STUB_CURL)
|
|
curl.chmod(0o755)
|
|
if jq16:
|
|
# Reproduce the runner's jq 1.6 (`-e` over EMPTY input exits 0, not 4). The shim is the one
|
|
# already used for pr-changed-files.sh in this file, so its fidelity is covered by that
|
|
# suite's own verify-the-verifier test.
|
|
jq = bindir / "jq"
|
|
jq.write_text(_JQ16_SHIM)
|
|
jq.chmod(0o755)
|
|
scripts = tmp_path / "scripts"
|
|
scripts.mkdir()
|
|
if enum_stub is not None:
|
|
enum = scripts / "pr-changed-files.sh"
|
|
enum.write_text(enum_stub)
|
|
enum.chmod(0o755)
|
|
|
|
env = dict(os.environ)
|
|
env["PATH"] = f"{bindir}{os.pathsep}{env['PATH']}"
|
|
env["STUB_DIR"] = str(tmp_path)
|
|
env["STUB_STATUS_MODE"] = status_mode
|
|
env["STUB_STATUS_CREATOR"] = status_creator or ""
|
|
env["STUB_STATUS_DESC"] = status_desc
|
|
env["STUB_TIMELINE_MODE"] = timeline_mode
|
|
env["STUB_PUSH_MODE"] = push_mode
|
|
env["STUB_TIMELINE_PAGES"] = str(timeline_pages)
|
|
env["STUB_TIMELINE_FILTERED_PAGES"] = timeline_filtered_pages
|
|
env["STUB_TIMELINE_FLAKY_FIRST"] = timeline_flaky_first
|
|
env["STUB_TIMELINE_TERMINATOR"] = timeline_terminator
|
|
env["STUB_STATUS_EMPTY_SHAPE"] = status_empty_shape
|
|
env["STUB_HISTORY_MODE"] = history_mode
|
|
env["STUB_HISTORY_CREATOR"] = history_creator
|
|
env["STUB_HISTORY_EXTRA"] = json.dumps(history_extra) if history_extra else ""
|
|
env["STUB_UNVERIFIED_DESC"] = UNVERIFIED_DESC
|
|
env["STUB_POST_FAILS"] = post_fails if isinstance(post_fails, str) else ("1" if post_fails else "")
|
|
env["STUB_MIDRUN_CREATOR"] = midrun_creator
|
|
env["STUB_PRE_ROW"] = pre_row
|
|
env["STUB_MIDRUN_ROW"] = midrun_row
|
|
env.update(
|
|
{
|
|
"GITEA_TOKEN": "stub",
|
|
"BASE_URL": "http://gitea.example/api/v1",
|
|
"GITEA_BASE_URL": "http://gitea.example/api/v1",
|
|
"REPO": "timothy/ersatztv",
|
|
"PR": "42",
|
|
"SHA": SHA,
|
|
"BASE_SHA": OTHER_SHA,
|
|
# The base branch the event was raised for, threaded to the enumeration (ersatztv#698).
|
|
# Omitting it is not a soft failure: `set -u` kills the step, no status is posted, and the
|
|
# absent required check blocks the merge — fail-closed, but it would take every PR with it.
|
|
"BASE_REF": "main",
|
|
"AUTHOR": author,
|
|
"PR_URL": "http://gitea.example/timothy/ersatztv/pulls/42",
|
|
}
|
|
)
|
|
|
|
script = tmp_path / "step.sh"
|
|
body = _classify_step()["run"]
|
|
if reviewers is not None:
|
|
# Substitute the allow-list into the REAL step body so the shipped membership loop runs
|
|
# against a multi-entry value. Every other test runs the singleton `timothy`, under which a
|
|
# loop that can only ever match its LAST entry is indistinguishable from a correct one — and
|
|
# the workflow's own diagnostic tells operators to add a second entry.
|
|
body, n = re.subn(
|
|
r'^(\s*)H10_REVIEWERS="[^"]*"$',
|
|
lambda m: f'{m.group(1)}H10_REVIEWERS="{reviewers}"',
|
|
body,
|
|
flags=re.MULTILINE,
|
|
)
|
|
assert n == 1, f"expected exactly one H10_REVIEWERS assignment to override, substituted {n}"
|
|
if mutate is not None:
|
|
# DISARM A CLAUSE OF THE SHIPPED BODY, for the mutation proofs (ersatztv#849).
|
|
#
|
|
# WHICH MUTANT EACH USE IS — a restored predecessor, a withdrawn draft, or a disarmed new
|
|
# clause — is enumerated at the ersatztv#849 section header below. Two restore `origin/main`
|
|
# verbatim; the rest disarm clauses that have no predecessor to restore, which is the precise
|
|
# mutation for a NEW guard and is not the same claim as "this is what the code used to be".
|
|
#
|
|
# The count assertion is the binding. Without it a clause that has since been reworded or
|
|
# moved substitutes ZERO times, the "mutant" is the unmutated body, and the proof asserts the
|
|
# unmutated behaviour while reporting success — a mutation claim that measures nothing.
|
|
old_clause, new_clause = mutate
|
|
occurrences = body.count(old_clause)
|
|
assert occurrences == 1, (
|
|
f"the mutation target appears {occurrences} times in the shipped step body, expected 1. "
|
|
f"It has been reworded, moved or duplicated, so this proof is no longer bound to the "
|
|
f"clause it names: {old_clause!r}"
|
|
)
|
|
body = body.replace(old_clause, new_clause, 1)
|
|
script.write_text(body)
|
|
r = subprocess.run(["bash", str(script)], cwd=tmp_path, env=env, capture_output=True, text=True)
|
|
# Assert the WIRING, not only the classification. The stub accepts every POST, so a status aimed
|
|
# at the wrong endpoint, sha, host or repo would otherwise leave these tests green while the real
|
|
# required check was never written.
|
|
#
|
|
# Two ways this check could disable itself:
|
|
# * guarding it with `if url_file.exists()` — deleting the recorder in the stub then turns it
|
|
# into a no-op and every test stays green, a verifier that silently opts out;
|
|
# * comparing only the URL SUFFIX — a POST to the right path on the WRONG HOST OR REPO then
|
|
# passes. Compare the whole URL against the env this job was given.
|
|
posted = tmp_path / "posted.json"
|
|
url_file = tmp_path / "posted_url.txt"
|
|
if posted.exists():
|
|
assert url_file.exists(), (
|
|
"a status was POSTed but its URL was not recorded — the wiring assertion below would have silently skipped"
|
|
)
|
|
expected = f"{env['BASE_URL']}/repos/{env['REPO']}/statuses/{SHA}"
|
|
assert url_file.read_text().strip() == expected, (
|
|
f"the status was POSTed to {url_file.read_text().strip()!r}, expected {expected!r}"
|
|
)
|
|
return (json.loads(posted.read_text()) if posted.exists() else None), r
|
|
|
|
|
|
DOCS_ONLY = 'printf "docs/a.md\\ndocs/b.md\\n"\n'
|
|
|
|
|
|
def _assert_withheld(tmp_path, r, why, expect_rc):
|
|
"""The gate refused to grant the exemption — which since ersatztv#849 means the head is MARKED,
|
|
not left alone.
|
|
|
|
Every one of these tests asserted `posted is None`. That was right while "withhold" meant "write
|
|
nothing", and it became a fail-open assertion when it stopped meaning that: declining to write
|
|
protects a REAL verdict on the head and leaves a FORGED one, and this job's own red status is not
|
|
a required check, so branch protection still sees whatever was already there.
|
|
|
|
THE WHOLE POST SEQUENCE, not the last write. Inspecting only the final status would accept a job
|
|
that posted `success` and then repaired it — and the green interval IS part of the threat model
|
|
here, since branch protection and an already-scheduled auto-merge can both observe it. Exactly
|
|
one POST, and it is the sentinel.
|
|
|
|
THE EXIT CODE IS PART OF THE CONTRACT and differs by path, so each caller states its own rather
|
|
than inheriting a default: the two read refusals were already non-zero exits before this change
|
|
and stay red, while the fence branch abstains cleanly. A single default would let one path's
|
|
regression hide behind the other's expectation.
|
|
"""
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
f"{why}: expected exactly one status write — the sentinel — got {seq}. Nothing posted at all "
|
|
f"means whatever this head already carries is standing unread; more than one means a green "
|
|
f"was published and taken back.\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "pending" and seq[0]["description"] == UNVERIFIED_DESC, (
|
|
f"{why}: expected the unverified-write sentinel, got {seq[0]}\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode == expect_rc, f"{why}: expected exit {expect_rc}, got {r.returncode}\n{r.stdout[-900:]}"
|
|
|
|
|
|
def test_a_FAILING_enumeration_withholds_the_exemption_even_when_stdout_looks_docs_only(tmp_path):
|
|
"""The mutation this closes: ignore the exit status, trust stdout."""
|
|
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n")
|
|
assert posted is not None, f"the job posted no status at all: {r.stderr}"
|
|
assert posted["state"] == "pending", (
|
|
f"a docs-only exemption was granted from the stdout of a script that FAILED "
|
|
f"(state={posted['state']}, desc={posted['description']!r}) — the exit status is not being "
|
|
"checked"
|
|
)
|
|
|
|
|
|
def test_workflow_positive_control_the_same_output_with_exit_0_DOES_exempt(tmp_path):
|
|
"""Without this, the test above could pass for any unrelated reason and prove nothing."""
|
|
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 0\n")
|
|
assert posted is not None, f"the job posted no status at all: {r.stderr}"
|
|
assert posted["state"] == "success", (
|
|
"the positive control failed, so the negative test above cannot be trusted to be measuring "
|
|
f"the exit status at all (state={posted['state']}, desc={posted['description']!r})"
|
|
)
|
|
|
|
|
|
def test_a_crashing_enumeration_also_withholds_the_exemption(tmp_path):
|
|
posted, _ = _run_classify(tmp_path, "#!/usr/bin/env bash\n" + DOCS_ONLY + "kill -TERM $$\n")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_a_MISSING_enumeration_script_withholds_the_exemption(tmp_path):
|
|
"""A PR whose BASE predates the script's introduction. It must post an actionable `pending`
|
|
rather than dying with no status — an absent required check blocks the merge either way, but a
|
|
stalled PR with no explanation is how a gate acquires a reputation for being flaky.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, None)
|
|
assert posted is not None, f"the job posted no status at all: {r.stderr}"
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def _emitting(*paths):
|
|
body = "".join(f'printf "%s\\n" "{p}"\n' for p in paths)
|
|
return "#!/usr/bin/env bash\n" + body + "exit 0\n"
|
|
|
|
|
|
def test_a_code_file_defeats_the_docs_only_exemption(tmp_path):
|
|
"""The workflow's own classification — deliberately NOT shared with the hook, whose allow-list
|
|
is wider because a match there falls through to a human prompt rather than a green status.
|
|
"""
|
|
posted, _ = _run_classify(tmp_path, _emitting("docs/a.md", "ErsatzTV/Program.cs"))
|
|
assert posted is not None
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_a_BOT_pr_touching_a_protected_path_is_NOT_exempt(tmp_path):
|
|
"""`PROTECTED` guards the BOT exemption specifically, and mutation testing is how that got
|
|
stated correctly. A docs-only+protected file list passes even with the `PROTECTED` clause
|
|
deleted — `PROTECTED` (`.claude/ .gitea/ .husky/ scripts/ docker/ci/`) and `DOCS_ONLY`
|
|
(`docs/`, root `*.md`) are DISJOINT, so on the docs-only path that clause can never fire and
|
|
the `DOCS_ONLY` check does all the work. Such a test looks like it covers the self-exemption
|
|
hole and covers nothing.
|
|
|
|
Renovate lands patch bumps unattended via Gitea's own auto-merge, so a bot PR that edits the
|
|
gate, CI, the hooks, or the scripts they call is the one path where an unreviewed change to the
|
|
merge gate could actually merge itself.
|
|
"""
|
|
posted, _ = _run_classify(
|
|
tmp_path, _emitting("ErsatzTV/Program.cs", "scripts/pr-changed-files.sh"), author="renovate"
|
|
)
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", (
|
|
"a bot PR editing the shared enumeration was auto-exempted — a PR that weakens the merge "
|
|
"gate must never be able to exempt itself from the merge gate"
|
|
)
|
|
|
|
|
|
def test_bot_positive_control_a_plain_bot_pr_IS_exempt(tmp_path):
|
|
"""Proves the test above measures `PROTECTED` and not merely 'bot PRs are never exempt'."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "success", (
|
|
"the bot exemption never fires at all, so the protected-path test above proves nothing"
|
|
)
|
|
|
|
|
|
# --- The STATUS READ: the ersatztv#647 guard and the never-overwrite short-circuit -------------
|
|
#
|
|
# These were unreachable until the stub's status response became configurable. Four mutations
|
|
# survived the full suite without them, including re-introducing the literal ersatztv#647 fail-open.
|
|
|
|
|
|
def test_a_transport_failure_on_the_STATUS_READ_REPLACES_the_unknown_state(tmp_path):
|
|
"""`gh()` is `curl -sf`, so an HTTP error yields exit 22 and EMPTY stdout. Reading that as "no
|
|
verdict exists" would let the job post over a real human verdict, so the exemption is still
|
|
withheld — but "withheld" now means REPLACED, not merely not-written (ersatztv#849 route 5).
|
|
|
|
Asserting `posted is None` here — on the reasoning that declining to write protects a verdict
|
|
this job cannot see — is true when the head carries a REAL verdict and exactly wrong when it
|
|
carries a FORGED one. An off-list credential's `review-verdict/h10=success` is the status #742
|
|
exists to revoke, revocation happens by re-deriving the row, and an unreadable read is the one
|
|
thing that stops it. The job went red — and its own job status is not a required check, so
|
|
branch protection still saw the green. Uncertainty resolved toward SUCCESS.
|
|
|
|
So the unknown state is durably replaced with the sticky unverified sentinel. Nothing is lost
|
|
that cannot be recovered: `/statuses/{sha}` keeps one row per POST, so a genuine verdict masked
|
|
here is still in the history and the reconciliation on the next run finds it.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="transport-error")
|
|
assert r.returncode != 0, "an unreadable status read must still fail the job"
|
|
assert posted is not None, (
|
|
f"an unreadable status read left whatever this head carries standing unread:\n{r.stdout[-900:]}"
|
|
)
|
|
assert posted["state"] == "pending" and posted["description"] == UNVERIFIED_DESC, (
|
|
f"expected the unverified-write sentinel, got {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_GARBAGE_status_response_REPLACES_the_unknown_state(tmp_path):
|
|
"""A proxy error page is a 200 with a non-JSON body — not an absent verdict, and not a state this
|
|
job may leave standing unread. Same treatment as the transport failure above."""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="garbage")
|
|
assert r.returncode != 0
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"expected the unverified-write sentinel, got {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
# Two mutations to this read are NOT covered, and both are behaviourally equivalent rather than gaps:
|
|
# * `first` -> `last`: `select(.context == $c)` yields exactly ONE row, because the job reads the
|
|
# COMBINED status endpoint, which returns latest-per-context by contract. A fixture with two
|
|
# `review-verdict/h10` rows would be testing something the API does not produce.
|
|
# * `test_a_GARBAGE_status_response_posts_NOTHING` does not, on its own, defend the type guard:
|
|
# with the guard gone, the following `jq -r` fails on non-JSON and `set -e` kills the job anyway.
|
|
# That is fail-closed by REDUNDANCY. The guard's own behaviour is pinned by the jq-1.6 test below,
|
|
# which is where it actually matters.
|
|
|
|
|
|
@pytest.mark.parametrize("existing", ["success", "failure"])
|
|
@pytest.mark.parametrize(
|
|
"paths,author",
|
|
[
|
|
(("ErsatzTV/Program.cs",), "timothy"), # non-exempt: only the short-circuit can stop it
|
|
(("docs/a.md",), "timothy"), # docs-only EXEMPT
|
|
(("Directory.Packages.props",), "renovate"), # bot EXEMPT
|
|
],
|
|
)
|
|
def test_an_existing_verdict_on_this_head_is_NEVER_overwritten(tmp_path, existing, paths, author):
|
|
"""A human verdict for this exact head may already exist — the reviewer ran
|
|
post-review-verdict.sh before this job finished, or the job re-ran. Re-posting would un-approve
|
|
a reviewed head, or (worse) approve one a human marked BLOCKED.
|
|
|
|
`failure` is the sharp case: that is a human saying NO, and an exemption posted over it would
|
|
turn a rejection into a merge.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting(*paths), author=author, status_mode=f"existing:{existing}")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, f"overwrote an existing '{existing}' verdict on this head with {paths} as {author}"
|
|
|
|
|
|
# --- The `count -eq 0` guard, on the path where it is the ONLY guard ---------------------------
|
|
|
|
|
|
def test_an_EMPTY_enumeration_is_not_exempt_even_for_a_BOT(tmp_path):
|
|
"""The author must be a BOT for this to test anything.
|
|
|
|
With a non-bot author the blank line an empty list produces already fails `DOCS_ONLY`, so the
|
|
`[ "${count:-0}" -eq 0 ]` guard never decides the outcome — the same short-circuit that made an
|
|
earlier `PROTECTED` test vacuous. On the bot path that guard is the ONLY thing between an
|
|
unreadable-but-successful enumeration and an unattended `success`.
|
|
|
|
Verified by mutation: changing `grep -c .` to `grep -c ''` (counting the blank line, so
|
|
`count=1`) grants a bot PR `success` here while every other test stays green.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, "#!/usr/bin/env bash\nexit 0\n", author="renovate")
|
|
assert posted is not None, f"the job posted no status at all: {r.stderr}"
|
|
assert posted["state"] == "pending", (
|
|
"an empty file list was treated as a bot exemption — the enumeration returning nothing is "
|
|
"not evidence that nothing was changed"
|
|
)
|
|
|
|
|
|
# --- Anchors in the classifier predicates ------------------------------------------------------
|
|
|
|
|
|
def test_the_BOT_match_is_whole_line_not_substring(tmp_path):
|
|
"""`grep -qxF` is anchored; plain `grep -qF` would exempt any author whose name CONTAINS a bot
|
|
name. `ova` is a substring of `renovate`."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), author="ova")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", "a substring of a bot name was granted the bot exemption"
|
|
|
|
|
|
def test_DOCS_ONLY_anchors_the_markdown_extension(tmp_path):
|
|
r"""`[^/]*\.md$` must match only a top-level file ENDING in .md. Losing the `$` exempts
|
|
`evil.mdx`, and the docs-only exemption posts a green status with nobody in the loop."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("evil.mdx"))
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", "a .mdx file was accepted as docs-only"
|
|
|
|
|
|
def test_a_transport_failure_under_jq_1_6_STILL_refuses_to_classify(tmp_path):
|
|
"""The ersatztv#647 fail-open, tested where it actually lives: jq 1.6.
|
|
|
|
The shell emptiness check exists because `jq -e` over EMPTY input exits 4 on jq >= 1.7 but **0 on
|
|
jq 1.6, which is what the runner ships**. Remove that check and, on a dev Mac's 1.8, the guard
|
|
still fires and every test stays green — the bug is invisible locally.
|
|
|
|
Pinning the construct STRUCTURALLY instead, on the grounds that "no behavioural test can catch
|
|
this on a dev machine", is wrong: this file already imports `_JQ16_SHIM` for
|
|
`pr-changed-files.sh`, so the runner's quirk is reproducible here. A structural version is also
|
|
weaker than it looks — it strips only FULL-LINE comments, so leaving the literal as a trailing
|
|
comment on the surviving `if` satisfies it while the real guard is gone.
|
|
|
|
This test is strictly stronger: it catches that mutant, needs no comment-stripping, and fails for
|
|
the right reason. Verified by mutation under both jq versions.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="transport-error", jq16=True)
|
|
assert r.returncode != 0, (
|
|
"under jq 1.6 an unreadable status read must still fail the job — this is the exact ersatztv#647 fail-open"
|
|
)
|
|
# THE MUTANT THIS CATCHES IS THE EXEMPTION, not the absence of a write (ersatztv#849 route 5
|
|
# changed what "withhold" means here — see the transport-failure test above). Drop the emptiness
|
|
# check and jq 1.6 reads "" as "no verdict exists", so the job posts `Exempt: docs-only change`.
|
|
# The sentinel and that exemption are the two distinguishable outcomes.
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"under jq 1.6 an empty body was read as 'no verdict exists' and classified normally: {posted}"
|
|
f"\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_DOCS_ONLY_anchors_the_START_of_the_path_too(tmp_path):
|
|
r"""`^(docs/|...)` must match only at the start. Losing the `^` exempts `ErsatzTV/docs/Evil.cs`,
|
|
which is a C# file — fail-OPEN, and the sibling of the `$` case above."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/docs/Evil.cs"))
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", "a path merely CONTAINING docs/ was accepted as docs-only"
|
|
|
|
|
|
# --- Route 2: a bot ACCOUNT does not attribute the CODE (ersatztv#698) --------------------------
|
|
#
|
|
# `AUTHOR` is `pull_request.user.login` — the PR's CREATOR, which is immutable. The head a PR points
|
|
# at is not. Force-push application code onto an open Renovate branch and the PR is still authored by
|
|
# `renovate`, still touches no protected path, and was exempted. Checking the PUSHER instead does not
|
|
# help: a git author/committer is self-asserted text. So the exemption is constrained by what a
|
|
# dependency bump can legitimately BE.
|
|
#
|
|
# The allow-list is measured, not guessed: across all 11 Renovate PRs this repo has ever had, the
|
|
# paths touched were `Directory.Packages.props` (10) and `.config/dotnet-tools.json` (1). The one
|
|
# historical outlier, PR #20, touched a `.csproj` AND two C# files — and received an unattended bot
|
|
# exemption for a source change.
|
|
|
|
|
|
def test_a_BOT_pr_carrying_a_CODE_file_is_NOT_exempt(tmp_path):
|
|
"""Route 2, stated as a behaviour: the hijacked-branch case."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", (
|
|
"a PR authored by the renovate account was exempted while changing a C# file — the bot "
|
|
"identity is being read as attribution for code it did not write"
|
|
)
|
|
|
|
|
|
def test_a_BOT_pr_mixing_a_manifest_WITH_code_is_NOT_exempt(tmp_path):
|
|
"""The realistic shape of the attack: keep the manifest edit so the PR still looks like a bump,
|
|
and smuggle the code alongside it. A rule that asked 'does it touch a manifest' rather than 'is
|
|
EVERY path a manifest' would exempt this."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props", "ErsatzTV/Program.cs"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", (
|
|
"a manifest edit was enough to carry a C# file through the bot exemption — the allow-list "
|
|
"is being applied as 'any' rather than 'all'"
|
|
)
|
|
|
|
|
|
def test_bot_positive_control_a_REAL_dependency_bump_IS_still_exempt(tmp_path):
|
|
"""Renovate uses platformAutomerge, so breaking this deadlocks every dependency PR. The negative
|
|
tests above prove nothing if the exemption no longer works at all."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "success", (
|
|
f"a plain dependency bump lost its exemption ({posted['description']!r}) — every Renovate PR "
|
|
"would now stall waiting on a human verdict"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["Directory.Packages.props", ".config/dotnet-tools.json"])
|
|
def test_every_manifest_in_the_allow_list_is_actually_exempt(tmp_path, path):
|
|
"""Pin the SET, not one example of it. A test that only ever exercises Directory.Packages.props
|
|
cannot see a typo in any of the other three alternations."""
|
|
posted, _ = _run_classify(tmp_path, _emitting(path), author="renovate")
|
|
assert posted is not None, path
|
|
assert posted["state"] == "success", f"{path} is in the allow-list but was not exempted"
|
|
|
|
|
|
def test_the_manifest_allow_list_is_ANCHORED(tmp_path):
|
|
"""Unanchored, `Directory.Packages.props` would match `evil/Directory.Packages.props.cs`. Same
|
|
class as the two DOCS_ONLY anchoring tests above, which is why it is tested the same way."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("evil/Directory.Packages.props.cs"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", "the manifest allow-list matched mid-path"
|
|
|
|
|
|
def test_a_BOT_docs_only_pr_is_STILL_exempt_via_the_docs_rule(tmp_path):
|
|
"""The regression the restructure exists to prevent.
|
|
|
|
Written as an `elif` chain, a Renovate PR touching only `docs/` enters the bot branch, fails the
|
|
manifest test, and never reaches the docs-only branch — silently withdrawing an exemption the
|
|
docs-only rule grants on its own merits for ANY author. The predicates are therefore evaluated
|
|
independently and the decision made afterwards.
|
|
"""
|
|
posted, _ = _run_classify(tmp_path, _emitting("docs/note.md"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "success", (
|
|
"a docs-only PR lost its docs-only exemption merely because its author is a bot — the "
|
|
"exemptions are chained rather than composed"
|
|
)
|
|
|
|
|
|
def test_a_BOT_pr_touching_a_protected_path_is_still_NOT_exempt(tmp_path):
|
|
"""PROTECTED must keep outranking both exemptions, including the manifest one."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("Directory.Packages.props", "scripts/evil.sh"), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
# --- Route 3: an inherited `success` is re-derived unless it is an ALLOW-LISTED reviewer's verdict
|
|
# (ersatztv#742; `attributable` alone is the weaker test that governs an inherited `failure`) -------
|
|
#
|
|
# The short-circuit used to exit on ANY existing `success`, so an exemption this job wrote was
|
|
# indistinguishable from a verdict a human wrote. Obtained once, a forged success was thereafter
|
|
# accepted unchanged on every later run, because the guard exited before looking at the PR, the base,
|
|
# the author or the files.
|
|
#
|
|
# MEASURED first on Gitea 1.25.4 and RE-MEASURED on 1.27.1, 2026-09-02 (ersatztv#869) over four
|
|
# merged PR heads, on the list endpoint as well as the combined one. It comes from the v1.27.1
|
|
# response builder: an Actions job's creator id is the synthetic -2, which no `user` row matches, so
|
|
# the lookup fails, the error is discarded and the field serializes null. Same builder on both
|
|
# endpoints, so they cannot disagree. Note null therefore means "no user row resolved", NOT "a
|
|
# machine wrote it" — a deleted author looks identical, which is safe here only because the test
|
|
# below fires on a POSITIVE match. The workflow comment this duplicates carries the full
|
|
# measurement and the full mechanism; keep the two in step.
|
|
# On the COMBINED endpoint this job reads: a status POSTed with a user
|
|
# credential carries `.creator.login` (`timothy`), one POSTed by an Actions job carries
|
|
# `"creator": null`. Both halves are required, and the test is written in the POSITIVE direction —
|
|
# short-circuit only on something identified as a human verdict — so an unrecognised shape is
|
|
# re-derived rather than trusted.
|
|
|
|
|
|
def test_a_MACHINE_written_exemption_success_is_RE_DERIVED_not_inherited(tmp_path):
|
|
"""Route 3. The status looks exactly like one this job writes: creator null, `Exempt:` wording.
|
|
|
|
The PR now carries a C# file, so re-deriving must downgrade it to `pending`. Inheriting it would
|
|
leave a forged exemption standing forever.
|
|
"""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator=None,
|
|
status_desc="Exempt: docs-only change (no code, no protected path)",
|
|
)
|
|
assert posted is not None, (
|
|
"an existing machine-written success was inherited unchanged — this is route 3, and it is "
|
|
"how a forgery obtained once survives every subsequent run"
|
|
)
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_a_success_with_NO_creator_but_a_VERDICT_LOOKING_description_is_re_derived(tmp_path):
|
|
"""Isolates the creator half. Both conditions are required; either alone is forgeable by the
|
|
other party. If a future Gitea populates `creator` for Actions, the description half still
|
|
fails — the guard degrades toward re-deriving, never toward trusting."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator=None,
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
)
|
|
assert posted is not None, "a creatorless status was trusted on the strength of its description"
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_a_success_with_a_creator_but_an_EXEMPT_description_is_re_derived(tmp_path):
|
|
"""Isolates the description half, the mirror of the test above."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Exempt: docs-only change (no code, no protected path)",
|
|
)
|
|
assert posted is not None, "a status was trusted on the strength of its creator alone"
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
# --- ersatztv#742: WHOSE creator counts -------------------------------------------------------
|
|
#
|
|
# "Non-null creator" proved only that a status was POSTed with SOME user credential. `RENOVATE_TOKEN`
|
|
# is a `write:repository` PAT of the `renovate` bot; it cannot be scoped down the way #697 scoped the
|
|
# registry credential, and secrets are a per-repo STORE, so any workflow can reference it. A status
|
|
# POSTed with it reads back `creator: renovate` — non-null, and therefore inherited as a verdict.
|
|
|
|
|
|
@pytest.mark.parametrize("creator", ["renovate", "someone-else"])
|
|
def test_a_verdict_shaped_SUCCESS_from_a_NON_allowlisted_account_is_RE_DERIVED(tmp_path, creator):
|
|
"""The route this issue closes. Everything else about the row is a perfect verdict: a real
|
|
creator, a `Review-verdict:` description recording THIS base. Only the account is wrong.
|
|
|
|
Paired with a C# file so re-deriving is visible as a downgrade to `pending`; inheriting would
|
|
leave the forgery standing on every later run, which is the property that made route 3 durable.
|
|
|
|
`success` ONLY — see the `failure` test below, which pins the opposite behaviour deliberately.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator=creator,
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
)
|
|
assert posted is not None, (
|
|
f"a verdict-shaped success written by '{creator}' was inherited — non-null creator is being "
|
|
f"read as 'a reviewer wrote this'. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert posted["state"] == "pending"
|
|
# THE DIAGNOSTIC IS PART OF THE CONTRACT, not decoration. It is the only thing that says so when
|
|
# a reviewer has been added to the repo and forgotten from `H10_REVIEWERS`, and a tidy-up that
|
|
# drops any of its three values must redden something. Asserted on the LOG LINE the job always
|
|
# reaches, not on a separate annotation: the annotation that used to carry this was withdrawn
|
|
# because deleting it left the whole suite green.
|
|
assert creator in r.stdout, (
|
|
f"the diagnostic does not name the account whose verdict was discarded. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
# The VALUE, not the word. Asserting `"H10_REVIEWERS" in stdout` is satisfied by the
|
|
# remediation sentence on its own — so deleting the allow-list from the diagnostic leaves it
|
|
# green. Derived from the shipped literal rather than restated here.
|
|
assert f"allow-list='{_h10_reviewers_literal()}'" in r.stdout, (
|
|
"the diagnostic does not print the accepted set, so a reviewer left off it cannot tell why "
|
|
f"their verdict was re-derived. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
assert "add them to H10_REVIEWERS" in r.stdout, (
|
|
f"the diagnostic does not say how to fix it. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("creator", ["renovate", "someone-else"])
|
|
def test_an_off_list_FAILURE_is_LEFT_ALONE_even_on_an_EXEMPT_PR(tmp_path, creator):
|
|
"""THE FAIL-OPEN A SYMMETRIC ALLOW-LIST SHIPS, and the reason `failure` is treated differently.
|
|
|
|
Scenario: a real reviewer (`alice`) was added to the repo and forgotten from `H10_REVIEWERS`.
|
|
They post `Review-verdict: BLOCKED`. The PR is docs-only, so it is EXEMPT. If the allow-list
|
|
governed `failure` too, every guard downstream declines in turn — the short-circuit
|
|
(`ex_human=no`), the mid-run abstain (the triple never changed), and the post-write repair (the
|
|
row predates the high-water mark) — and the job posts an exemption `success` over an explicit
|
|
human rejection.
|
|
|
|
Deliberately paired with a DOCS-ONLY file list: with a C# file the job would post `pending`
|
|
anyway and the test would pass for the wrong reason, measuring nothing.
|
|
|
|
The asymmetry costs nothing #742 bought. An inherited `failure` cannot green anything; the worst
|
|
a forged one achieves is a stall, and `post-review-verdict.sh` POSTs unconditionally, so any
|
|
human clears it in one command.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:failure",
|
|
status_creator=creator,
|
|
status_desc="Review-verdict: BLOCKED @ a9e3e23 (base: main)",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
f"an exemption was posted over a BLOCKED verdict from '{creator}' — the allow-list is "
|
|
f"governing `failure`, which turns a rejection green. Posted: {posted}. "
|
|
f"Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("creator", ["timothy2", "xtimothy", "TIMOTHY", "timoth", " timothy"])
|
|
def test_an_account_that_merely_RESEMBLES_an_allowlisted_login_is_RE_DERIVED(tmp_path, creator):
|
|
"""Membership must be whole-value equality, not a substring, prefix or case-folded test.
|
|
|
|
Gitea logins are first-come, so `timothy2` is registerable by anyone; a containment test would
|
|
hand them the gate. `TIMOTHY` is the case arm — Gitea logins are case-insensitive for LOGIN but
|
|
the API returns the stored form, and accepting a fold would widen the set to whatever Gitea
|
|
happens to consider equal rather than to what this file lists.
|
|
"""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator=creator,
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
)
|
|
assert posted is not None, f"'{creator}' was accepted as the allow-listed reviewer"
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_the_RACED_verdict_check_still_counts_a_NON_allowlisted_creator(tmp_path):
|
|
"""THE POLARITY TEST, and the reason the allow-list was not applied everywhere `creator` is read.
|
|
|
|
NOT for the obvious reason, which does not hold: a genuine reviewer is on
|
|
`$H10_REVIEWERS` by construction, so membership here would still count every real verdict, and an
|
|
off-list row is one the job deliberately overwrote a few lines earlier.
|
|
|
|
The reason is the MISCONFIGURATION case — a second human account added to the repo and forgotten
|
|
from the literal — which is the only way an off-list row is ever a real verdict. This is the last
|
|
net before a green stands, and the errors are not symmetric: a repair to `pending` is recoverable
|
|
by re-posting, a wrongly-standing `success` on an unreviewed head is not. The cost of being broad
|
|
is real and is documented at the site: any verdict-shaped row landing in the write window trips
|
|
the sticky sentinel and costs that head its exemption until a human clears it.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
history_mode="human-after-post",
|
|
history_creator="renovate",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"the exemption was not repaired after a non-allow-listed account raced the write — the "
|
|
f"allow-list has been applied to the raced check too. Posts: {seq}. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success"
|
|
assert seq[1]["state"] == "pending"
|
|
assert posted is not None and posted["state"] == "pending"
|
|
|
|
|
|
def test_the_MID_RUN_abstain_still_fires_for_a_NON_allowlisted_creator(tmp_path):
|
|
"""The SECOND mirror-image site, and the one a #742-style narrowing gets wrong.
|
|
|
|
`read_existing_verdict` is called twice. The first call decides "may this be INHERITED" and is
|
|
correctly allow-list-gated. The second — the last-moment re-read immediately before the POST —
|
|
asks the OTHER question: "did a reviewer post a verdict while we were classifying?" Narrowing
|
|
that one does not make it stricter, it makes the job stop abstaining: it posts its exemption
|
|
`success` straight over the row instead.
|
|
|
|
The post-write repair usually catches that, but it is skipped entirely whenever the high-water
|
|
mark could not be established (`max_id_before=-1`, a `::warning::` and nothing more), so the
|
|
green would stand over a `BLOCKED`. Fail-toward-SUCCESS, at the exact site the design says must
|
|
stay broad.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
midrun_creator="renovate",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"the job posted over a verdict that landed mid-run from a non-allow-listed account — the "
|
|
f"allow-list has been applied to the mid-run abstain as well. Posted: {posted}. "
|
|
f"Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_verdict_REPLACING_a_pre_existing_row_mid_run_still_makes_the_job_ABSTAIN(tmp_path):
|
|
"""The REPLACEMENT half of the mid-run abstain, and the clause a presence test cannot reach.
|
|
|
|
The abstain used to be free: both reads computed an identical predicate, so "fired at the second
|
|
and not the first" meant "arrived in between". Splitting `ex_human` from `ex_attributable` broke
|
|
that, and the repair is an explicit comparison against the state/creator/description triple
|
|
snapshotted at the first read. `[ -z "$pre_state" ]` — abstain only when the first read saw
|
|
NOTHING — is the plausible wrong version, and every other mid-run test starts from an empty head,
|
|
so nothing distinguishes the two.
|
|
|
|
This is the ordering that actually co-occurs with #742: the head ALREADY carries an off-list
|
|
`Review-verdict:` success (which is why the job is re-deriving at all), and a real reviewer posts
|
|
`BLOCKED` over it while the job classifies. Under the presence-only version `pre_state` is
|
|
non-empty, the abstain is skipped, and the docs-only exemption is posted straight over the
|
|
rejection — with the post-write repair blind to it whenever the high-water mark is unestablished.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
pre_row="renovate|success|Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
midrun_creator="timothy",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"the job posted an exemption over a verdict that REPLACED a pre-existing row mid-run — the "
|
|
"abstain is testing presence ('was the head empty before?') rather than change. "
|
|
f"Posted: {posted}. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
|
|
|
|
def test_every_test_name_cited_by_the_exemption_provenance_record_EXISTS():
|
|
"""A record that names a test as its mutation proof must name one that can be run.
|
|
|
|
This is not hypothetical: renaming a test while splitting the `success`/`failure` cases left
|
|
`ci.exemption-provenance` citing `..._status_from_...` when the tree had `..._SUCCESS_from_...`.
|
|
A citation that cannot be resolved is worse than none — it reads as a checked proof and sends
|
|
the next reader looking for a test that does not exist.
|
|
|
|
Deliberately scoped to the ONE record that carries this change's proofs, and derived from the
|
|
record's own text rather than from a list restated here, so a newly cited name is covered the
|
|
moment it is written.
|
|
"""
|
|
record = REPO_ROOT / "docs" / "decisions" / "records" / "ci" / "exemption-provenance.md"
|
|
assert record.exists(), f"{record} is missing — the record this suite is the proof for"
|
|
cited = set(re.findall(r"`(test_[A-Za-z0-9_]+)`", record.read_text()))
|
|
assert cited, "the record cites no test by name; if that is deliberate, delete this guard"
|
|
defined = set(re.findall(r"^def (test_[A-Za-z0-9_]+)", Path(__file__).read_text(), re.MULTILINE))
|
|
for other in sorted(Path(__file__).parent.glob("test_*.py")):
|
|
defined |= set(re.findall(r"^def (test_[A-Za-z0-9_]+)", other.read_text(), re.MULTILINE))
|
|
missing = sorted(cited - defined)
|
|
assert not missing, (
|
|
f"ci.exemption-provenance cites {missing}, which no test in scripts/tests defines — the "
|
|
"record names a mutation proof nobody can run"
|
|
)
|
|
|
|
|
|
_PRE = "renovate|success|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"clause,midrun_row",
|
|
[
|
|
("description", "renovate|success|Review-verdict: BLOCKED @ a9e3e23 (base: main)"),
|
|
("creator", "timothy|success|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"),
|
|
("state", "renovate|failure|Review-verdict: MERGEABLE @ a9e3e23 (base: main)"),
|
|
],
|
|
)
|
|
def test_EACH_clause_of_the_changed_row_comparison_is_load_bearing(tmp_path, clause, midrun_row):
|
|
"""One case per field, because a triple whose cases all move all three fields pins none of them.
|
|
|
|
The mid-run row here differs from the pre-existing one in EXACTLY the named field, so dropping
|
|
that field's comparison makes the job miss the change, skip the abstain, and post its docs-only
|
|
exemption over the row. Reviewed cold: with only the two "everything moves at once" cases, three
|
|
single-clause deletions and a reduction of the whole triple to one comparison all shipped green.
|
|
|
|
The pre-existing row is an OFF-LIST `success` so the first read does not short-circuit — which is
|
|
also the realistic setup, since an off-list row is exactly why the job would be re-deriving.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
pre_row=_PRE,
|
|
midrun_row=midrun_row,
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
f"the mid-run row differed from the pre-existing one only in its {clause}, and the job did "
|
|
f"not notice — that clause of the changed-row comparison is not being evaluated. "
|
|
f"Posted: {posted}. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
|
|
|
|
def test_an_allow_list_with_SEVERAL_entries_honours_the_FIRST_one(tmp_path):
|
|
"""Multi-entry semantics, which every other test leaves unexercised.
|
|
|
|
All the behavioural cases run the singleton `H10_REVIEWERS="timothy"`, under which a loop that
|
|
can only ever match its LAST entry — drop the `break`, add an `else ex_human=no` — behaves
|
|
identically to a correct one. That matters more than it looks: the `::warning::` this change adds
|
|
tells an operator whose reviews are being discarded to ADD THEMSELVES to the list, so a
|
|
two-entry list is the first thing the fix produces, and the first entry is the one that breaks.
|
|
|
|
Docs-only on purpose: if the verdict is inherited the job posts nothing, and if it is not, the
|
|
exemption `success` is posted. So `posted is None` IS the inheritance.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="alice",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
reviewers="alice timothy",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"a verdict from the FIRST entry of a two-entry H10_REVIEWERS was not inherited — membership "
|
|
f"is not scanning the whole list. Posted: {posted}. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_multi_entry_control_the_LAST_entry_is_honoured_too(tmp_path):
|
|
"""Without this, the test above could pass for any reason unrelated to list position."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
reviewers="alice timothy",
|
|
)
|
|
assert posted is None, "the positive control failed, so the first-entry test proves nothing"
|
|
|
|
|
|
def test_an_account_on_NEITHER_entry_is_still_re_derived(tmp_path):
|
|
"""The negative control for the two above: widening the list must not widen it to everyone."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
reviewers="alice timothy",
|
|
)
|
|
assert posted is not None and posted["state"] == "pending", (
|
|
"an account absent from a two-entry allow-list was inherited"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"shape,creator,desc",
|
|
[
|
|
("a null creator (an Actions-token write)", None, "Review-verdict: BLOCKED @ a9e3e23 (base: main)"),
|
|
("a description that is not a verdict", "timothy", "blocked by hand"),
|
|
("a verdict recorded against ANOTHER base", "timothy", "Review-verdict: BLOCKED @ a9e3e23 (base: other)"),
|
|
],
|
|
)
|
|
def test_an_UNATTRIBUTABLE_failure_is_still_RE_DERIVED(tmp_path, shape, creator, desc):
|
|
"""`failure` needs only attributability — but "only attributability" is still a REQUIREMENT.
|
|
|
|
The clause is `[ "$ex_attributable" = yes ] && [ "$ex_state" = "failure" ]`, and the comment at
|
|
the site says plainly that widening it to a bare `[ "$ex_state" = failure ]` is deliberately NOT
|
|
done: that would quietly opt `failure` out of the base binding, which exists because a verdict
|
|
earned against another base is not a verdict for this diff. Every existing `failure` case
|
|
supplied a well-formed, base-matching, non-null-creator row, so the bare-widening mutation
|
|
passed the whole suite.
|
|
|
|
Docs-only on purpose: re-deriving is then visible as an exemption `success` being posted, where
|
|
short-circuiting would post nothing at all.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:failure",
|
|
status_creator=creator,
|
|
status_desc=desc,
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is not None, (
|
|
f"a `failure` with {shape} was left alone — the failure short-circuit has been widened past "
|
|
f"attributability, taking the base binding with it. Log:\n{r.stdout[-1200:]}"
|
|
)
|
|
assert posted["state"] == "success"
|
|
|
|
|
|
def _h10_reviewers_literal():
|
|
"""The allow-list as the workflow actually declares it — read from the shipped file, never
|
|
restated here, so a test cannot pass against a value the gate does not use."""
|
|
# THE WHOLE FILE, not `_classify_step()["run"]` — and the difference is load-bearing, not
|
|
# tidiness. `scripts/lib/h10-reviewers.sh` derives the same literal for the WRITE side by
|
|
# `grep -c` over the whole YAML, and refuses unless it finds exactly one. A guard scoped to the
|
|
# classify step therefore does not walk the same tree as its subject: a second anchored
|
|
# assignment in ANY OTHER `run:` block is invisible here while making the writer refuse every
|
|
# verdict. Measured on this file: injecting one into the last step left this test green and
|
|
# reddened about half the writer suite. Those failures DO carry the library's
|
|
# refusal, which names the count and the variable — what they do NOT do is point at the
|
|
# workflow, since they surface under unrelated test names. No figure is given for how many name
|
|
# it: `grep -c` over pytest output counts LINES rather than failures, and `--tb=line` prints
|
|
# each matching failure twice, so such a count overstates; raising the traceback detail moves it
|
|
# the other way. Any figure is a property of the pytest flags, not of the code. Such an edit
|
|
# reaching `main` would be a repo-wide merge outage — no verdict could be posted and
|
|
# `review-verdict/h10` is required — but the honest claim for THIS guard is narrower: the writer
|
|
# suite already reddens, in the ADVISORY `script-tests` job, so what aligning the populations
|
|
# buys is ONE named guard failure pointing at the workflow, ALONGSIDE those unrelated reds —
|
|
# it adds a diagnosis, it does not replace them.
|
|
# The classify step remains what the GATE runs, and
|
|
# `_classify_step()` is still the right scope for every other assertion about that body.
|
|
body = WORKFLOW.read_text()
|
|
# findall + a SINGLE-ASSIGNMENT assertion, not `search`. `search` returns the FIRST assignment
|
|
# while the shell runs the LAST one executed, so a later or conditional reassignment would leave
|
|
# this guard validating a value the gate does not use — the same extractor-vs-classifier gap
|
|
# ersatztv#774 hit on `POS_RE`.
|
|
found = re.findall(r'^\s*H10_REVIEWERS="([^"]*)"\s*$', body, re.MULTILINE)
|
|
assert found, "review-verdict.yml no longer declares H10_REVIEWERS as a double-quoted literal"
|
|
assert len(found) == 1, (
|
|
f"H10_REVIEWERS is assigned {len(found)} times ({found}); the shell uses the last one "
|
|
"executed, so this guard can no longer tell which value the gate runs with"
|
|
)
|
|
return found[0]
|
|
|
|
|
|
def test_the_H10_REVIEWERS_list_is_glob_free_and_non_empty():
|
|
"""The membership loop word-splits `$H10_REVIEWERS`, so pathname expansion applies to each entry.
|
|
|
|
An entry containing `*`, `?` or `[` would be expanded against the runner's CWD before the
|
|
comparison — normally to itself (no match leaves the word alone), but to a FILENAME whenever one
|
|
happens to match, which silently changes who the gate accepts. Cheaper to forbid the characters
|
|
than to reason about the working directory.
|
|
|
|
Empty is checked too: `for rv in ""` iterates zero times, so an emptied list would make every
|
|
verdict non-inheritable and re-post `pending` over real human verdicts forever.
|
|
|
|
A FILE-TEXT LINT, deliberately, and it claims nothing more. It reads the literal out of the
|
|
shipped workflow so it cannot drift from the gate, but it does not execute the membership loop;
|
|
the behavioural consequences above are argued from the shell's semantics, not measured here.
|
|
"""
|
|
reviewers = _h10_reviewers_literal()
|
|
assert reviewers.strip(), "H10_REVIEWERS is empty — no verdict could ever be inherited"
|
|
for entry in reviewers.split():
|
|
assert re.fullmatch(r"[A-Za-z0-9._-]+", entry), (
|
|
f"H10_REVIEWERS entry {entry!r} is not a plain login; `*`, `?` and `[` are glob "
|
|
"metacharacters in the membership loop's word-splitting"
|
|
)
|
|
|
|
|
|
def test_the_allowlisted_reviewer_in_the_TESTS_is_the_one_the_WORKFLOW_ships():
|
|
"""De-shadowing, not a safety net — and the distinction was a review finding.
|
|
|
|
The BEHAVIOURAL tests do not need this: change the literal to `alice` and
|
|
`test_a_REAL_human_verdict_is_still_NEVER_overwritten` reddens on its own, because a `timothy`
|
|
verdict would stop being inherited. What this pins is the set of tests that use
|
|
`status_creator="timothy"` to reach a DIFFERENT clause — the base-marker cases below. If the
|
|
literal drifted they would start failing via the membership clause instead, still green, while
|
|
the clause they were written for went unexercised.
|
|
"""
|
|
assert "timothy" in _h10_reviewers_literal().split(), (
|
|
"the tests' allow-listed account is no longer in the workflow's H10_REVIEWERS, so every "
|
|
"inheritance test above is asserting against a value the gate does not use"
|
|
)
|
|
|
|
|
|
def test_an_UNRECOGNISED_success_shape_is_re_derived_rather_than_trusted(tmp_path):
|
|
"""The direction-of-test check. Written as 'skip if it looks machine-written', anything novel
|
|
would fall through to TRUSTED. Written as 'skip only if positively identified as human', novel
|
|
shapes are re-derived. This test fails under the first spelling and passes under the second."""
|
|
posted, _ = _run_classify(
|
|
tmp_path, _emitting("ErsatzTV/Program.cs"), status_mode="existing:success", status_creator="", status_desc=""
|
|
)
|
|
assert posted is not None
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
@pytest.mark.parametrize("existing", ["success", "failure"])
|
|
def test_a_REAL_human_verdict_is_still_NEVER_overwritten(tmp_path, existing):
|
|
"""The property the short-circuit exists for, which the fix must not break.
|
|
|
|
SCOPE, because the test name says NEVER: this is about a verdict already VISIBLE AT THE FIRST
|
|
READ. A verdict landing later, inside this run's own write window, is a different route with its
|
|
own (incomplete) guard — see ersatztv#849.
|
|
|
|
`failure` is the sharp case: that is a human saying NO, and an exemption posted over it would
|
|
turn a rejection into a merge. Deliberately paired with a docs-only file list, so the job WOULD
|
|
have posted an exemption `success` had it not short-circuited — without that, a passing test
|
|
would prove only that nothing was posted for some unrelated reason.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode=f"existing:{existing}",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
f"overwrote a human '{existing}' verdict written by timothy — the never-overwrite property "
|
|
"has been lost while fixing route 3"
|
|
)
|
|
# NO "the diagnostic did not fire" ASSERTION HERE. Both parametrizations exit at a
|
|
# short-circuit two guards earlier and never reach the diagnostic at all, so such an assertion
|
|
# is true by construction and cannot fail. The annotation it would guard has since been
|
|
# withdrawn; the note is kept because a caption claiming a proof is worse than no assertion.
|
|
|
|
|
|
def test_a_PENDING_status_from_a_previous_run_is_replaced_normally(tmp_path):
|
|
"""`pending` was never short-circuited and must stay that way, or a PR that becomes exempt after
|
|
an earlier pending run could never reach `success`."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc="Awaiting review verdict for a9e3e23",
|
|
)
|
|
assert posted is not None
|
|
assert posted["state"] == "success"
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["web/package.json", "web/package-lock.json"])
|
|
def test_the_npm_manifests_are_NOT_exempt(tmp_path, path):
|
|
"""Including these in the exemption is a Blocker.
|
|
|
|
`renovate.json` sets `enabledManagers: ["nuget", "github-actions", "dockerfile"]`, so Renovate does
|
|
not manage npm here at all — the entry bought nothing. Meanwhile `package.json` carries `scripts`
|
|
that CI EXECUTES (`npm ci`, `npm run build`), so exempting it lets a hijacked bot branch run
|
|
arbitrary shell in CI while every path still matches a "manifest" allow-list. Widening an exemption
|
|
to a code-execution vector for no operational benefit is strictly worse than the hole being closed.
|
|
"""
|
|
posted, _ = _run_classify(tmp_path, _emitting(path), author="renovate")
|
|
assert posted is not None
|
|
assert posted["state"] == "pending", (
|
|
f"{path} was exempted; package.json scripts execute in CI, and npm is not a managed ecosystem in this repo"
|
|
)
|
|
|
|
|
|
# --- The `grep -q` + `pipefail` inversion ---------------------------------------------------------
|
|
#
|
|
# `grep -q` exits at its FIRST match, so the upstream writer takes SIGPIPE (141) once the path list
|
|
# exceeds the pipe buffer. Under `set -o pipefail` the pipeline is then a FAILURE even though grep
|
|
# MATCHED, inverting every guard built on `printf … | grep -q`. Reproduced at 171KB / 1901 paths,
|
|
# comfortably inside the enumerator's 2000-file cap.
|
|
#
|
|
# These are the regression guards. Every earlier case used a handful of short paths, far below the
|
|
# buffer, so the whole class was invisible. The construct PREDATES #698, so `main` carried this hole
|
|
# with no retarget or bot account required.
|
|
|
|
|
|
def _many_docs(n=1900):
|
|
return [f"docs/{'d' * 40}-{i:040d}.md" for i in range(n)]
|
|
|
|
|
|
def test_a_LARGE_pr_with_a_code_file_is_not_classified_docs_only(tmp_path):
|
|
"""The code file goes FIRST so `grep -qv` matches immediately and the writer is left with ~171KB
|
|
still to push — the exact shape that produced exit 141 and `docs_only=yes`."""
|
|
posted, r = _run_classify(tmp_path, _emitting("A.cs", *_many_docs()))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending", (
|
|
"a PR containing A.cs was exempted as docs-only because the classification pipeline was "
|
|
f"inverted by SIGPIPE on a large file list (desc={posted['description']!r})"
|
|
)
|
|
|
|
|
|
def test_a_LARGE_pr_touching_a_PROTECTED_path_still_voids_the_exemptions(tmp_path):
|
|
"""The worse direction: here the inversion makes the PROTECTED guard MISS, so a PR editing the
|
|
gate's own workflow falls through to the docs-only exemption and self-exempts."""
|
|
posted, r = _run_classify(tmp_path, _emitting(".gitea/workflows/review-verdict.yml", *_many_docs()))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending", (
|
|
"a PR editing .gitea/workflows/ was exempted — the protected-path guard was skipped on a "
|
|
f"large file list (desc={posted['description']!r})"
|
|
)
|
|
|
|
|
|
def test_a_LARGE_bot_pr_with_a_code_file_is_not_manifest_exempt(tmp_path):
|
|
"""The same inversion reached through the third guard added by this change."""
|
|
posted, r = _run_classify(tmp_path, _emitting("A.cs", *_many_docs()), author="renovate")
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_positive_control_a_LARGE_genuinely_docs_only_pr_IS_still_exempt(tmp_path):
|
|
"""Without this the three tests above could pass because large lists now fail outright, which
|
|
would be a merge deadlock rather than a fix."""
|
|
posted, r = _run_classify(tmp_path, _emitting(*_many_docs()))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "success", (
|
|
f"a large but genuinely docs-only PR lost its exemption (desc={posted['description']!r})"
|
|
)
|
|
|
|
|
|
def test_a_human_verdict_landing_MID_RUN_is_not_overwritten(tmp_path):
|
|
"""ersatztv#706, the half that IS mitigated here.
|
|
|
|
The first status read finds nothing, so the job proceeds to classify. A reviewer then posts a human
|
|
`failure` (BLOCKED). Without the re-read immediately before the POST, this docs-only PR would post
|
|
an exemption `success` OVER an explicit human rejection — turning a "no" into a merge, which is the
|
|
worst outcome in this class.
|
|
|
|
Note precisely what this does and does not prove: it pins the re-read, not atomicity. A verdict
|
|
landing between the re-read and the POST is still lost — there is no compare-and-set on Gitea's
|
|
status API. That remainder is #706, not a claim made here.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="appears-on-read:2")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"an exemption success was posted over a human BLOCKED verdict that landed while the job was "
|
|
"classifying — the pre-POST re-read is missing or ineffective"
|
|
)
|
|
|
|
|
|
def test_positive_control_no_late_verdict_still_posts_normally(tmp_path):
|
|
"""Without this, the test above would pass against a job that had simply stopped posting."""
|
|
posted, _ = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="none")
|
|
assert posted is not None and posted["state"] == "success"
|
|
|
|
|
|
# --- The PROTECTED branch must actually EXECUTE, not merely coincide with the right answer --------
|
|
#
|
|
# `count_matching` was called before its definition, so it was `command not found` on every run and
|
|
# the PROTECTED branch never fired. Three "protected path" tests
|
|
# passed anyway, because a protected path is also not a manifest and not docs-only, so the job reached
|
|
# `pending` down a different route. Asserting the STATE could not see it; the guard was dead and the
|
|
# suite was green.
|
|
#
|
|
# The lesson generalises: when several branches produce the same outcome, asserting the outcome cannot
|
|
# tell you which branch ran. Assert the DISCRIMINATOR — here the reason string the branch writes.
|
|
|
|
|
|
def test_a_protected_path_is_rejected_BY_THE_PROTECTED_BRANCH(tmp_path):
|
|
posted, r = _run_classify(tmp_path, _emitting("scripts/evil.sh", "docs/a.md"))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending"
|
|
# The DISCRIMINATOR is the job's `Decision:` line, not the status description: for `pending` the
|
|
# description is always "Awaiting review verdict for <sha>", identical no matter which branch
|
|
# produced it. Asserting on the description fails against a WORKING guard — the assertion has
|
|
# to be aimed at something that actually differs per branch.
|
|
assert "protected" in r.stdout.lower(), (
|
|
"the PR was not exempted, but NOT via the protected-path branch — it reached the same verdict "
|
|
f"by another route, so that guard may be dead. Decision log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_protected_path_defeats_the_BOT_exemption_by_the_protected_branch(tmp_path):
|
|
"""A bot PR whose every path IS a manifest, plus one protected path. Without a live PROTECTED
|
|
branch this still lands on `pending` (the protected file is not a manifest), so again only the
|
|
reason string distinguishes a working guard from a dead one."""
|
|
posted, r = _run_classify(
|
|
tmp_path, _emitting("Directory.Packages.props", ".gitea/workflows/renovate.yml"), author="renovate"
|
|
)
|
|
assert posted is not None
|
|
assert posted["state"] == "pending"
|
|
assert "protected" in r.stdout.lower(), (
|
|
f"the bot exemption was refused, but not by the PROTECTED branch:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"paths,author",
|
|
[
|
|
(("docs/a.md",), "timothy"),
|
|
(("Directory.Packages.props",), "renovate"),
|
|
(("ErsatzTV/Program.cs",), "timothy"),
|
|
(("scripts/evil.sh",), "timothy"),
|
|
],
|
|
)
|
|
def test_the_classify_step_runs_without_SHELL_ERRORS(tmp_path, paths, author):
|
|
"""A cheap, general trap-catcher for the whole step.
|
|
|
|
`command not found`, `integer expression expected`, `unbound variable` — each of these silently
|
|
skips a branch inside an `if`/`elif` (the condition just evaluates false) while the job exits 0 and
|
|
posts a plausible status. That is precisely how the dead PROTECTED guard survived a green suite.
|
|
|
|
Scope, stated so this is not mistaken for more than it is: it catches guards that die NOISILY on
|
|
stderr. It is NOT a general liveness check — a clean mutation such as hardcoding `n_protected=0`
|
|
emits none of these and passes here. That case is covered by the branch-discriminator test above,
|
|
which is the actual liveness guard; this one is the cheap net for the whole error-emitting family.
|
|
"""
|
|
_, r = _run_classify(tmp_path, _emitting(*paths), author=author)
|
|
bad = [
|
|
ln
|
|
for ln in r.stderr.splitlines()
|
|
if "command not found" in ln
|
|
or "integer expression expected" in ln
|
|
or "unbound variable" in ln
|
|
or "syntax error" in ln
|
|
]
|
|
assert not bad, f"the classification step emitted shell errors, so a guard is not running: {bad}"
|
|
|
|
|
|
def test_a_human_verdict_formed_against_ANOTHER_BASE_is_not_inherited(tmp_path):
|
|
"""The sha-binding is escapable through the HUMAN verdict path.
|
|
|
|
Get a genuine `success` on head H while it targets scratch base S (benign diff there), then
|
|
retarget H onto `main`, where its diff contains unreviewed code. Creator is real, prefix is real,
|
|
so the short-circuit preserved it — a green required check over code nobody reviewed. The
|
|
merge-consent hook compares the base, but that is advisory; a merge through the Gitea UI or API
|
|
only sees the status.
|
|
|
|
`post-review-verdict.sh` already records the base it reviewed (`(base: …)`, ersatztv#632); this
|
|
asserts the gate actually READS it.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: probe/scratch-base)",
|
|
)
|
|
assert posted is not None, (
|
|
"a human verdict formed against a DIFFERENT base was inherited unchanged — the reviewed diff "
|
|
f"is not this PR's diff. Log:\n{r.stdout[-600:]}"
|
|
)
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_a_human_verdict_for_THIS_base_is_still_honoured(tmp_path):
|
|
"""The positive control the test above needs: matching bases must still short-circuit, or the
|
|
check has simply broken every verdict."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)",
|
|
)
|
|
assert posted is None, "a verdict formed against THIS base was re-derived; the base check is too strict"
|
|
|
|
|
|
def test_a_LEGACY_verdict_with_no_recorded_base_is_still_honoured(tmp_path):
|
|
"""Verdicts predating ersatztv#632 carry no `(base: …)`. Absent is deliberately not treated as a
|
|
mismatch: re-deriving over one would un-approve a genuinely reviewed head. Only a base that is
|
|
PRESENT and DIFFERENT is rejected."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23",
|
|
)
|
|
assert posted is None
|
|
|
|
|
|
def test_an_APPENDED_base_cannot_override_the_real_one(tmp_path):
|
|
"""The description is attacker-influencable by anyone who can POST a status — the registry
|
|
credential's own route closed with #697, but `GITEA_TOKEN`, `RENOVATE_TOKEN`, and a
|
|
collaborator's own token still can — so the parse must not be trickable into reading a second,
|
|
appended base.
|
|
|
|
This defeated the greedy `##` parse. The implementation no longer parses at all — it requires the
|
|
description to END with the exact literal marker AND to contain exactly one marker — so a second
|
|
appended marker makes the count 2 and is rejected outright.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: probe/scratch) (base: main)",
|
|
)
|
|
assert posted is not None, (
|
|
"an appended '(base: main)' overrode the real recorded base, so a verdict formed elsewhere was "
|
|
f"inherited. Log:\n{r.stdout[-600:]}"
|
|
)
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
def test_an_EMPTY_recorded_base_is_treated_as_a_mismatch(tmp_path):
|
|
"""`(base: )` is not 'absent' — it is present and not equal to the PR's base, so it fails closed
|
|
rather than being waved through by the legacy-verdict allowance."""
|
|
posted, _ = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: )",
|
|
)
|
|
assert posted is not None and posted["state"] == "pending"
|
|
|
|
|
|
def test_a_branch_name_containing_a_PAREN_cannot_truncate_into_the_current_base(tmp_path):
|
|
"""The sharpest of these escapes: it needs no forgery and no #697.
|
|
|
|
`main)evil` is a VALID git branch name (`git check-ref-format --branch 'main)evil'` succeeds). A
|
|
genuine verdict earned while head H targeted it is written `(base: main)evil)`. Any implementation
|
|
that extracts the value by truncating at the first `)` gets exactly `main`, matches a PR that now
|
|
targets `main`, and inherits a verdict covering a completely different diff.
|
|
|
|
An earlier comment in the workflow asserted that a `)` in a branch name "mismatches — safe
|
|
direction". That was generalised from `feat/foo)bar` (which does mismatch) and is false for every
|
|
branch whose name STARTS with the target base. Hence the rule the code now follows: compare against
|
|
the exact expected literal, never parse a value out of attacker- or user-influenced text.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("ErsatzTV/Program.cs"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: main)evil)",
|
|
)
|
|
assert posted is not None, (
|
|
"a verdict recorded against branch 'main)evil' was inherited by a PR targeting 'main' — the "
|
|
f"base value is being truncated at the first ')'. Log:\n{r.stdout[-600:]}"
|
|
)
|
|
assert posted["state"] == "pending"
|
|
|
|
|
|
# --- ersatztv#706: the retarget fence (race 1) and the post-write repair (race 2) -----------------
|
|
#
|
|
# These three stay 1.25.4-dated on purpose, but NOT all for the same reason — a blanket reason fits
|
|
# only the third. The first two were measured on probe PR #722, and a scratch PR is still raisable;
|
|
# they were simply not re-run (ersatztv#747 did not re-measure the `pull_request_target` half
|
|
# either). Only the third — Gitea
|
|
# auto-cancelling superseded `push` runs — used the disposable scratch-BRANCH push route, and that
|
|
# route is genuinely gone. See `ci.verdict-write-retarget-fence`, re-checked 2026-09-02
|
|
# (ersatztv#869).
|
|
#
|
|
# Race 1 was reproduced LIVE before any of this was written, because the fix turns on which Gitea
|
|
# behaviours are real rather than on what the docs imply. Measured on Gitea 1.25.4:
|
|
# * `pull_request_target` runs for one PR are NOT auto-cancelled — probe PR #722, run 7520
|
|
# (`opened`) ran to completion 20s AFTER run 7521 (`synchronize`) started. The older run finished
|
|
# LAST, which is exactly the mechanism by which a stale classification overwrites a fresh one.
|
|
# * A NON-CANCELLING concurrency group does not serialize them either: with the group active, runs
|
|
# 7528 and 7529 still overlapped and 7528 ended 36s after 7529 began. #706's headline proposal is
|
|
# therefore refuted, not merely unattractive.
|
|
# * Gitea DOES auto-cancel superseded `push` runs on a branch — a control workflow carrying no
|
|
# `concurrency:` key at all showed that — which is why the concurrency-group probe was confounded
|
|
# until the control separated the two.
|
|
# The fence keys on the timeline's `change_target_branch` COUNT because the branch NAME is
|
|
# ABA-vulnerable (`main → S → main` reads `main` at both ends), while the count is monotonic.
|
|
|
|
|
|
def _posted_sequence(tmp_path):
|
|
"""Every status POST the job made, in order — not just the last one."""
|
|
f = tmp_path / "posted_all.jsonl"
|
|
return [json.loads(line) for line in f.read_text().splitlines() if line.strip()] if f.exists() else []
|
|
|
|
|
|
def test_a_RETARGET_DURING_the_run_posts_NOTHING(tmp_path):
|
|
"""THE race-1 test. A docs-only PR that would otherwise be exempted is retargeted while the job
|
|
classifies, so this run's answer describes a base the PR may no longer target. It must write
|
|
nothing at all and leave the field to the successor run the retarget's `edited` event queues."""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="moves:0,1")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"a stale run posted its classification after the PR was retargeted underneath it — the "
|
|
f"retarget fence did not fire. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
# Assert the DISCRIMINATOR, not just the outcome: several unrelated failures also end in "posted
|
|
# nothing", so the state alone cannot tell a working fence from a broken job.
|
|
assert "retargeted while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the retarget fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_a_QUIET_run_still_posts_its_exemption(tmp_path):
|
|
"""Without this, the test above passes against a job that simply never posts."""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none")
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"an undisturbed docs-only PR lost its exemption. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def _fold_continuations(lines):
|
|
r"""Join backslash-continued shell lines the way bash does: delete the backslash AND the newline,
|
|
joining with NOTHING.
|
|
|
|
The exit-status detector is per-line, so a `jq \` whose flag sits on the next line would slip
|
|
past it. Folding first closes that — but only if the fold matches the shell, and the separator is
|
|
where it goes wrong. Joining with a SPACE is not cosmetic: it turns
|
|
`jq -\` + `e '.'` into `jq - e '.'` (two tokens, no flag) where bash produces `jq -e '.'`.
|
|
|
|
WHICH SPLITS ARE ACTUALLY REACHABLE, measured rather than assumed, because the obvious
|
|
construction is not one. Bash does NOT strip the continuation line's leading whitespace, so
|
|
`echo -\` followed by an INDENTED `e X` prints `- e X` — three tokens, no flag re-formed, and
|
|
`jq -\` + indented `e` exits 3 rather than acting as `-e`. The reachable case is a continuation
|
|
whose next line begins at COLUMN 0, and in this file that is not exotic: YAML strips the `run:`
|
|
block's common indentation (10 spaces here), so any line written at the block's base indent runs
|
|
at column 0. Verified end to end: a mutant of that exact shape parses as YAML, passes `bash -n`,
|
|
forms a live `jq -e` — and reddens the guard under this fold while passing under the space-joining
|
|
one.
|
|
|
|
NO `rstrip()`, for the same reason. A backslash followed by trailing whitespace is not a
|
|
continuation to bash — the backslash escapes the SPACE — and tolerating it was the one route by
|
|
which this fold could DESTROY a `jq` token rather than fabricate one: `ok=y\` plus invisible
|
|
trailing spaces, then a dedented `jq -e ...`, folds to `okyjq -e`, and `\bjq\b` finds no word
|
|
boundary after the `y`. Every other divergence here over-joins, which can only ever cost a false
|
|
positive. Matching the shell exactly is both simpler and the only direction that cannot lose a
|
|
`jq`.
|
|
"""
|
|
joined, buf = [], ""
|
|
for ln in lines:
|
|
if ln.endswith("\\"):
|
|
buf += ln[:-1]
|
|
else:
|
|
joined.append(buf + ln)
|
|
buf = ""
|
|
if buf:
|
|
joined.append(buf)
|
|
return joined
|
|
|
|
|
|
def _uses_jq_exit_status(line: str) -> bool:
|
|
"""True if `line` invokes jq with the exit-status flag, in any spelling of the FLAG.
|
|
|
|
Scope, stated because the obvious reading is wider than the truth: this is per-line and knows
|
|
nothing about shell continuations. `jq \\` on one line and `-e` on the next is a real evasion,
|
|
and it is closed by the CALLER, which folds continuations before scanning — not here.
|
|
|
|
THIS IS THE THIRD VERSION AND IT IS DELIBERATELY THE DUMBEST. The first was
|
|
`"jq -e" in line`, which missed `jq -re`, `jq -e` and `jq --exit-status` (it caught `jq -er`
|
|
only because `jq -e` is a substring of it). The second parsed
|
|
the LEADING option tokens and stopped at the first non-option, on the reasoning that everything
|
|
after the jq program is source — and that reasoning bought two more false negatives:
|
|
`jq --argjson e 1 -e '.'` (the walk stops at the bare `e` argument) and `jq '.a' -e` (jq accepts
|
|
options after the filter). Three defects in one string-matching predicate is this repo's
|
|
documented signal to stop adding cases and remove the cleverness instead.
|
|
|
|
So: scan EVERY whitespace token after each `jq`, with no attempt to find where the program
|
|
begins. That over-approximates — a jq program containing a token that looks like a short flag
|
|
with an `e` in it would trip this — and the over-approximation is the SAFE direction. It guards
|
|
one short function we own: a false positive costs a rewrite, a false negative restores the
|
|
jq-1.6 exit-status divergence inside the code that writes a required status.
|
|
|
|
BE EXACT ABOUT WHY THERE IS NO FALSE POSITIVE TODAY, because the loose version of that sentence
|
|
is false. The guarded function DOES contain `-`-prefixed tokens carrying an `e`: `-eq`, `-ne`
|
|
and `-le`, the shell test operators. They are invisible to this only because none of them shares
|
|
a LINE with a `jq` invocation, and the scan starts at `jq`. Measured: of its five jq-bearing
|
|
lines, none carries a `-`-prefixed token after `jq` other than `-r`. A future
|
|
`if [ "$(... | jq -r .)" -eq 0 ]` would trip this guard for no real reason — and the fix then is
|
|
to split the line, not to make the detector clever again.
|
|
"""
|
|
for m in re.finditer(r"\bjq\b", line):
|
|
for tok in line[m.end() :].split():
|
|
if tok == "--exit-status":
|
|
return True
|
|
if tok.startswith("-") and not tok.startswith("--") and "e" in tok[1:]:
|
|
return True
|
|
return False
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"raw,folded",
|
|
[
|
|
# The reachable shape: the continuation's next line is at column 0, which is what a line
|
|
# written at the `run:` block's base indent becomes once YAML strips it. Bash re-forms the
|
|
# token; so must the fold, or the detector sees `jq - e` and reports nothing.
|
|
([" x=$(... | jq -\\", "e 'length')"], [" x=$(... | jq -e 'length')"]),
|
|
# An INDENTED continuation does NOT re-form the token — bash keeps the leading whitespace —
|
|
# so the fold must not invent one either. Measured: `echo -\` + indented `e X` prints `- e X`.
|
|
([" x=$(... | jq -\\", " e 'length')"], [" x=$(... | jq - e 'length')"]),
|
|
(["a\\", "b\\", "c"], ["abc"]), # two consecutive continuations
|
|
(["plain", "lines"], ["plain", "lines"]), # nothing to fold
|
|
(["trailing\\"], ["trailing"]), # continuation with nothing after it
|
|
# A backslash followed by whitespace is NOT a continuation — bash escapes the space.
|
|
# Nothing bound this before, so dropping the shell-faithfulness survived every other
|
|
# case while opening the one route by which the fold can HIDE a `jq` token instead of
|
|
# inventing one: `ok=y\` + trailing spaces then a dedented `jq -e` folds to `okyjq -e`,
|
|
# and `\\bjq\\b` finds no boundary after the `y`.
|
|
(["a\\ ", "b"], ["a\\ ", "b"]),
|
|
],
|
|
)
|
|
def test_the_continuation_fold_matches_the_SHELL(raw, folded):
|
|
r"""It joins with NOTHING, because that is what `\<newline>` removal does.
|
|
|
|
Joining with a space is the intuitive choice and it silently defeats the guard downstream: it
|
|
turns `jq -\` + `e` into `jq - e`, two tokens carrying no flag, where bash produces `jq -e`.
|
|
"""
|
|
assert _fold_continuations(raw) == folded
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"line,expected",
|
|
[
|
|
(""" if ! printf '%s' "$raw" | jq -e 'all(.[]; .a)' ; then""", True),
|
|
(""" x=$(jq --exit-status '.a' <<< "$raw")""", True),
|
|
(""" x=$(jq -re '.a')""", True),
|
|
(""" x=$(jq -er '.a')""", True),
|
|
(""" x=$(jq -e '.a')""", True),
|
|
# The two the leading-token parser missed.
|
|
(""" x=$(jq --argjson e 1 -e '.')""", True),
|
|
(""" x=$(jq '.a' -e)""", True),
|
|
(""" n=$(printf '%s' "$raw" | jq -r 'length')""", False),
|
|
(''' kind=$(printf '%s' "$raw" | jq -r 'type' 2>/dev/null) || kind=""''', False),
|
|
# The jq PROGRAM may legitimately contain an `e`, or a `-e` inside a string. These pass
|
|
# because neither carries a `-`-prefixed token at all — NOT because the scan stops at the
|
|
# program. It does not: that was the v2 reasoning this file withdrew, and restating it here
|
|
# would invite someone to "restore" the leading-token walk and its two false negatives.
|
|
(""" bad=$(jq -r '[.[] | select(.type != "e")] | length')""", False),
|
|
(""" m=$(jq -r --arg e x '[.[] | select(.type == $e)] | length')""", False),
|
|
],
|
|
)
|
|
def test_the_jq_exit_status_detector_reads_FLAGS_not_a_substring(line, expected):
|
|
"""The detector is itself a string-matching predicate, so it gets its own cases.
|
|
|
|
A bare `"jq -e" in line` misses `jq -re`, `jq -er`, `jq -e` and
|
|
`jq --exit-status` — three spellings of the same request (it catches `jq -er`, since `jq -e`
|
|
is a substring of it) — and a guard a rename walks straight
|
|
through is not a guard. The false-positive rows matter as much: an `e` inside the jq PROGRAM is
|
|
not a flag, and a detector that reddens on one would make the rule unusable and get waived.
|
|
"""
|
|
assert _uses_jq_exit_status(line) is expected
|
|
|
|
|
|
def test_count_pr_mutations_uses_NO_jq_e(tmp_path):
|
|
"""The timeline walk reads jq output as VALUES; `jq -e`'s exit status is banned inside it.
|
|
|
|
The function's own comment states the rule and the reason: `jq -e` reports the truthiness of its
|
|
LAST OUTPUT, and over empty input jq 1.6 exits 0 where >= 1.7 exits 4 — the divergence that
|
|
already took this workflow's enforced gate down once (ersatztv#647), on a runner that still ships
|
|
1.6. Every tally here is therefore read with `jq -r` and range-checked in shell.
|
|
|
|
This exists because the rule was BROKEN one commit after it was quoted (ersatztv#803 added a
|
|
row-validation guard using `jq -e`, three lines below the comment forbidding it) and nothing
|
|
caught it. A convention stated only in prose is one refactor from being false, and this one is
|
|
function-scoped: `jq -e` is legitimate elsewhere in this same workflow and in the hook, so a
|
|
file-wide grep would be wrong.
|
|
|
|
Comments are stripped, so the paragraphs explaining the rule do not redden it.
|
|
"""
|
|
body = _classify_step()["run"]
|
|
lines = [ln for ln in body.splitlines() if not ln.lstrip().startswith("#")]
|
|
fn, depth, inside = [], 0, False
|
|
for ln in lines:
|
|
if "count_pr_mutations() {" in ln:
|
|
inside = True
|
|
if inside:
|
|
fn.append(ln)
|
|
depth += ln.count("{") - ln.count("}")
|
|
if depth == 0 and len(fn) > 1:
|
|
break
|
|
assert inside and depth == 0, (
|
|
"could not locate the `count_pr_mutations` body in the classify step — if it was renamed, "
|
|
"rename it here too rather than deleting this guard"
|
|
)
|
|
# A BALANCED PREFIX IS NOT THE WHOLE FUNCTION. `depth == 0` is reached by the real closing brace
|
|
# today, but it would also be reached early by any line with a net-negative brace count, and the
|
|
# assertion above cannot tell the two apart — so a `jq -e` past that point would be invisible
|
|
# while the guard reported success.
|
|
#
|
|
# The check is STRUCTURAL, not a pinned statement. Asserting that the body contains
|
|
# `page=$(( page + 1 ))` would work — it is a real line near the end — and would also redden
|
|
# on any refactor of the increment, for no reason connected to
|
|
# this guard. A guard that cries wolf gets waived, which is how the repo already has to
|
|
# document one CI job as "do nothing". Requiring the capture to END on the closing brace says
|
|
# the same thing about extent while surviving every edit to the body.
|
|
assert fn[-1].strip() == "}", (
|
|
"the extracted body does not end on `count_pr_mutations`'s closing brace, so it is a PREFIX "
|
|
f"of the function rather than the whole of it — this guard would be scanning only part of "
|
|
f"what it claims to cover. Last captured line: {fn[-1]!r}"
|
|
)
|
|
joined = _fold_continuations(fn)
|
|
offenders = [ln.strip() for ln in joined if _uses_jq_exit_status(ln)]
|
|
assert not offenders, (
|
|
"`jq -e` appeared inside `count_pr_mutations`, whose own comment bans it because the runner "
|
|
"ships jq 1.6 and `-e`'s exit status over empty input diverges there (ersatztv#647/#803). "
|
|
"Read the value with `jq -r` and range-check it in shell instead:\n " + "\n ".join(offenders)
|
|
)
|
|
|
|
|
|
def test_a_HEAD_ABA_DURING_the_run_posts_NOTHING(tmp_path):
|
|
"""THE ersatztv#803/#664 test: `H1 -> H2 -> H1` across the enumeration.
|
|
|
|
This is the case no sha comparison can see, and the reason the fence keys on a COUNT. The head
|
|
is force-pushed away and back while the job classifies, so `$SHA` still equals `.head.sha` at
|
|
both ends — `pr-changed-files.sh` exits 0, having observed no movement — while the
|
|
middle pages were served from `H2`. A mixed file list can then produce a docs-only exemption
|
|
that no single head ever justified.
|
|
|
|
TWO push events, not one, because that is what the ABA costs: away, and back. The retarget axis
|
|
is held at `none` deliberately, so a pass here cannot be the BASE fence firing.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none", push_mode="moves:0,2")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"the job posted an exemption for a head that was force-pushed away and back underneath it "
|
|
f"— the head-mutation fence did not fire. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
# The DISCRIMINATOR, and specifically the head one: the retarget arm also ends in "posted
|
|
# nothing", so without this the test would pass against a job whose base fence fired for the
|
|
# wrong reason.
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "retargeted while this job was classifying" not in r.stdout, (
|
|
"the BASE fence fired on a run whose base never moved, so this test is not measuring the "
|
|
f"head axis. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("terminator", ["null", "[]"])
|
|
def test_the_head_fence_holds_under_EITHER_terminator_shape(tmp_path, terminator):
|
|
"""The `[]` and `null` walk-exits are separate arms, and the head axis must publish from both.
|
|
|
|
Deleting `hp_count=$ptotal` from the empty-ARRAY arm alone left every fence test green, because
|
|
every one of them terminated on the default `null`. The head fence would then be silently dead
|
|
for any timeline ending in `[]` — a shape this code explicitly supports and this server really
|
|
produces on a sibling endpoint.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="moves:0,2",
|
|
timeline_terminator=terminator,
|
|
)
|
|
assert posted is None, (
|
|
f"an exemption was granted over an ABA on a timeline terminating with `{terminator}` — the "
|
|
f"head count is not published from that arm. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_head_arm_DEFERS_to_the_trust_guard_when_the_second_count_fails(tmp_path):
|
|
"""An untrusted re-count is "cannot tell", never "the head moved".
|
|
|
|
Both outcomes post nothing, so only the DIAGNOSTIC separates them — and the difference is not
|
|
cosmetic: the untrusted branch is an `::error::` saying an exemption cannot be shown to have been
|
|
computed at a single head and that a later run only helps if the cause was transient, while the
|
|
head arm is a `::notice::` promising a successor run that the `synchronize` event has queued.
|
|
Reporting a push that was never observed would send a reader looking for a force-push that did
|
|
not happen.
|
|
|
|
Dropping the `rt_ok` conditions from the head arm leaves the rest of the suite green, because
|
|
every other test has both counts trusted (guard is a no-op) or both untrusted (`pushes_before`
|
|
is 0, so the arm cannot fire). Only a trusted-then-untrusted pair separates them.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="trusted-then-unreadable",
|
|
push_mode="stable:3",
|
|
)
|
|
_assert_withheld(
|
|
tmp_path, r, "test_the_head_arm_DEFERS_to_the_trust_guard_when_the_second_count_fails", expect_rc=0
|
|
)
|
|
assert "Could not establish a trusted retarget/push count" in r.stdout, (
|
|
f"the untrusted re-count was not reported as such. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "head branch was pushed while this job was classifying" not in r.stdout, (
|
|
"an untrusted re-count was reported as observed head movement — the head arm is firing "
|
|
f"without its trust guard. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_SINGLE_head_push_during_the_run_also_posts_NOTHING(tmp_path):
|
|
"""One push is the ordinary one-way move, and the fence covers it too.
|
|
|
|
`pr-changed-files.sh` already fails closed on this via its own sha comparison, so the exemption
|
|
was withheld before #803 — but through a DIFFERENT mechanism, one that reports an enumeration
|
|
error rather than a handoff. Asserting the fence arm here pins that the head axis is not
|
|
narrowed to "two or more pushes" by some future edit reasoning that one-way is already covered.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none", push_mode="moves:0,1")
|
|
assert r.returncode == 0, f"the job died rather than abstaining cleanly: {r.stderr[-600:]}"
|
|
assert posted is None, f"an exemption was posted for a PR pushed mid-classification. Log:\n{r.stdout[-900:]}"
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_PR_PUSHED_BEFORE_the_run_but_QUIET_during_it_is_STILL_exempt(tmp_path):
|
|
"""The head fence keys on MOTION during this run, never on "has ever been pushed".
|
|
|
|
This is the test that would go red on the most plausible wrong implementation. EVERY pull
|
|
request has a non-zero `pull_push` count — it is created by a push, and PR #802 carries
|
|
eighteen — so a fence keying on the count being non-zero, rather than on it CHANGING, would
|
|
withhold the exemption from every PR that has ever existed. That is not a subtle regression: it
|
|
is the #751 shape, where a fence shipped and silently refused every exemption on the instance.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none", push_mode="stable:18")
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"a PR with a settled, non-zero push history was refused its exemption — the head fence is "
|
|
f"testing the count's VALUE instead of its MOVEMENT. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
# Each row: (timeline_mode, push_mode, arm expected to fire, arm that must stay silent).
|
|
# BOTH DIRECTIONS ARE POSED, because they fail to different mutations and only one of them is
|
|
# reachable at all. The arms are evaluated base-first, so a tally that folds PUSHES into the retarget
|
|
# total is caught only by the push-moves row (the base arm fires and pre-empts the head message);
|
|
# folding retargets into the push total is NOT observable from here for the same ordering reason,
|
|
# and is deliberately not claimed to be. Stating that is the point — asserting the base-moves row
|
|
# alone while claiming to catch the folding mutation stays GREEN under exactly that mutation.
|
|
_AXIS_ROWS = [
|
|
(
|
|
"moves:0,1",
|
|
"stable:18",
|
|
"retargeted while this job was classifying",
|
|
"head branch was pushed while this job was classifying",
|
|
),
|
|
(
|
|
"stable:2",
|
|
"moves:0,2",
|
|
"head branch was pushed while this job was classifying",
|
|
"retargeted while this job was classifying",
|
|
),
|
|
]
|
|
|
|
|
|
@pytest.mark.parametrize("tl_mode,push_mode,expected,silent", _AXIS_ROWS)
|
|
def test_the_head_and_base_axes_are_counted_SEPARATELY(tmp_path, tl_mode, push_mode, expected, silent):
|
|
"""One walk, two tallies, and neither may be read for the other.
|
|
|
|
The cheapest wrong implementation reuses one tally for both axes; it passes every single-axis
|
|
test above, because with only one axis in motion a shared counter still moves. What distinguishes
|
|
it is holding one axis STABLE AND NON-ZERO while the other moves, then demanding that the arm
|
|
which reports is the one that actually moved.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode=tl_mode, push_mode=push_mode)
|
|
assert posted is None, f"neither fence fired. Log:\n{r.stdout[-900:]}"
|
|
assert expected in r.stdout, f"the moving axis was not the one reported. Log:\n{r.stdout[-900:]}"
|
|
assert silent not in r.stdout, (
|
|
f"a STABLE axis was reported as having moved — the two counts are sharing a tally. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_push_landing_BEYOND_page_1_is_still_counted(tmp_path):
|
|
"""The walk must ACCUMULATE across pages, not answer from the first one.
|
|
|
|
Timeline rows are ordered ASCENDING, so the newest events — exactly the ones a fence looks for —
|
|
are furthest from page 1. A walk that read page 1 and stopped would see a stable zero on both
|
|
counts and let the ABA through, while passing every other fence test in this file: until the
|
|
double grew `timeline_pages`, every event it served was on page 1.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="moves:0,2",
|
|
timeline_pages=3,
|
|
)
|
|
assert posted is None, (
|
|
"an exemption was granted over an ABA whose push events lived past page 1 — the walk is "
|
|
f"answering from the first page. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_a_MULTI_PAGE_quiet_timeline_still_exempts(tmp_path):
|
|
"""Without this, the test above passes against a walk that refuses every multi-page timeline."""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="stable:2",
|
|
timeline_pages=3,
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"a quiet PR with a multi-page timeline lost its exemption. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
# THE FILTERED-INTERMEDIATE-PAGE DEFEAT (ersatztv#870), on both axes.
|
|
#
|
|
# The walk used to read an empty page past page 1 as proof of exhaustion. Gitea does not mean that:
|
|
# `ListIssueCommentsAndTimeline` applies the LIMIT/OFFSET in `FindComments` and drops
|
|
# `CommentTypeCode` rows and inaccessible cross-references AFTERWARDS, into a nil slice that
|
|
# serializes as bare `null`. So a page of 50 inline review comments — which a PR author can create on
|
|
# their own PR — comes back looking exactly like the end of the list while later pages still hold
|
|
# events, and rows are ASCENDING, so the events a fence looks for are the furthest from page 1.
|
|
#
|
|
# Both of these fixtures are the same construction with a different event on the far page, because
|
|
# the defeat is not axis-specific: one walk publishes both counts. Covering only the head axis would
|
|
# have repeated `process.fix-one-path-then-check-its-twin`.
|
|
|
|
|
|
@pytest.mark.parametrize("terminator", ["null", "[]"])
|
|
def test_a_FILTERED_intermediate_page_does_not_END_the_walk_head_axis(tmp_path, terminator):
|
|
"""A page that is empty because it was FILTERED must not certify the count. Head axis.
|
|
|
|
Page 1 is full filler, page 2 comes back empty, page 3 carries the pushes. A walk that stops at
|
|
page 2 certifies a zero push count on both reads, so the two agree, the sha comparison agrees,
|
|
and the ABA is exempted — which is precisely the attack: 50 inline comments buy silence.
|
|
|
|
Parameterised over both empty shapes because the walk has to treat them identically here; the
|
|
`null`-only fixture would leave the `[]` path free to keep the old early return.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="moves:0,2",
|
|
timeline_pages=3,
|
|
timeline_filtered_pages="2",
|
|
timeline_terminator=terminator,
|
|
)
|
|
assert posted is None, (
|
|
f"an exemption was granted over an ABA hidden behind a filtered page 2 (`{terminator}`) — "
|
|
f"the walk is reading a filtered page as the end of the timeline. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_FILTERED_intermediate_page_does_not_END_the_walk_base_axis(tmp_path):
|
|
"""The twin: the same construction hides a RETARGET rather than a push.
|
|
|
|
Same page layout, retargets on the far page. The base axis is the older of the two fences and
|
|
reads the same `rt_ok`/`rt_count` pair, so a fix that only reached the head arm would leave the
|
|
original #706 fence defeated by a construction its own test suite never built.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="moves:0,1",
|
|
timeline_pages=3,
|
|
timeline_filtered_pages="2",
|
|
)
|
|
assert posted is None, (
|
|
f"an exemption was granted over a retarget hidden behind a filtered page 2. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "retargeted while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the base fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_a_FILTERED_intermediate_page_on_a_QUIET_pr_still_exempts(tmp_path):
|
|
"""Without this, the two tests above pass against a walk that refuses every filtered page.
|
|
|
|
Refusing outright would be safe and useless: every PR carrying inline review comments would lose
|
|
its exemption. The walk must SKIP the empty page and keep counting, which is only visible when a
|
|
quiet PR with the same page layout still gets its `success`.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="stable:2",
|
|
timeline_pages=3,
|
|
timeline_filtered_pages="2",
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"a quiet PR lost its exemption because a page of its timeline was filtered. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_timeline_that_FILLS_the_page_cap_is_untrusted(tmp_path):
|
|
"""The cap is not exhaustion, and the LAST page read has to be the empty one.
|
|
|
|
Now that an empty page no longer ends the walk, the trust decision moved to after the loop — and
|
|
the whole of it rests on that decision still being conditional. Making it unconditional (a bare
|
|
`rt_ok=yes` once the loop finishes) would certify a partial count over an over-long timeline,
|
|
which is the failure the sibling status walk already refuses by the same rule. 25 real pages
|
|
means the walk never sees an empty one, so it must report "could not establish" and withhold.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="stable:0",
|
|
timeline_pages=25,
|
|
)
|
|
# NOT `posted is None`. An untrusted count on the exemption path does not merely stay silent —
|
|
# since ersatztv#849 it replaces the unknown state with the sticky unverified-write sentinel, so
|
|
# something IS posted and the assertion has to discriminate on the STATE. Asserting silence here
|
|
# passed a `pending` sentinel off as a failure and would have gone red on correct behaviour.
|
|
assert posted is None or posted["state"] != "success", (
|
|
"an exemption was granted over a timeline that never terminated inside the page cap — the "
|
|
f"post-loop trust decision is unconditional. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "Could not establish a trusted retarget/push count" in r.stdout, (
|
|
"no exemption was granted, but NOT because the count was untrusted — this test is not "
|
|
f"measuring the page cap. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_FILTERED_page_before_a_FULL_cap_page_is_still_untrusted(tmp_path):
|
|
"""A filtered page must not survive as trust when the walk then runs into the cap.
|
|
|
|
Page 19 comes back empty (filtered), page 20 is full. The walk must finish `empty=no` and refuse.
|
|
This is what makes the per-iteration `empty=no` reset load-bearing in the direction the other
|
|
fixtures cannot see: they all END on an empty page, so a stale flag and a correct one agree
|
|
there. Here they disagree — without the reset the flag is still `yes` from page 19 when the loop
|
|
falls out, and line `if [ "$empty" = yes ]` would certify a walk that never reached a terminator.
|
|
|
|
(The reset is ALSO covered from the other side: deleting it makes the three filtered-page tests
|
|
above go red, because a stale `yes` suppresses the tally on every later non-empty page and the
|
|
hidden events stop being counted. Measured, not assumed — the intuition that those three would
|
|
stay green is wrong.)
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="stable:0",
|
|
timeline_pages=25,
|
|
timeline_filtered_pages="19",
|
|
)
|
|
assert posted is None or posted["state"] != "success", (
|
|
"an exemption was granted over a walk that hit the page cap on a NON-empty page after a "
|
|
f"filtered one — the empty flag is stale. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "Could not establish a trusted retarget/push count" in r.stdout, (
|
|
f"nothing was exempted, but NOT because the count was untrusted. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("blip", ["transport", "body"])
|
|
def test_a_SINGLE_transient_blip_does_not_cost_the_exemption(tmp_path, blip):
|
|
"""The walk retries each page once, as `page_statuses` does and for its stated reason.
|
|
|
|
This walk had no retry while it made ~2 requests per call. At a fixed 20 it makes 10x as many,
|
|
so a blip is 10x likelier — and `rt_ok=no` on the exemption path writes the STICKY sentinel,
|
|
which costs that head its exemption until a human clears it by hand. That is a regression this
|
|
change would otherwise have shipped.
|
|
|
|
`transport-error` and `unreadable` cannot measure this: they fail every attempt, so a retrying
|
|
walk and a non-retrying one refuse identically. Only a blip that clears on the second attempt
|
|
separates them.
|
|
|
|
BOTH SHAPES, because they reach different arms and only one was pinned. `transport` leaves `raw`
|
|
empty. `body` returns a well-formed JSON OBJECT — what this Gitea actually sends on an error
|
|
(measured: `?since=NOTATIME` returns an object) — so `jq -r type` succeeds and `kind=object`.
|
|
|
|
That is what kills the narrowing `if [ -n "$kind" ]; then break; fi`, under which an error object
|
|
ends the retry and is then refused by the `case`, costing the exemption. An UNREADABLE body does
|
|
not kill it: jq fails, `kind=""`, and the walk takes `transport`'s path. With a 502 HTML page
|
|
as the fixture, for exactly that reason, the mutant passes — measured, not assumed.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="none",
|
|
push_mode="stable:0",
|
|
timeline_flaky_first=blip,
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"a docs-only PR lost its exemption to ONE transient timeline blip ({blip}) — the walk does "
|
|
f"not retry this shape. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_timeline_read_is_TIME_BOUNDED(tmp_path):
|
|
"""`gh` is bare `curl -sf` with no timeout, and this walk now issues 20 requests instead of ~2.
|
|
|
|
The third call site runs AFTER the exemption `success` is posted, so a hang there leaves the
|
|
green standing with no repair attempted. The bound is applied at THIS call rather than inside
|
|
`gh`, so the other call sites keep the behaviour they were reviewed with. Asserted structurally
|
|
because a hang cannot be provoked through the stub — the stub answers instantly by construction.
|
|
"""
|
|
body = _classify_step()["run"]
|
|
line = [ln for ln in body.splitlines() if "/timeline?limit=50&page=" in ln]
|
|
assert len(line) == 1, f"expected exactly one timeline read, found {len(line)}: {line}"
|
|
assert "--max-time" in line[0], (
|
|
"the timeline read carries no --max-time; a hung request on the POST-WRITE walk leaves an "
|
|
f"exemption success standing with no repair attempted. Line: {line[0]}"
|
|
)
|
|
assert "--connect-timeout" in line[0], f"the timeline read carries no --connect-timeout. Line: {line[0]}"
|
|
|
|
|
|
@pytest.mark.parametrize("terminator", ["null", "[]"])
|
|
def test_an_EMPTY_FIRST_page_is_untrusted_WHICHEVER_empty_shape_it_is(tmp_path, terminator):
|
|
"""Both empty shapes must fall through to `rt_ok=no`, and only one of them did.
|
|
|
|
The `null` arm required `page > 1`; the empty-ARRAY arm trusted any page, so a `[]` first page
|
|
certified "nothing was retargeted and nothing was pushed" over a response the walk could not
|
|
explain, and a docs-only PR was exempted on it. The comment three lines above already claimed the
|
|
safe behaviour, so the code contradicted its own stated rule for one of the two shapes — and this
|
|
server produces both, on different endpoints. Parameterised rather than corrected in place,
|
|
because the point is that the two shapes now take the SAME rule.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="empty-first-page",
|
|
timeline_terminator=terminator,
|
|
)
|
|
_assert_withheld(tmp_path, r, "test_an_EMPTY_FIRST_page_is_untrusted_WHICHEVER_empty_shape_it_is", expect_rc=0)
|
|
|
|
|
|
def test_a_timeline_row_with_NO_READABLE_TYPE_is_not_counted_as_nothing(tmp_path):
|
|
"""A shape we cannot read is not evidence that nothing happened.
|
|
|
|
Both tallies select on `.type`. A row lacking one, or carrying a non-string one, matches neither
|
|
and was silently treated as some other event — so a page of unclassifiable rows certified a zero
|
|
count. This is the same validate-what-you-consume rule `pr-changed-files.sh` applies to every row
|
|
IT extracts, which the walk gating the write did not have.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="untyped-rows")
|
|
_assert_withheld(tmp_path, r, "test_a_timeline_row_with_NO_READABLE_TYPE_is_not_counted_as_nothing", expect_rc=0)
|
|
|
|
|
|
def test_the_head_fence_reports_the_OBSERVED_counts_in_its_notice(tmp_path):
|
|
"""The notice must name the numbers it fenced on.
|
|
|
|
A fence whose diagnostic says only "something moved" cannot be triaged from a run log, and this
|
|
workflow's history is full of runs that posted nothing for reasons nobody could reconstruct
|
|
afterwards (#751 went unnoticed for exactly that reason). Pin the before/after pair.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="none", push_mode="moves:3,5")
|
|
assert posted is None, f"the head fence did not fire. Log:\n{r.stdout[-900:]}"
|
|
assert "(3 -> 5 push events)" in r.stdout, (
|
|
f"the head fence notice does not report the counts it fenced on. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_PROTECTED_pr_pushed_mid_run_ALSO_posts_nothing(tmp_path):
|
|
"""The fence covers `pending` as well as the exemption `success`.
|
|
|
|
Same rule as the base axis, and the same reason it is one rule rather than a
|
|
reason-about-it-per-state condition: the successor run is guaranteed either way, so there is
|
|
nothing to buy by writing a value this run already knows was computed across two heads.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), timeline_mode="none", push_mode="moves:0,2")
|
|
assert r.returncode == 0, f"the job died rather than abstaining cleanly: {r.stderr[-600:]}"
|
|
assert posted is None, (
|
|
"a run that saw its head move still wrote a status for a non-exempt PR — the fence is "
|
|
f"scoped to `success` only. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
# Assert the DISCRIMINATOR. "Posted nothing" is also produced by a crash, an unreadable timeline
|
|
# and several unrelated abstentions, so without this the test proves only that something went
|
|
# wrong somewhere.
|
|
assert "head branch was pushed while this job was classifying" in r.stdout, (
|
|
f"nothing was posted, but NOT via the head fence. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_PR_retargeted_BEFORE_the_run_but_QUIET_during_it_is_STILL_exempt(tmp_path):
|
|
"""The fence keys on MOTION during this run, never on "has ever been retargeted".
|
|
|
|
A PR legitimately retargeted once, long before this run, carries a permanently non-zero event
|
|
count. Keying on the count being non-zero — rather than on it CHANGING — would deadlock that PR's
|
|
exemption forever, which is a worse failure than the race being fixed.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="stable:2")
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"a PR with a settled, non-zero retarget history was refused its exemption — the fence is "
|
|
f"testing the count's VALUE instead of its MOVEMENT. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("mode", ["unreadable", "transport-error"])
|
|
def test_an_UNTRUSTED_retarget_count_withholds_the_EXEMPTION(tmp_path, mode):
|
|
"""A `success` that cannot be shown to describe the PR's current base must not be written. An
|
|
absent required check blocks the merge, which is the safe direction."""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode=mode)
|
|
_assert_withheld(tmp_path, r, "test_an_UNTRUSTED_retarget_count_withholds_the_EXEMPTION", expect_rc=0)
|
|
|
|
|
|
@pytest.mark.parametrize("mode", ["unreadable", "transport-error"])
|
|
def test_an_UNTRUSTED_retarget_count_STILL_LETS_PENDING_THROUGH(tmp_path, mode):
|
|
"""The other half, and the one that keeps a timeline outage from taking every PR down with it.
|
|
|
|
`pending` blocks the merge, so letting it through is the right anti-stranding trade — withholding
|
|
it would leave ordinary PRs with no status at all. NOT because it is inert: a generic `pending`
|
|
masks a rejection landing in its own write window, gets no post-write verification, and a later
|
|
run re-derives it green (ersatztv#849). Immediate block versus later re-derivation risk, not
|
|
"buys no safety". Only the exemption is gated.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), timeline_mode=mode)
|
|
assert posted is not None and posted["state"] == "pending", (
|
|
"an ordinary PR was left with NO status because the timeline was unreadable; only the "
|
|
f"exemption success should be gated on the count. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_human_verdict_landing_AFTER_the_POST_is_repaired_to_pending(tmp_path):
|
|
"""Race 2's remainder. The pre-POST re-read cannot see a verdict that lands between it and the
|
|
write, and there is no compare-and-set to make the two one operation. So the write is verified
|
|
AFTERWARDS and repaired in the safe direction: silently greening an explicit human BLOCKED is the
|
|
worst outcome this gate can produce, and strictly worse than a stall.
|
|
|
|
The repair is `pending`, never a copy of the human's `failure` — re-posting their state under the
|
|
machine credential would attribute a human verdict to this job.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="human-after-post")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"expected the exemption write followed by a repair; the post-write verification did not "
|
|
f"fire. Posts: {seq}. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success"
|
|
assert seq[1]["state"] == "pending", f"the raced exemption was not repaired: {seq}"
|
|
assert posted is not None and posted["state"] == "pending"
|
|
# WHICH SENTINEL, not merely `pending` (ersatztv#849). A verdict was COUNTED above the mark, so
|
|
# this arm may assert the strong fact — a human verdict existed and was buried — which only a
|
|
# human can clear. The uncertainty arms write the reconcilable sentinel instead, and a test that
|
|
# checked only the state could not tell the two apart.
|
|
assert seq[1]["description"] == REPAIR_DESC, (
|
|
f"a counted human verdict was recorded with the wrong sentinel: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_PRE_EXISTING_human_row_does_NOT_trigger_a_repair(tmp_path):
|
|
"""The false-positive guard, and the reason the check compares ids instead of asking "does a human
|
|
verdict exist".
|
|
|
|
`read_existing_verdict` deliberately declines to honour a human verdict whose recorded base does
|
|
not match this PR's — so such a row sits in the history forever. A presence test would fire on it
|
|
on EVERY later run, repair each exemption to `pending`, and permanently deadlock a PR that had one
|
|
mismatched verdict once. Only rows newer than the high-water mark count.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="stale-human-already-present")
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"an old, already-present human verdict row was mistaken for one that raced this run's write, "
|
|
f"so the exemption was repaired away. Posts: {seq}. Log:\n{r.stdout[-900:]}"
|
|
)
|
|
assert posted is not None and posted["state"] == "success"
|
|
|
|
|
|
# --- ersatztv#711: the PROTECTED list must cover every mirror of the enforcement hooks -----------
|
|
|
|
|
|
def test_a_CODEX_hook_copy_is_a_PROTECTED_path(tmp_path):
|
|
"""`.codex/hooks/` is a byte-identical mirror of `.claude/hooks/`, including
|
|
`pretooluse-merge-consent.sh`. Editing the `.claude/` copy correctly voids both exemptions;
|
|
before this, editing its `.codex/` twin did not — the rule "a PR that can weaken the gate must
|
|
not exempt itself from the gate" was written as a path list and the list had gone incomplete.
|
|
|
|
Latent rather than live today (`.codex/` is untracked and gitignored, and a PR cannot touch a
|
|
path that is not in the repo), which is exactly why it needs a test: it becomes live silently,
|
|
the moment anyone tracks the directory.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md", ".codex/hooks/pretooluse-merge-consent.sh"))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending", (
|
|
"a PR editing a .codex/ copy of the merge-consent hook was granted the docs-only exemption "
|
|
f"(desc={posted['description']!r})"
|
|
)
|
|
# Assert the DISCRIMINATOR: a non-exempt outcome is reachable by several routes, so the state
|
|
# alone cannot show the PROTECTED branch is what rejected it.
|
|
assert "protected" in r.stdout.lower(), (
|
|
f"rejected, but NOT by the protected-path branch. Decision log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_BOTH_hook_directories_are_in_the_PROTECTED_pattern():
|
|
"""A structural pin over the list itself. The behavioural test above proves `.codex/` is covered
|
|
today; this one states the INVARIANT — the two hook directories are mirrors, so any future edit
|
|
that drops one while keeping the other is the bug, not a simplification."""
|
|
src = _code_lines(WORKFLOW)
|
|
protected = [line for line in src.splitlines() if line.strip().startswith("PROTECTED=")]
|
|
assert len(protected) == 1, f"expected exactly one PROTECTED definition, found {protected}"
|
|
for d in (r"\.claude/", r"\.codex/"):
|
|
assert d in protected[0], (
|
|
f"{d} is missing from PROTECTED ({protected[0].strip()!r}) — a directory carrying a copy "
|
|
"of the enforcement hooks can exempt itself from the gate it enforces"
|
|
)
|
|
|
|
|
|
# --- ersatztv#706: further findings on the fence and the repair ----------------------------------
|
|
|
|
|
|
def test_the_high_water_MARK_is_captured_BEFORE_the_last_moment_re_read():
|
|
"""The ORDERING property the high-water mark rests on, pinned as an order rather than an output.
|
|
|
|
Taking the mark "as late as possible", just before the POST, leaves everything between the
|
|
last-moment re-read and the mark as a blind gap: a human verdict landing there is invisible to
|
|
the re-read (already done) and excluded from the post-write check (its id is BELOW a mark taken
|
|
afterwards), so it is overwritten with no repair. The gap spans the whole retarget re-count — up
|
|
to 20 timeline round-trips, not the single round-trip it looks like.
|
|
|
|
Taking the mark FIRST closes the read side: any row newer than the mark is caught either by the
|
|
re-read (abstain) or by the post-write check (repair). This is asserted structurally because the
|
|
defect is an order, not an output — with the mark late the job still posts and still repairs in
|
|
every scenario a stub can pose; only the id arithmetic silently changes.
|
|
"""
|
|
src = _classify_step()["run"]
|
|
# Assert on BOTH the init and the FETCH: keying only on the init line would let a refactor that
|
|
# splits initialisation from the read slide the actual round-trip back past the re-read while this
|
|
# test stayed green. The FETCH is now the first bare `page_statuses` CALL (ersatztv#763 replaced
|
|
# the single
|
|
# `hist_before=$(gh …)` read with the paged walk). Anchoring on the call and not on the function
|
|
# definition matters: the definition sits with the other helpers near the top of the step, so
|
|
# keying on it would place the "fetch" far earlier than the round-trip actually happens and this
|
|
# assertion would hold vacuously.
|
|
assert "\npage_statuses\n" in src, (
|
|
"no bare `page_statuses` call site found; the mark is no longer fetched where this test "
|
|
"believes it is, and the ordering assertion below would be vacuous"
|
|
)
|
|
mark = max(src.index("max_id_before=-1"), src.index("\npage_statuses\n"))
|
|
# The LAST-MOMENT re-read is the second bare `read_existing_verdict` call.
|
|
calls = [i for i in range(len(src)) if src.startswith("read_existing_verdict\n", i)]
|
|
assert len(calls) >= 2, f"expected two read_existing_verdict call sites, found {len(calls)}"
|
|
assert mark < calls[-1], (
|
|
"the high-water mark is captured AFTER the last-moment re-read, reopening the blind window "
|
|
"in which a human verdict is neither seen by the re-read nor repaired by the post-write check"
|
|
)
|
|
|
|
|
|
def test_a_previously_REPAIRED_head_is_never_re_exempted(tmp_path):
|
|
"""Without a sentinel, the repair lasts exactly one event.
|
|
|
|
After a repair, the status is a machine-written `pending` — indistinguishable, to the next run,
|
|
from an ordinary one. That run re-derived it, posted `success`, and took a fresh high-water mark
|
|
ABOVE the human row, so the post-write check stayed silent and the human's rejection went green
|
|
again one event later. The repair description is now a sentinel the classification recognises.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc="Human verdict raced this exemption write — re-post the verdict",
|
|
)
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-2000:]}"
|
|
assert posted["state"] == "pending", (
|
|
"a head whose human verdict was previously overwritten was granted a FRESH exemption, burying "
|
|
f"the rejection again (desc={posted['description']!r})"
|
|
)
|
|
assert "raced a previous exemption" in r.stdout, (
|
|
f"rejected, but not via the repair-sentinel branch. Log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_an_ORDINARY_machine_pending_is_still_re_derived(tmp_path):
|
|
"""Without this, the test above would pass against a job that had stopped exempting anything with
|
|
a pre-existing `pending`. An ordinary machine `pending` — no sentinel — must still re-derive to
|
|
`success` for a docs-only PR."""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc="Awaiting review verdict for a9e3e23",
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"an ordinary pending was not re-derived to an exemption. Log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_the_fence_gates_PENDING_TOO_not_only_the_exemption(tmp_path):
|
|
"""The fence deliberately refuses to write `pending` as well as `success` when the PR was
|
|
retargeted mid-run. Untested, a success-only-fence mutant would have passed the whole suite.
|
|
|
|
A stale `pending` is only a stall rather than a forged green, so gating it is not strictly
|
|
required — but the successor run is guaranteed either way, so there is nothing to buy by writing a
|
|
value computed against a base the PR may no longer target.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("ErsatzTV/Program.cs"), timeline_mode="moves:0,1")
|
|
assert posted is None, (
|
|
"the fence let a stale `pending` through; it is gating only the exemption path "
|
|
f"(desc={posted['description']!r})"
|
|
if posted
|
|
else ""
|
|
)
|
|
assert "retargeted while this job was classifying" in r.stdout
|
|
|
|
|
|
def test_an_UNTRUSTED_count_withholds_the_exemption_BY_THAT_BRANCH(tmp_path):
|
|
"""Asserting only "posted nothing" is satisfied by a crash too. Assert the discriminator and a
|
|
clean exit."""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="unreadable")
|
|
_assert_withheld(tmp_path, r, "test_an_UNTRUSTED_count_withholds_the_exemption_BY_THAT_BRANCH", expect_rc=0)
|
|
assert r.returncode == 0, f"the job died rather than declining cleanly: {r.stderr[-800:]}"
|
|
# The wording covers BOTH axes since ersatztv#803 — one walk certifies one trust flag, so an
|
|
# unreadable page abandons the push count and the retarget count together.
|
|
assert "Could not establish a trusted retarget/push count" in r.stdout, (
|
|
f"nothing posted, but not via the untrusted-count branch. Log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_the_repair_sentinel_is_a_FIXED_POINT_across_consecutive_runs(tmp_path):
|
|
"""Durability is a fixed point, and only a CHAIN can assert a fixed point.
|
|
|
|
A sentinel that refuses the exemption but posts the GENERIC pending description erases the
|
|
marker it depends on. The next run then sees an ordinary machine `pending`, re-derives it, and
|
|
posts `success` — burying the human rejection two events after the repair instead of one. A
|
|
single-hop test passes throughout, and the positive control asserting that an ordinary machine
|
|
`pending` DOES re-derive is itself the proof of the second hop.
|
|
|
|
So chain two runs: feed run N's posted description in as run N+1's existing status. The property
|
|
is that the sentinel branch's own output re-triggers the sentinel branch, forever.
|
|
"""
|
|
first, r1 = _run_classify(
|
|
tmp_path / "run1",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc="Human verdict raced this exemption write — re-post the verdict",
|
|
)
|
|
assert first is not None and first["state"] == "pending", f"run 1: {r1.stdout[-600:]}"
|
|
|
|
second, r2 = _run_classify(
|
|
tmp_path / "run2",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=first["description"],
|
|
) # <-- the chain
|
|
assert second is not None, f"run 2 posted nothing: {r2.stderr[-800:]}"
|
|
assert second["state"] == "pending", (
|
|
"the repair decayed: run 1's own posted description did not re-trigger the sentinel, so run 2 "
|
|
f"re-derived the exemption and buried the human verdict again (run1 desc={first['description']!r}, "
|
|
f"run2 state={second['state']}, run2 desc={second['description']!r})"
|
|
)
|
|
assert second["description"] == first["description"], (
|
|
"the sentinel is not a fixed point — run 2 wrote a different description than run 1, so run 3 "
|
|
f"would not recognise it ({first['description']!r} -> {second['description']!r})"
|
|
)
|
|
|
|
|
|
def test_a_sentinel_APPEARING_MID_RUN_stops_a_stale_run_overwriting_it(tmp_path):
|
|
"""The only case in this series that failed toward SUCCESS.
|
|
|
|
Two runs overlap for the same sha — the regime this branch measured live (probe PR #722: the older
|
|
run finished 20s after the newer one started). Run B catches a raced human BLOCKED and repairs to
|
|
the sentinel. Run A is still in flight: its FIRST read predates all of it, so it classified
|
|
`success`; its high-water mark was taken after the human row, so the post-write check stays silent;
|
|
and the retarget fence sees nothing. Until this guard, A posted its stale `success` straight over
|
|
the sentinel — burying the human rejection with no repair and no log.
|
|
|
|
The re-read recomputes `ex_repair`; the bug was that nothing downstream consulted it.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="sentinel-appears-on-read:2")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"a stale run posted its exemption over a repair sentinel written mid-run, burying the human "
|
|
f"verdict it records (posted={posted})"
|
|
)
|
|
assert "repair sentinel was written" in r.stdout, (
|
|
f"nothing posted, but not via the mid-run sentinel guard. Log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_a_sentinel_present_from_the_START_still_posts_pending(tmp_path):
|
|
"""Without this, the guard above could be an unconditional abstain whenever a sentinel exists.
|
|
|
|
A sentinel present at the FIRST read is the ordinary repaired-head case: it must still POST
|
|
`pending` carrying the sentinel forward (the fixed point), not abstain. That difference is exactly
|
|
what makes the mid-run guard's condition exact rather than conservative.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc="Human verdict raced this exemption write — re-post the verdict",
|
|
)
|
|
assert posted is not None, f"the repaired head stopped posting entirely: {r.stdout[-800:]}"
|
|
assert posted["state"] == "pending"
|
|
assert posted["description"] == "Human verdict raced this exemption write — re-post the verdict"
|
|
|
|
|
|
def test_a_PENDING_path_run_also_refuses_to_clobber_a_mid_run_sentinel(tmp_path):
|
|
"""A guard testing `state = success` is one branch too narrow.
|
|
|
|
A run can reach the POST on `pending` carrying the GENERIC description — most realistically after a
|
|
transient enumeration failure — and a success-only guard waves it through. It then overwrites the
|
|
sentinel with ordinary text, the next run sees no sentinel, re-derives, and posts `success`: the
|
|
same buried human rejection, two steps instead of one.
|
|
|
|
Hence the rule is "never replace a sentinel with a non-sentinel", compared on the DESCRIPTION. That
|
|
is strictly more general and exactly as precise, because the carry-forward branch guarantees a
|
|
first-read sentinel already sets `desc` to the sentinel.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n", # enumeration fails -> pending, generic desc
|
|
status_mode="sentinel-appears-on-read:2",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is None, (
|
|
"a pending-path run overwrote a repair sentinel written mid-run with the generic description; "
|
|
f"the next run would re-derive the exemption and bury the human verdict (posted={posted})"
|
|
)
|
|
assert "repair sentinel was written" in r.stdout, (
|
|
f"nothing posted, but not via the mid-run sentinel guard. Log:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_SENTINEL_landing_above_the_mark_also_triggers_the_repair(tmp_path):
|
|
"""Counting only HUMAN rows in the post-write filter is not sufficient.
|
|
|
|
With two overlapping runs A and B, the human BLOCKED can land BELOW A's high-water mark — so A
|
|
cannot see it — while B masks it with an exemption `success` and only afterwards writes the
|
|
sentinel. A then finds nothing human above its mark, does not repair, and posts its own `success`
|
|
on top of the sentinel: a PERMANENT forged green over a human rejection, and the repair race
|
|
failing toward success rather than pending, which the decision record explicitly promises it does
|
|
not do.
|
|
|
|
A sentinel above the mark can only have been written by another run mid-flight — a pre-existing one
|
|
would have been seen at the first read and forced the pending path — so counting it cannot
|
|
false-fire, and it converges both runs on the fixed point.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="sentinel-after-post")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"the run posted its exemption on top of a sentinel written by an overlapping run and did not "
|
|
f"repair, leaving a human rejection permanently green. Posts: {seq}\n{r.stdout[-800:]}"
|
|
)
|
|
assert seq[0]["state"] == "success"
|
|
assert seq[1]["state"] == "pending"
|
|
assert seq[1]["description"] == "Human verdict raced this exemption write — re-post the verdict", (
|
|
f"repaired, but not to the sentinel, so the fixed point is broken: {seq[1]}"
|
|
)
|
|
|
|
|
|
# --- The expression-delimiter class (ersatztv#751) ----------------------------------------------
|
|
#
|
|
# WHAT HAPPENED, because the shape of it is what these tests are written against. A `run:` body is
|
|
# not shell yet when the runner reads it. The runner scans the whole scalar for the expression
|
|
# opener, and a single occurrence makes it rewrite the ENTIRE body into one `format(...)` call so
|
|
# the evaluated result can be spliced back in. That rewrite is all-or-nothing, so a payload that
|
|
# does not parse takes the whole step with it — and the runner then DROPS THE STEP AND CONCLUDES THE
|
|
# JOB `success`.
|
|
#
|
|
# The occurrence that did it was in a SHELL COMMENT: the #706 note explaining why a concurrency
|
|
# group does not work quoted a `concurrency:` snippet containing a PR-number expression as an
|
|
# illustration. `pr number` is not an expression. The step stopped running on 2026-08-03 and nothing
|
|
# went red until 2026-08-06.
|
|
#
|
|
# WHY EVERY EXISTING GUARD IN THIS FILE WAS BLIND TO IT, which is the part worth keeping: they all
|
|
# read `_code_lines()`, which strips comment lines. That choice is correct for what it was for — its
|
|
# own docstring explains that prose legitimately discusses `pulls/N/files`, and a raw scan would
|
|
# redden the repo over a piece of writing. But it encodes an assumption this bug falsifies: that a
|
|
# comment in a workflow cannot change behaviour. Inside a `run:` scalar it can. So the test below
|
|
# reads the RAW text, deliberately, and is the one test here that must never adopt `_code_lines`.
|
|
|
|
|
|
_EXPR = re.compile(r"\$\{\{(.*?)\}\}", re.S)
|
|
|
|
# Roots of a dotted context path, and the callable functions. Both lists are what the runner
|
|
# actually accepts; anything outside them cannot evaluate, and an expression that cannot evaluate
|
|
# does not fail loudly — it silently removes the step it appears in.
|
|
_EXPR_CONTEXTS = frozenset(
|
|
{
|
|
"github",
|
|
"env",
|
|
"vars",
|
|
"secrets",
|
|
"inputs",
|
|
"runner",
|
|
"steps",
|
|
"needs",
|
|
"matrix",
|
|
"job",
|
|
"jobs",
|
|
"strategy",
|
|
}
|
|
)
|
|
_EXPR_FUNCTIONS = frozenset(
|
|
{
|
|
"always",
|
|
"success",
|
|
"failure",
|
|
"cancelled",
|
|
"hashFiles",
|
|
"format",
|
|
"toJSON",
|
|
"toJson",
|
|
"fromJSON",
|
|
"fromJson",
|
|
"contains",
|
|
"startsWith",
|
|
"endsWith",
|
|
"join",
|
|
}
|
|
)
|
|
_EXPR_LITERALS = frozenset({"true", "false", "null"})
|
|
|
|
|
|
def _yaml_string_scalars(path: Path):
|
|
"""Every string scalar in the parsed document — keys and values, recursively.
|
|
|
|
Deliberately PARSED rather than raw (ersatztv#751). A `${{ … }}` in an ordinary
|
|
top-level YAML comment is inert: the runner never evaluates it, so redding on it would be a false
|
|
positive of exactly the kind this file has now produced twice. PyYAML drops those comments, which
|
|
is the behaviour wanted here.
|
|
|
|
A `run:` body IS one of these scalars, and it keeps its SHELL comments — which is the whole point,
|
|
because inside a `run:` scalar a comment is not inert. So this covers the real defect class in
|
|
every workflow while ignoring the one place a delimiter is genuinely harmless.
|
|
"""
|
|
import yaml
|
|
|
|
out: list[str] = []
|
|
|
|
def walk(node):
|
|
if isinstance(node, str):
|
|
out.append(node)
|
|
elif isinstance(node, dict):
|
|
for k, v in node.items():
|
|
if isinstance(k, str):
|
|
out.append(k)
|
|
walk(v)
|
|
elif isinstance(node, list):
|
|
for v in node:
|
|
walk(v)
|
|
|
|
walk(yaml.safe_load(path.read_text()))
|
|
return out
|
|
|
|
|
|
def _workflow_files():
|
|
# `*.yml` AND `*.yaml`: a workflow added as `.yaml` is just as executable and would otherwise go
|
|
# unscanned. From the GIT INDEX rather than a directory listing (ersatztv#806) — a stray
|
|
# untracked `.yaml` in `.gitea/workflows/` is not part of the repo, and enumerating it made
|
|
# these two completeness claims red on that checkout and green in CI, which is #778's shape.
|
|
return tracked_paths(".gitea/workflows", ("*.yml", "*.yaml"))
|
|
|
|
|
|
def test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body():
|
|
"""The absolute rule, and it is scoped to this one file on purpose.
|
|
|
|
Other workflows legitimately interpolate into a `run:` body (5 occurrences today across
|
|
ci-image, docker-build's `api-docs`/`format`, and pr-checks' two git-diff gates), so a
|
|
repo-wide ban would be false and would be deleted the first time it got in someone's way.
|
|
`test`, `migrations` and `build` carry their own absolute ban — see
|
|
scripts/tests/test_ci_dropped_step_guard.py (ersatztv#756). This file is different in two ways that justify the
|
|
strict rule: it writes the branch-protection-required status, so a dropped step here is a dead
|
|
merge gate rather than a failed build; and its `run:` bodies are ~700 lines of dense prose,
|
|
which is the only place the delimiter has ever appeared by accident.
|
|
|
|
The rule is also what keeps the dropped-step guard trustworthy. A guard that the guarded
|
|
mechanism can silently delete is worse than no guard, because its absence is silent too.
|
|
|
|
The `run:` SCALAR AS PARSED, comments and all — never `_code_lines`, which strips them. The
|
|
distinction matters and is easy to garble: what must not be filtered is the SHELL comments inside
|
|
the body, because those are what the runner scans. Ordinary YAML comments outside a `run:` body are
|
|
genuinely inert and are handled by the repo-wide test below, which is why neither test reads the
|
|
file as flat text any more.
|
|
"""
|
|
offenders = []
|
|
for job_name, step in _iter_workflow_steps(WORKFLOW):
|
|
for m in _EXPR.finditer(step.get("run") or ""):
|
|
offenders.append(f"{job_name}/{step.get('name', '?')}: {m.group(1).strip()!r}")
|
|
assert not offenders, (
|
|
f"review-verdict.yml has an expression delimiter inside a run: body — {offenders}. Even in "
|
|
"a comment this is unsafe: the runner rewrites the WHOLE body into a format(...) call, and "
|
|
"if the payload does not parse it drops the step and reports the job GREEN, leaving the "
|
|
"required review-verdict/h10 unposted (ersatztv#751). To describe an expression in prose, "
|
|
"name it (`a github.event.pull_request.number expression`) instead of quoting the "
|
|
"delimiters. Pass values in through the step's `env:` block, which is interpolated per "
|
|
"value, so a bad payload there cannot take the body with it."
|
|
)
|
|
# ANTI-VACUITY, and it has to be the right property. A `run:` body the YAML walk never reached
|
|
# would make the assertion above vacuously green — the failure mode to guard against. Comparing
|
|
# the file's TOTAL delimiter count against the count inside `with:`/`env:` values is a different
|
|
# and wrong claim: it bans expressions everywhere else in the file too. The false red is
|
|
# reproducible — writing `if: ${{ always() }}`, the standard and equivalent spelling of the
|
|
# `if:` two steps below, turns this test red, as does a delimiter in an inert top-level YAML
|
|
# comment. Neither is unsafe, and a red here blocks every merge through the combined status, so
|
|
# such a guard is strictly more dangerous than the thing it protects against.
|
|
#
|
|
# ANTI-VACUITY WITHOUT A SECOND PARSER. Counting `run:` keys in the raw text and comparing that
|
|
# to the walk needs a regex over YAML: one recognising only an indented `run:` whose value
|
|
# starts with `|` or `>` counts legal spellings (`- run: |`, a single-line `run: echo ok`) as
|
|
# zero declarations and false-reds the file, while a `run: |` line sitting INSIDE a shell
|
|
# heredoc counts as a declaration. Hand-parsing YAML to check a YAML parse
|
|
# is the wrong shape: it adds a second, worse parser whose disagreements are all false alarms, and
|
|
# a red here blocks every merge through the combined status.
|
|
#
|
|
# The property that actually matters is that the walk reached the body that carries the risk. The
|
|
# classifier is ~700 lines; a walk that returned nothing, or only the short steps, is the failure
|
|
# to catch. Both are asserted on content, which no spelling change can spoof.
|
|
bodies = [s["run"] for _, s in _iter_workflow_steps(WORKFLOW) if s.get("run")]
|
|
# NOT `>= 3`, and not merely non-empty either. `>= 3` had zero slack — deleting the optional
|
|
# jq-preflight step, a legitimate simplification, redded this claiming the classifier had not
|
|
# been examined, which was untrue. But relaxing it to `assert bodies` threw away the only
|
|
# check that the walk reached ALL run-bearing steps: `max(len) > 5000` proves it reached the
|
|
# classifier and nothing about the short ones, so a helper that silently stopped yielding them
|
|
# would let an unscanned delimiter through.
|
|
#
|
|
# So count against the job's own step list, read here rather than through the helper under test.
|
|
# That catches the failure the count existed for (a helper looking at the wrong key, or dropping
|
|
# steps) without breaking when a step is legitimately added or removed.
|
|
import yaml as _yaml
|
|
|
|
_steps = _yaml.safe_load(WORKFLOW.read_text())["jobs"]["set-verdict-status"]["steps"] or []
|
|
declared = sum(1 for s in _steps if isinstance(s, dict) and s.get("run"))
|
|
assert len(bodies) == declared, (
|
|
f"the YAML walk reached {len(bodies)} run: bodies but the job declares {declared} — the "
|
|
"assertion above did not examine every body, so a green here proves nothing"
|
|
)
|
|
assert max(len(b) for b in bodies) > 5000, (
|
|
"the YAML walk did not reach a substantial run: body — the ~700-line classifier is the one "
|
|
"that must be scanned, so this test would be vacuous"
|
|
)
|
|
|
|
|
|
def _iter_workflow_steps(path: Path):
|
|
import yaml
|
|
|
|
doc = yaml.safe_load(path.read_text()) or {}
|
|
for job_name, job in (doc.get("jobs") or {}).items():
|
|
for step in job.get("steps") or []:
|
|
yield job_name, step
|
|
|
|
|
|
def test_every_workflow_expression_names_a_REAL_context_or_function():
|
|
"""The general form of #751, across every workflow and every field.
|
|
|
|
The strict test above hardens the gate file. It cannot see the class, which is not "prose in a
|
|
run body" but "a payload that does not evaluate" — a typo in an `if:`, a renamed output, a
|
|
context that does not exist. All of them fail the same silent way, and in an `if:` the
|
|
consequence is the same shape as #751: the step does not run and nothing is red.
|
|
|
|
BE PRECISE ABOUT WHAT THIS ENFORCES: it checks that THE HEAD TOKEN of each dotted path is a
|
|
known context or function. Nothing more. That catches the historical defect — `pr number` fails
|
|
on `pr` — and a payload naming a context that does not exist. It does NOT catch:
|
|
|
|
* syntactically invalid expressions whose tokens are all known: `${{ github.ref == }}` and
|
|
`${{ github.event.pull_request.head.sha + }}` both pass, verified;
|
|
* a renamed or misspelled output or property, because every token after the first is preceded
|
|
by `.` and is deliberately skipped: `steps.metadata.outputs.shortsha` passes;
|
|
* an unclosed opener, since `_EXPR` requires the closing braces to match at all.
|
|
|
|
A real fix for those is an expression parser, which is a different and much larger change. This is
|
|
a cheap net under the specific class that has bitten us, deliberately kept permissive so it cannot
|
|
false-red the repo (a red here blocks every merge through the combined status). Verified against
|
|
all 31 distinct payloads in this repo today, which pass. Do not restate this test as "catches any
|
|
payload that cannot evaluate" — that claim was in the docs and the decision record and was false.
|
|
"""
|
|
offenders = []
|
|
for wf in _workflow_files():
|
|
for scalar in _yaml_string_scalars(wf):
|
|
for m in _EXPR.finditer(scalar):
|
|
payload = m.group(1).strip()
|
|
# Strip string literals first: a path inside `hashFiles('web/package-lock.json')` is
|
|
# data, not an identifier, and would otherwise read as an unknown context.
|
|
bare = re.sub(r"'[^']*'", "''", payload)
|
|
for ident in re.finditer(r"(?<![.\w'])([A-Za-z_][A-Za-z0-9_-]*)", bare):
|
|
name = ident.group(1)
|
|
if name in _EXPR_CONTEXTS or name in _EXPR_FUNCTIONS or name in _EXPR_LITERALS:
|
|
continue
|
|
offenders.append(f"{wf.name}: {payload!r} -> unknown '{name}'")
|
|
assert not offenders, (
|
|
"these workflow expressions name something the runner cannot resolve, so they will fail to "
|
|
f"interpolate — which DROPS THE STEP and still reports the job green (ersatztv#751): "
|
|
f"{offenders}. If this is prose describing an expression, do not write the delimiters; if "
|
|
"it is a real new context or function, add it to _EXPR_CONTEXTS / _EXPR_FUNCTIONS here."
|
|
)
|
|
|
|
|
|
def test_a_dropped_classify_step_FAILS_the_job_instead_of_going_green():
|
|
"""The silent-green half of #751 — the actual defect, and the only part that generalises.
|
|
|
|
The stray delimiter was one bug in one comment. What made it cost three days of a dead merge
|
|
gate was the REPORTING: a dropped step concludes `success`, the workflow's own status context
|
|
goes green, and the required `review-verdict/h10` is merely ABSENT — which on a normal PR is
|
|
indistinguishable from the correct "not reviewed yet" state. Nothing surfaced it. So the
|
|
classifier writes a start marker and a later `if: always()` step fails the job when it is
|
|
missing.
|
|
|
|
Asserted structurally, and the marker PATH is compared across the two steps rather than
|
|
hardcoded here twice: the failure mode of a divergent path is a job that reddens on every run
|
|
(fail-closed, but it would look like this guard is broken rather than like a real drop), and a
|
|
test carrying its own third copy of the literal could not see the divergence at all.
|
|
|
|
The premise — that the runner still executes a LATER step after dropping an earlier one — is
|
|
MEASURED, not assumed, and could not be settled from the #751 report because the classifier was
|
|
the job's last step. Gitea 1.27.1, 2026-08-06: probe run 1863 dropped the classifier on a
|
|
reintroduced bad payload, ran the guard anyway (`always()` evaluated true), and the job concluded
|
|
`failure`; run 1866 is the positive control. So the guard is load-bearing in production and the
|
|
body does not need to move into `scripts/`. This test pins the SHAPE; the probe pinned the
|
|
BEHAVIOUR, and neither substitutes for the other — which is why the behavioural test below
|
|
EXECUTES the guard body rather than only reading it.
|
|
"""
|
|
# ONE parse, and both steps located inside it. Using `_classify_step()` here instead was wrong
|
|
# in a way worth recording, because it went green-adjacent rather than loud: that helper does its
|
|
# own `yaml.safe_load`, so the dict it returns is never the same OBJECT as the equivalent step in
|
|
# this list. An `is not` filter against it therefore excluded nothing, the classify step matched
|
|
# as its own guard, and the assertions below ran against the wrong step.
|
|
steps = [s for _, s in _iter_workflow_steps(WORKFLOW)]
|
|
classifiers = [i for i, s in enumerate(steps) if "review-verdict/h10" in (s.get("run") or "")]
|
|
assert classifiers, "no step in review-verdict.yml posts review-verdict/h10"
|
|
classify = steps[classifiers[0]]
|
|
# The FULL path expansion is compared between the two steps, not a basename. The path is keyed on
|
|
# the run id, so a basename match would accept two steps that agree on the prefix and disagree on
|
|
# the key — which is precisely the divergence that would make the guard fail on every run.
|
|
marker_write = re.search(r'(\w+)="(\$\{RUNNER_TEMP[^"]*)"', classify["run"])
|
|
assert marker_write, (
|
|
"the classify step no longer assigns a start-marker path under RUNNER_TEMP, so a step the "
|
|
"runner drops goes green again (ersatztv#751)"
|
|
)
|
|
var, marker_name = marker_write.group(1), marker_write.group(2)
|
|
assert re.search(rf':\s*>\s*"\${var}"', classify["run"]), (
|
|
f"the classify step defines {var} but never creates the marker, so the guard below will "
|
|
"fail on every run and read as broken rather than as a real dropped step"
|
|
)
|
|
|
|
guard_idx = [i for i, s in enumerate(steps) if i != classifiers[0] and marker_name in (s.get("run") or "")]
|
|
assert guard_idx, (
|
|
f"no step checks for the {marker_name!r} start marker. Without it a dropped classify step "
|
|
"concludes success and the merge gate is silently dead (ersatztv#751)"
|
|
)
|
|
# AFTER the classifier, not merely present. A guard placed before it would read a marker that
|
|
# has not been written yet and fail on every run — fail-closed, but it would deadlock `main` and
|
|
# read as this guard being broken, which is how a correct-looking guard gets deleted.
|
|
assert guard_idx[0] > classifiers[0], (
|
|
f"the dropped-step guard is step {guard_idx[0]} but the classifier is step "
|
|
f"{classifiers[0]} — a guard that runs first always fails"
|
|
)
|
|
guard = steps[guard_idx[0]]
|
|
# `always()` and `${{ always() }}` are the same condition; the runner accepts both and the second
|
|
# is the more common spelling. Pinning the bare form EXACTLY would red the repo over a
|
|
# semantically identical edit, so normalise instead. What must not change is that the guard runs
|
|
# when the classifier failed.
|
|
guard_if = re.sub(r"\s+", "", str(guard.get("if", "")))
|
|
assert guard_if in ("always()", "${{always()}}"), (
|
|
f"the dropped-step guard's `if:` is {guard.get('if')!r}; it must be `always()` (bare or "
|
|
"wrapped), or it will be skipped on exactly the runs where the classifier failed"
|
|
)
|
|
# INSIDE the missing-marker branch, not merely somewhere in the body. A bare `exit 1` substring
|
|
# is satisfied by an unreachable `if false; then exit 1; fi` while the real branch says
|
|
# `exit 0` — the test passes and a dropped classifier goes green again. The
|
|
# behavioural test below is the real proof; this keeps the structural one from being satisfiable
|
|
# by dead code.
|
|
missing_branch = re.search(r'if \[ ! -f "\$marker" \]; then(.*?)\bfi\b', guard["run"], re.S)
|
|
assert missing_branch, (
|
|
'the dropped-step guard no longer tests for a MISSING marker with `if [ ! -f "$marker" ]`, '
|
|
"so the assertion below cannot locate the branch that must fail the job"
|
|
)
|
|
assert re.search(r"exit\s+1", missing_branch.group(1)), (
|
|
"the dropped-step guard detects the missing marker but does not `exit 1` inside that branch, "
|
|
"so it observes the failure and still lets the job go green — which is the whole defect"
|
|
)
|
|
assert not _EXPR.search(guard["run"]), (
|
|
"the dropped-step guard's own run body contains an expression delimiter, so the mechanism "
|
|
"it guards against can drop the guard too — and that absence would be silent as well"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("terminator", ["null", "[]"], ids=["null-page", "empty-array-page"])
|
|
def test_the_fence_TRUSTS_the_count_and_POSTS_when_the_timeline_terminates(tmp_path, terminator):
|
|
"""The second defect found by the ersatztv#751 probe, and the one that actually kept the gate
|
|
from posting anything.
|
|
|
|
A page past the end of `/issues/{n}/timeline` is the JSON value `null` on this instance, not `[]`.
|
|
`count_pr_mutations` (named `count_retargets` then) gated on `type == "array"` and so treated
|
|
the real terminator as unreadable: the
|
|
walk never reached a validated empty page, `rt_ok` was never `yes` for ANY pull request, and the
|
|
fence therefore withheld every exemption `success`. Renovate and docs-only PRs got NO status —
|
|
the same user-visible outcome as #751, by an unrelated route.
|
|
|
|
It hid for two reasons worth keeping written down. It shipped in the same commit (8f6d4f443) that
|
|
stopped the step from executing, so the fence had never once run in production; and the test
|
|
double printed `[]` while its comment claimed to mirror measured reality, so the type gate was
|
|
never exercised by the suite either. A green suite over an unfaithful double is what let a
|
|
fail-closed-by-accident branch look deliberate.
|
|
|
|
Both shapes are asserted because both are live on this server: comments really do return `[]`.
|
|
Asserting the POSTED STATUS, not the log, because the log said `Decision: state=success` on the
|
|
real probe run and the job still posted nothing — the decision and the write are different events,
|
|
and only the write is what a merge reads.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path, _emitting("docs/a.md"), timeline_mode="stable:0", timeline_terminator=terminator
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is not None, (
|
|
f"a docs-only PR whose timeline terminates with `{terminator}` got NO status at all. The "
|
|
"fence could not establish a trusted retarget/push count, so it withheld the exemption — which "
|
|
"leaves the required review-verdict/h10 absent and the PR unmergeable with no bypass "
|
|
f"(ersatztv#751).\n{r.stdout[-1200:]}"
|
|
)
|
|
assert posted["state"] == "success", f"expected the docs-only exemption, got {posted}"
|
|
assert "trusted=yes" in r.stdout, (
|
|
"the exemption was posted but the fence did not report a trusted count — the two must agree, "
|
|
f"or this test is passing for a different reason than it claims.\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def _marker_steps():
|
|
"""(classify body PREFIX through the marker write, guard body that CHECKS it) — from one parse.
|
|
|
|
THE PREFIX, NOT THE MATCHING LINES. Picking out the lines matching `RAN_MARKER=`
|
|
and `: > "$RAN_MARKER"` and running those two alone passes even if the
|
|
write is moved into a function nobody calls, or under `if false`: the extractor finds the text,
|
|
runs it at top level, the marker appears, and the test reports the guard proven while production
|
|
writes no marker at all. Executing the real prefix — everything from the top of the body down to
|
|
and including the write — reproduces the production control flow instead of a reconstruction of
|
|
it, so a write that is defined-but-not-reached simply does not happen and case B fails.
|
|
#
|
|
It also pins the property the marker needs anyway: that the write happens EARLY, before anything
|
|
in the classifier can fail. If it drifts down past code that needs credentials or network, this
|
|
prefix stops executing cleanly and the test says so.
|
|
"""
|
|
steps = [s for _, s in _iter_workflow_steps(WORKFLOW)]
|
|
idx = [i for i, s in enumerate(steps) if "review-verdict/h10" in (s.get("run") or "")]
|
|
classify = steps[idx[0]]
|
|
lines = classify["run"].splitlines()
|
|
write_at = [i for i, ln in enumerate(lines) if re.match(r'\s*:\s*>\s*"\$RAN_MARKER"', ln)]
|
|
assert len(write_at) == 1, f'expected exactly one `: > "$RAN_MARKER"` in the classify body, found {len(write_at)}'
|
|
prefix = "\n".join(lines[: write_at[0] + 1])
|
|
guard = next(s for i, s in enumerate(steps) if i != idx[0] and "h10-classifier-started" in (s.get("run") or ""))
|
|
return prefix, guard["run"]
|
|
|
|
|
|
def test_the_dropped_step_guard_BEHAVIOURALLY_fails_without_the_marker_and_passes_with_it(tmp_path):
|
|
"""Executes the guard, instead of reading it — the structural test above cannot prove the exit
|
|
code, and a bare `exit 1` substring is satisfiable by dead code.
|
|
|
|
The marker is created by running THE CLASSIFY STEP'S OWN two prologue lines under the same
|
|
environment, never by rebuilding the path in Python. That is the point: it proves the two steps
|
|
AGREE on the path by construction. A test that computed the name itself would keep passing after
|
|
the two steps drifted apart, which is the one divergence that makes the guard fail on every run
|
|
and get deleted as broken.
|
|
|
|
`GITHUB_RUN_ID`/`GITHUB_RUN_ATTEMPT` are set to fixed values, so this also covers the run-keying
|
|
added after the live probes: if either step stopped interpolating them, the paths would differ and
|
|
case B would fail.
|
|
"""
|
|
prologue, guard = _marker_steps()
|
|
assert prologue.count("RAN_MARKER=") == 1 and ': > "$RAN_MARKER"' in prologue, (
|
|
f"could not extract the classify step's marker prologue; got {prologue!r}"
|
|
)
|
|
# The prefix must be the REAL top of the body, so a write hidden in an uncalled function is not
|
|
# executed by this test either.
|
|
assert prologue.lstrip().startswith("set -euo pipefail"), (
|
|
"the extracted prefix does not start at the top of the classify body, so it is a "
|
|
f"reconstruction rather than the production path: {prologue[:120]!r}"
|
|
)
|
|
env = {
|
|
"PATH": os.environ["PATH"],
|
|
# Built from scratch, so the isolated hook-fire log dir is carried explicitly rather than
|
|
# letting the sink fall back to a shared log (ersatztv#809).
|
|
hook_fire_isolation.ENV_VAR: os.environ[hook_fire_isolation.ENV_VAR],
|
|
"RUNNER_TEMP": str(tmp_path),
|
|
"GITHUB_RUN_ID": "424242",
|
|
"GITHUB_RUN_ATTEMPT": "7",
|
|
}
|
|
|
|
# A — the step was DROPPED: no marker exists. The job must fail.
|
|
a = subprocess.run(["bash", "-c", guard], env=env, capture_output=True, text=True)
|
|
assert a.returncode != 0, (
|
|
"the guard exited 0 with NO start marker present — a dropped classify step would go green "
|
|
f"again, which is the whole defect (ersatztv#751).\nstdout: {a.stdout}\nstderr: {a.stderr}"
|
|
)
|
|
assert "did not execute" in (a.stdout + a.stderr), (
|
|
f"the guard failed but without an actionable message: {a.stdout!r} {a.stderr!r}"
|
|
)
|
|
assert not list(tmp_path.glob("h10-classifier-started*")), (
|
|
"the guard itself created the marker it is supposed to be checking for"
|
|
)
|
|
|
|
# B — the classifier RAN: its own prologue created the marker. The guard must pass.
|
|
b = subprocess.run(["bash", "-c", prologue + "\n" + guard], env=env, capture_output=True, text=True)
|
|
assert b.returncode == 0, (
|
|
"the guard rejected a marker written by the classify step's OWN prologue — the two steps "
|
|
f"disagree on the path, so this guard would fail on every run.\nstdout: {b.stdout}\n"
|
|
f"stderr: {b.stderr}"
|
|
)
|
|
# C — the write must be reached by STRAIGHT-LINE code. Guarding the prefix trick itself: if the
|
|
# write were wrapped in a function or an `if`, the prefix would still contain it but production
|
|
# might not reach it. Executing the prefix with the function/conditional intact is the test; this
|
|
# assertion makes the intent explicit and fails loudly rather than subtly.
|
|
body_before = "\n".join(ln for ln in prologue.splitlines() if not ln.lstrip().startswith("#"))
|
|
# Covers all four bash spellings: `mk() {`, `mk(){`, `function mk {`, `function mk() {`. The third
|
|
# was missed by a regex over the first two spellings, and the FOURTH still slipped the regex extended to it —
|
|
# the union form is the natural next spelling once `function mk {` is caught. Budget three rounds
|
|
# for any string-matching predicate.
|
|
assert not re.search(r"^\s*(function\s+)?\w+\s*(\(\s*\))?\s*\{", body_before, re.M), (
|
|
"a function is defined before the marker write, so the write may be inside it and unreached "
|
|
f"in production while this test still passes:\n{body_before}"
|
|
)
|
|
|
|
written = [q.name for q in tmp_path.glob("h10-classifier-started*")]
|
|
assert written == ["h10-classifier-started-424242-7"], (
|
|
f"the marker is not keyed on the run id/attempt as intended; found {written}. A fixed name in "
|
|
"a shared RUNNER_TEMP lets a stale marker satisfy this guard on a run whose step was dropped"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("shape", ["null", "array"], ids=["statuses-null", "statuses-empty-array"])
|
|
def test_a_head_with_NO_statuses_YET_is_readable_and_still_gets_its_exemption(tmp_path, shape):
|
|
"""The twin of the timeline terminator (#751).
|
|
|
|
`GET /commits/{sha}/status` returns `{"state":"pending","total_count":0,"statuses":null}` for a
|
|
head that has no statuses yet — measured on PR #739's head 5fa672e2. `read_existing_verdict`
|
|
gated on `.statuses | type == "array"`, so it read that as unreadable and took its `exit 1`
|
|
path: the job posted NOTHING. Fail-closed, but the user-visible result is exactly the outcome
|
|
this issue is about — an exempt PR with no status and, since #743, no bypass.
|
|
|
|
Both shapes are asserted because a head that HAS statuses really does return an array, so the job
|
|
must read either. Restoring the `array`-only gate turns 40+ tests red with the corrected double,
|
|
which is the measure of how thoroughly the unfaithful stub was hiding this.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_empty_shape=shape)
|
|
assert r.returncode == 0, (
|
|
f"the classifier died reading a `statuses: {shape}` body instead of treating it as "
|
|
f"'no verdict yet', so nothing was posted at all.\n{r.stdout[-1000:]}\n{r.stderr[-600:]}"
|
|
)
|
|
assert posted is not None, (
|
|
f"a docs-only PR whose head has no statuses yet (shape: {shape}) got NO status at all.\n{r.stdout[-1000:]}"
|
|
)
|
|
assert posted["state"] == "success", f"expected the docs-only exemption, got {posted}"
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["CLAUDE.md", "AGENTS.md"])
|
|
def test_the_GOVERNANCE_docs_are_protected_and_get_no_docs_only_exemption(tmp_path, path):
|
|
"""`CLAUDE.md`/`AGENTS.md` are not prose about the project — they define the completion protocol,
|
|
the merge-consent convention and the H10 rule itself. `DOCS_ONLY` (`^(docs/|[^/]*\\.md$)`) matched
|
|
them, so a PR editing the document that specifies what `.claude/` enforces was auto-exemptible
|
|
while `.claude/` itself was protected: the same self-exemption the workflow header rules out, one
|
|
directory over.
|
|
|
|
Latent until #751, because no exemption `success` was writable at all while the classify step was
|
|
dropped — restoring the exemptions is what makes it reachable, which is why it is fixed there.
|
|
|
|
Asserted through the PROTECTED branch specifically, not merely "not exempt": `pending` is reached
|
|
by several routes and a test that accepted any of them could not tell a working guard from a dead
|
|
one.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting(path))
|
|
assert posted is not None, f"the job posted nothing: {r.stderr[-1500:]}"
|
|
assert posted["state"] == "pending", (
|
|
f"{path} was granted an exemption ({posted}) — the document that DEFINES the merge gate must "
|
|
f"not be able to exempt itself from it.\n{r.stdout[-800:]}"
|
|
)
|
|
assert "protected" in r.stdout.lower(), (
|
|
f"{path} was not exempted, but not via the protected-path branch either, so that guard may be "
|
|
f"dead for it.\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_terminator_on_PAGE_ONE_does_not_certify_a_zero_retarget_count(tmp_path):
|
|
"""`null` means "exhausted" only after a real page has been read.
|
|
|
|
THE INVARIANT, not a figure that rots: a real PR's timeline always carries at least one event on
|
|
page 1, because the PR is created by a push and that is itself an event. Spot-checked non-empty
|
|
across #752/#753/#749/#739/#717; the counts are not recorded, because they rot — of five once
|
|
cited here, three were stale within days. So a terminator on page 1 is anomalous, not
|
|
empty. Trusting a zero count from it would mean
|
|
certifying that no retarget happened on the strength of a response we cannot explain, which is the
|
|
one thing the fence exists to refuse. Withholding the exemption is the safe direction: the PR asks
|
|
for a human verdict instead.
|
|
|
|
This narrows rather than closes the concern — a wrong `null` on page 3 is still read as exhaustion,
|
|
and no bounded number of round-trips can rule that out.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="empty-first-page")
|
|
assert r.returncode == 0, r.stderr
|
|
_assert_withheld(tmp_path, r, "test_a_terminator_on_PAGE_ONE_does_not_certify_a_zero_retarget_count", expect_rc=0)
|
|
assert "trusted=no" in r.stdout, (
|
|
f"the exemption was withheld, but not because the count was untrusted:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_SECOND_PAGE_of_statuses_refuses_to_conclude_that_no_verdict_exists(tmp_path):
|
|
"""Completeness for the "no verdict exists" conclusion — the one that licenses posting over a
|
|
verdict this job cannot see.
|
|
|
|
THE TWO OBVIOUS GUARDS ARE BOTH NO-OPS HERE, which is why this costs a round-trip:
|
|
|
|
* `.statuses | length` vs `.total_count` — `total_count` is the count for the PAGE RETURNED, not
|
|
the commit. Measured at 1.27.1 on 3aed43c6 (6 contexts): `?limit=1` -> `len=1, total_count=1`.
|
|
Equal by construction. The stub mirrors that, so this test cannot pass for that wrong reason.
|
|
* "refuse when the page came back full at the requested limit of 100" — this instance caps `limit`
|
|
at `MAX_RESPONSE_ITEMS`, measured at 50 (`/issues?limit=100` returns 50), so a response can never
|
|
hold 100 rows and the comparison was DEAD CODE. The repo already documented that cap in three
|
|
places; the guard was written against 100 anyway.
|
|
|
|
So the job asks the server, and only when the row is absent from page 1. Any rows on page 2 mean
|
|
the list is longer than one page and the verdict may be beyond it.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="twopage")
|
|
_assert_withheld(
|
|
tmp_path, r, "test_a_SECOND_PAGE_of_statuses_refuses_to_conclude_that_no_verdict_exists", expect_rc=1
|
|
)
|
|
assert "page 2" in (r.stdout + r.stderr).lower(), (
|
|
f"nothing was posted, but not because of the page-2 completeness probe:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_SINGLE_page_of_statuses_reads_normally(tmp_path):
|
|
"""Positive control for the probe above: without it, "refuse when page 2 has rows" could be
|
|
satisfied by refusing always, which deadlocks every PR while looking safe. 40 decoy contexts on
|
|
page 1, nothing on page 2.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="rows:40")
|
|
assert r.returncode == 0, r.stderr
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"a single page of statuses should read normally and still exempt; got {posted}\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_STRING_total_count_is_not_accepted_as_numeric_zero(tmp_path):
|
|
"""`jq -r` renders the JSON number 0 and the JSON string "0" identically, so a text compare
|
|
accepts a schema-corrupted `"total_count": "0"` as "no statuses" (reproduced, not assumed).
|
|
|
|
The live schema uses an integer, so this is not a live failure — it is the difference between a
|
|
guard that holds because the input happens to be well-formed and one that holds because it checks.
|
|
The accept path requires the TYPE to be number.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="total-count-string")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a string total_count was accepted as numeric zero, so a body that merely lost its statuses "
|
|
f"array read as 'no verdict exists' and was classified normally: {posted}\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"mode,why",
|
|
[
|
|
("page2-garbage", "a non-JSON page 2"),
|
|
("page2-error", "an HTTP error on page 2"),
|
|
],
|
|
ids=["garbage", "transport-error"],
|
|
)
|
|
def test_an_UNREADABLE_page_2_refuses_to_conclude_that_no_verdict_exists(tmp_path, mode, why):
|
|
"""The two refuse branches of the completeness probe.
|
|
|
|
Worth a test rather than trusting the shape: this file's history is two consecutive guards that
|
|
were UNREACHABLE — the timeline walk's type gate that never saw a real terminator, and a
|
|
full-page check written against a limit of 100 on a server that caps at 50. An unexercised branch
|
|
here has a track record.
|
|
|
|
Both must fail CLOSED: the point of reading page 2 is to justify "no verdict exists", so a page 2
|
|
that cannot be read justifies nothing.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode=mode)
|
|
_assert_withheld(tmp_path, r, "test_an_UNREADABLE_page_2_refuses_to_conclude_that_no_verdict_exists", expect_rc=1)
|
|
assert "page 2" in (r.stdout + r.stderr).lower(), (
|
|
f"nothing was posted, but not because of the page-2 probe:\n{r.stdout[-800:]}"
|
|
)
|
|
|
|
|
|
def test_a_status_history_RUNNING_PAST_PAGE_1_is_PAGED_and_the_exemption_STANDS(tmp_path):
|
|
"""THE INVERSION (ersatztv#763). This test asserted the opposite until real paging existed, and
|
|
the change of expectation IS the fix.
|
|
|
|
Before: the post-write check read one clamped page, so "no raced row on page 1" did not establish
|
|
"no race", and #751's conservative probe treated any rows on page 2 as a race. A head with more
|
|
status rows than the cap therefore lost its exemption to a STICKY sentinel with no race anywhere
|
|
near it. That is not hypothetical — it fired on Renovate PR #761, whose head accumulated rows over
|
|
ordinary CI re-runs until the gate started reporting a human verdict as overwritten when none
|
|
existed.
|
|
|
|
After: the walk pages to a validated empty page, so rows beyond the first page are simply READ.
|
|
The stub's history is 60 ORDINARY rows — no verdict, no sentinel — and the correct answer is that
|
|
nothing raced this write, so the exemption must STAND.
|
|
|
|
This is the discriminating half of the pair: it is the assertion that goes red against the old
|
|
single-page read (which would repair here), while
|
|
`test_a_raced_verdict_on_PAGE_2_is_detected_and_repaired` goes red against a walk that pages but
|
|
stops early. Neither test alone can tell "pages correctly" from "repairs on everything".
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="second-page")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"the exemption was repaired away even though the whole history was readable and carried no "
|
|
f"verdict — the page-2 'assume raced' probe is still in force. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success", f"expected the exemption to stand; got {seq}"
|
|
# The false `::error::` from the old probe is the thing PR #761 could not be diagnosed from, so
|
|
# pin its absence too — a run that stands green but still shouts about an overwritten verdict
|
|
# would satisfy the count assertion above while leaving the log as misleading as before.
|
|
assert "was overwritten" not in (r.stdout + r.stderr), (
|
|
f"the exemption stood, but the log still claims a verdict was overwritten:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_raced_verdict_on_PAGE_2_is_detected_and_repaired(tmp_path):
|
|
"""The hole #763 exists to close, and the half a paging change can still get wrong.
|
|
|
|
A verdict sits BEYOND the first page — 60 ordinary rows with it inserted at index 55 — so only a
|
|
walk that continues past page 1 can find it. Under the pre-#763 code the exemption would still
|
|
have been repaired here, but for the wrong reason (rows merely EXISTED on page 2), which is why
|
|
this test is paired with the one above rather than standing alone: together they separate
|
|
"actually read page 2" from "repaired on any uncertainty".
|
|
|
|
SCOPE, stated because the double is ordering-blind: this proves WALK COMPLETENESS, not that a real
|
|
raced verdict would otherwise be missed. The live endpoint serves `created_unix DESC`, so a row
|
|
created inside the write window is among the newest and lands on page 1. The walk exists so that
|
|
the one fail-toward-SUCCESS path does not rest on that undocumented ordering.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="verdict-on-page-2")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a human verdict sitting on page 2 of the status history was not detected, so the exemption "
|
|
f"stands over a rejection nobody read. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
assert seq[1]["description"] == "Human verdict raced this exemption write — re-post the verdict", (
|
|
f"repaired, but not to the sentinel, so the fixed point is broken: {seq[1]}"
|
|
)
|
|
# It must be reported as a REAL overwrite, not as uncertainty — the two now say different things
|
|
# in the log, and a verdict genuinely found on page 2 is the case that earns the strong wording.
|
|
assert "was overwritten" in (r.stdout + r.stderr), (
|
|
f"repaired, but the log did not report an overwritten verdict:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_partial_mark_is_SAFE_because_the_newest_rows_are_on_page_1(tmp_path):
|
|
"""The partial-mark fallback's safety is a claim ABOUT THE ORDERING, so it is measured under the
|
|
real one — this is the only order-faithful fixture here.
|
|
|
|
The fallback exists because refusing a mark skips the post-write check entirely, which is a
|
|
fail-open. It is safe only because the server serves `created_unix DESC` and ids are monotonic
|
|
with `created_at`: page 1 therefore carries the true maximum, so a walk that fails on page 2 has
|
|
still seen it.
|
|
|
|
The history holds a PRE-EXISTING base-mismatched verdict at id 7055 — older than the newest row,
|
|
so nothing it does raced this write. Under DESC the salvaged mark is 7059 and that row is below
|
|
it: the exemption correctly STANDS. Under ASC (`sort=highestindex`, withdrawn) page 1
|
|
would be 7000..7049, the mark 7049, and that same untouched row would test as NEWER than the mark
|
|
— a sticky repair on a head nothing raced, which is exactly #761.
|
|
|
|
So re-adding the sort parameter reddens this test by BEHAVIOUR, not merely by the structural
|
|
assertion in `test_the_walk_does_not_request_a_SORT_order`.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="premark-page2-error-desc")
|
|
assert r.returncode == 0, r.stderr
|
|
log = r.stdout + r.stderr
|
|
# This pins that the walk really was INCOMPLETE — it does not prove the fallback ran, because the
|
|
# warning is emitted by a different condition than the one that salvages the mark. Deleting the
|
|
# fallback leaves this line green; the sibling test
|
|
# `test_a_PRE_WRITE_paging_failure_still_yields_a_usable_high_water_mark` is what reddens then.
|
|
assert "taking the high-water mark over the" in log, (
|
|
f"the pre-write walk completed, so this test is not exercising a partial mark at all:\n{r.stdout[-900:]}"
|
|
)
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"a PRE-EXISTING verdict was read as newer than the salvaged mark, so the exemption was "
|
|
f"repaired away on a head nothing raced. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success", f"expected the exemption to stand; got {seq}"
|
|
assert "was overwritten" not in log, (
|
|
f"nothing raced this write, but the log claims a verdict was overwritten:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_an_EMPTY_post_write_history_is_not_evidence_that_nothing_raced(tmp_path):
|
|
"""A well-formed response that cannot be true must not be trusted, and this one is free to reach.
|
|
|
|
The walk terminates on an empty page — correct BEFORE the write, where a head nothing has posted
|
|
to genuinely has no statuses. AFTER the write it is impossible: this job has just POSTed, and
|
|
`/statuses/{sha}` returns one row per POST, so "no statuses exist" contradicts a write that
|
|
succeeded. Nothing retries it either, because the body is valid JSON of the right type.
|
|
|
|
Accepting it yields `raced=0` from a list that cannot be real, on the one path whose failure
|
|
direction is toward SUCCESS — and silently, since the walk itself reports success. Whether a
|
|
verdict actually raced this particular write is not the point and is not modelled here: the
|
|
response is not evidence either way, so concluding "nothing raced" from it is unsound regardless.
|
|
|
|
This is NOT the withdrawn currency witness. That asked "is there ANY row above the mark", which an
|
|
unrelated newer row satisfied while the rejection stayed hidden. This asks only whether the list is
|
|
EMPTY — a state no unrelated row can produce and no ordering can disguise.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="null-page1-after-post")
|
|
assert r.returncode == 0, r.stderr
|
|
log = r.stdout + r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"an empty post-write history was accepted as proof that nothing raced this write, so an "
|
|
f"exemption stands on a head whose write window was never inspected. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
# THE RECONCILABLE SENTINEL, NOT THE REPAIR ONE (ersatztv#849). Nothing here established that a
|
|
# verdict was buried — the response was impossible, not informative — so claiming the repair
|
|
# sentinel's meaning would be the same overclaim the `::error::` below is careful to avoid. It
|
|
# also matters operationally: the repair sentinel is clearable only by a human, this one by the
|
|
# next run that can read the history.
|
|
assert seq[1]["description"] == UNVERIFIED_DESC, (
|
|
f"an unverifiable read was recorded as a buried human verdict: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "came back empty" in log, (
|
|
f"repaired, but the impossible empty read was not reported as the reason:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "was overwritten" not in log, (
|
|
f"an unverifiable read was reported as an overwritten human verdict:\n{r.stdout[-900:]}"
|
|
)
|
|
# THE REASON IS PART OF THE FIX, so it is asserted POSITIVELY. An absence assertion alone leaves
|
|
# `raced_why` unpinned — any string at all, including a placeholder, satisfies it while the
|
|
# operator gets a sticky sentinel explained by nothing. The em-dash and full stop discriminate the
|
|
# `::error::` reason from the `::warning::` text, which continues "…, which cannot be true".
|
|
assert "— the status history came back empty after this job posted to it." in log, (
|
|
"the repair did not report its own reason; the ::error:: carries whatever `raced_why` held "
|
|
f"and nothing pins it:\n{r.stdout[-900:]}"
|
|
)
|
|
# The walk COMPLETED here, on a validated terminator — the answer was impossible, not unreadable.
|
|
# Borrowing the unreadable-history wording would send an operator staring at a sticky sentinel
|
|
# looking for an API failure that never happened.
|
|
assert "could not be read completely" not in log, (
|
|
"an empty-but-complete read was reported as an incomplete one, which points at the wrong "
|
|
f"cause:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_NULL_page_terminates_the_walk_as_an_empty_one(tmp_path):
|
|
"""`/statuses/{sha}` returns `[]` past the end today; `/issues/{n}/timeline` returns bare `null`.
|
|
|
|
An array-only gate on the timeline was ersatztv#751: the walk never reached a validated empty
|
|
page, `rt_ok` was never `yes` for any PR, and the fence withheld EVERY exemption `success` from
|
|
the day it shipped. Re-adopting that narrowing on this endpoint would be worse, because this
|
|
walk's failure mode is the STICKY sentinel — every exempt PR would need a human verdict, per head,
|
|
clearable only by hand.
|
|
|
|
Tolerating `null` cannot misread `[]`, so it costs nothing. This asserts it rather than trusting
|
|
that the endpoint will never change its empty shape.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="null-terminator")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"a `null` page past the end was read as unreadable rather than as exhaustion, so the walk "
|
|
f"never validated and the exemption was repaired away. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success", f"expected the exemption to stand; got {seq}"
|
|
|
|
|
|
def test_a_STRING_id_on_a_PRE_EXISTING_row_is_not_read_as_raced(tmp_path):
|
|
"""The twin of `test_a_STRING_id_cannot_inflate_the_high_water_mark`, on the COMPARISON rather
|
|
than the maximum — and the twin was live on `main`, one expression away from the fix.
|
|
|
|
jq orders strings above every number, so `.id > $since` reads a string id as newer than ANY mark.
|
|
A pre-existing row carrying one — such as a base-mismatched verdict `read_existing_verdict`
|
|
deliberately declines to honour — is therefore counted as having raced this write on EVERY run of
|
|
that PR. Each run posts the sticky sentinel and a false "was overwritten", so the exemption is
|
|
lost permanently and no re-trigger clears it.
|
|
|
|
The row here is genuinely older than the mark (`"3"` against a mark of 9999) and must be invisible.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="string-id-preexisting-row")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"a PRE-EXISTING row with a string id was read as newer than the high-water mark and treated "
|
|
f"as a raced verdict, so this PR loses its exemption permanently. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success", f"expected the exemption to stand; got {seq}"
|
|
assert "was overwritten" not in (r.stdout + r.stderr), (
|
|
f"nothing raced this write, but the log claims a verdict was overwritten:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_history_with_NO_numeric_ids_SKIPS_the_check_instead_of_marking_zero(tmp_path):
|
|
"""An empty history and an unreadable one are different answers, and only one of them is 0.
|
|
|
|
A head nothing has posted to has no rows and its mark is 0 — correct, every later row is newer.
|
|
A NON-empty history in which no row carries a numeric id is a schema this job cannot read.
|
|
Reporting that as 0 claims a mark was established when none was, and every pre-existing row then
|
|
tests as newer than it.
|
|
|
|
The observable difference is the log: the unusable case says so and skips the post-write check.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="no-numeric-ids")
|
|
assert r.returncode == 0, r.stderr
|
|
log = r.stdout + r.stderr
|
|
assert "was not numeric" in log, (
|
|
"a history carrying no numeric id at all was silently treated as a mark of 0, so the "
|
|
f"degradation is invisible in the log:\n{r.stdout[-900:]}"
|
|
)
|
|
seq = _posted_sequence(tmp_path)
|
|
# THE SECOND HALF OF THE SAME ANSWER (ersatztv#849 route 1). Refusing to collapse the mark to 0
|
|
# is only half of it: with no mark, nothing verifies the write, so the exemption is withheld and
|
|
# the head is marked with the sticky sentinel rather than greened unverified.
|
|
assert len(seq) == 1, f"expected exactly one write, not a green plus a repair; got {seq}"
|
|
assert seq[0]["state"] == "pending" and seq[0]["description"] == UNVERIFIED_DESC, (
|
|
f"an unreadable id schema skipped the check AND kept the exemption: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_history_LONGER_THAN_THE_PAGE_CAP_is_not_treated_as_exhausted(tmp_path):
|
|
"""The page cap is a refusal, not a terminator.
|
|
|
|
Twenty requests of fifty rows cannot validate a history of 1050: the walk never reaches an empty
|
|
page, so it has NOT established that it saw everything. Treating the cap as exhaustion would hand
|
|
the post-write check a list it knows is partial and let it conclude "nothing raced" from it — on
|
|
the one path whose failure direction is toward SUCCESS.
|
|
|
|
Only 950 rows can ever be validated, incidentally, because the twentieth request has to be the
|
|
empty terminator.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="over-cap")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a history longer than the page cap was treated as fully read, so a raced verdict beyond it "
|
|
f"would have been concluded absent. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
assert "could not be read completely" in (r.stdout + r.stderr), (
|
|
f"repaired, but not reported as an incomplete read:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_walk_does_not_request_a_SORT_order(tmp_path):
|
|
"""A regression guard on a WITHDRAWAL, which otherwise nothing mechanical protects.
|
|
|
|
`sort=highestindex` returns index ASCENDING, which would put the OLDEST rows on page 1. The
|
|
partial-mark fallback then takes its maximum over the oldest rows, leaving every pre-existing row
|
|
above the mark and read as raced — a spurious STICKY repair, which is the #761 failure this change
|
|
exists to remove. It was added, measured, and withdrawn for exactly that reason.
|
|
|
|
Nothing about the code's behaviour reveals a re-added sort parameter (the tests' double is
|
|
ordering-blind by design), so the guard is structural: the request must not carry one.
|
|
"""
|
|
src = _classify_step()["run"]
|
|
# The REQUEST LINES ONLY — comments are excluded deliberately, because the withdrawal note inside
|
|
# `page_statuses` names the parameter in order to explain why it is not used. A guard that reads
|
|
# prose would fire on its own documentation.
|
|
requests = [line for line in src.splitlines() if "statuses/$SHA?" in line and not line.lstrip().startswith("#")]
|
|
assert requests, (
|
|
"no `/statuses/{sha}` request line found; the walk has been reshaped, so re-point this guard "
|
|
"rather than letting the sort-order assertion below pass vacuously"
|
|
)
|
|
offenders = [line.strip() for line in requests if "sort=" in line]
|
|
assert not offenders, (
|
|
"the status-history request carries a sort order again. ASC ordering puts the OLDEST rows on "
|
|
"page 1, which inverts the partial-mark fallback into a spurious sticky repair — the #761 "
|
|
f"failure this change removes. See the withdrawal note in `page_statuses`. Offending: {offenders}"
|
|
)
|
|
|
|
|
|
def test_a_STRING_id_cannot_inflate_the_high_water_mark(tmp_path):
|
|
"""jq sorts strings above every number, so one schema-corrupt id silently blinds the race check.
|
|
|
|
`max` over raw ids returns `"99999"` rather than the largest real id. `jq -r` then renders it as
|
|
`99999`, which sails through the `*[!0-9]*` numeric gate, and the mark is set far above anything
|
|
that exists. Every subsequent row — including a genuine human rejection racing the write — tests
|
|
as OLDER than the mark and is invisible, so the exemption stands over it.
|
|
|
|
It fails toward SUCCESS and needs no attacker: one corrupt row is enough. So the mark is taken
|
|
over numeric ids only, and a non-numeric id is excluded rather than coerced.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="string-id-inflates-mark")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a string id inflated the high-water mark, so the raced verdict tested as older than it and "
|
|
f"the exemption stands over a human rejection. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
|
|
|
|
def test_a_TRANSIENT_page_failure_is_retried_and_the_walk_still_completes(tmp_path):
|
|
"""The retry clause, which was unproven code until this fixture existed.
|
|
|
|
Every other error mode here fails on EVERY attempt, so a one-shot walk and a retrying walk behave
|
|
identically against them — mutating `for try in 1 2` to `for try in 1` reddened nothing. That
|
|
matters because the retry is the stated reason failing closed is affordable: without it, one
|
|
momentary blip costs the head its exemption permanently, since the repair sentinel is sticky.
|
|
|
|
Here page 2 fails the first attempt of each logical read and succeeds on the retry. The history
|
|
carries no verdict, so the correct outcome is a completed walk and a STANDING exemption.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="flaky-page2")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, (
|
|
"a transient page failure was not retried, so the walk gave up and the exemption was repaired "
|
|
f"away on a head that nothing raced. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success", f"expected the exemption to stand; got {seq}"
|
|
|
|
|
|
def test_a_PRE_WRITE_read_that_returns_NOTHING_WITHHOLDS_the_exemption(tmp_path):
|
|
"""The boundary of the partial-list fallback, and what happens past it (ersatztv#849 route 1).
|
|
|
|
A partial list still yields a usable mark. A read that returns NO rows at all cannot: there is
|
|
nothing to take a maximum over, so `max_id_before` stays -1 and NOTHING can check the write
|
|
afterwards. This used to post the exemption anyway, with the race check skipped — a green on a
|
|
head whose write window was never inspected, which is the fail-toward-SUCCESS direction on the
|
|
one path where it is never acceptable.
|
|
|
|
The exemption is now withheld BEFORE the write rather than posted and repaired: the defect is
|
|
known before the POST, so publishing a green and taking it back would only open a window for
|
|
branch protection — and an already-scheduled auto-merge — to see it.
|
|
|
|
THE SENTINEL, NOT A GENERIC `pending`. That distinction is the whole reason #742's attempt at
|
|
this was withdrawn: a generic `pending` is precisely what a later run re-derives into `success`.
|
|
Asserting the description here is asserting the fix, not its packaging.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="premark-page1-error")
|
|
assert r.returncode == 0, r.stderr
|
|
log = r.stdout + r.stderr
|
|
assert "Could not establish a status high-water mark" in log, (
|
|
f"the mark was abandoned silently, with no record of the degradation:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "taking the high-water mark over the" not in log, (
|
|
f"a mark was salvaged from an empty read, which there is nothing to compute:\n{r.stdout[-900:]}"
|
|
)
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1, f"expected exactly one write — the sentinel, not a green followed by a repair; got {seq}"
|
|
assert seq[0]["state"] == "pending" and seq[0]["description"] == UNVERIFIED_DESC, (
|
|
f"an exemption was posted with nothing able to verify it: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_creator_row_does_not_kill_the_job_after_the_green_is_posted(tmp_path):
|
|
"""A row whose `creator` is not an object used to be fatal, at the worst possible moment.
|
|
|
|
`.creator != null and .creator.login` hard-errors in jq on any non-object creator; jq exits 5 and
|
|
under `set -euo pipefail` the assignment takes the step down. That happens AFTER the exemption
|
|
`success` has been posted and BEFORE the repair is attempted, so one schema-corrupt row leaves a
|
|
green standing on a head that carries a genuine human rejection — and the job reports failure in a
|
|
way that looks like an unrelated infrastructure error.
|
|
|
|
The fix type-tests `creator` before indexing it, so the malformed row is dropped from the count
|
|
while the real verdict beside it is still counted and still repaired.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="malformed-creator-beside-verdict")
|
|
assert r.returncode == 0, f"the step died instead of skipping a malformed row: {r.stderr[-1200:]}"
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a malformed `creator` row suppressed the repair, leaving the exemption green over the real "
|
|
f"verdict beside it. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
# THE REASON, not just the outcome — this is what makes the type guard individually provable.
|
|
# Three clauses can each rescue this fixture (the type test, the `|| raced=""` guard, and the
|
|
# fail-closed unusable-count branch), so the repair alone cannot tell them apart. Only the type
|
|
# test lets the REAL verdict beside the malformed row be counted as a genuine human race; without
|
|
# it the count comes back empty and the repair is reported as an unusable count instead.
|
|
assert "was overwritten" in (r.stdout + r.stderr), (
|
|
"the malformed row suppressed the real verdict beside it — repaired, but as an unverifiable "
|
|
f"read rather than as the human rejection it is:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_verdict_BEYOND_A_SHORT_PAGE_is_still_found(tmp_path):
|
|
""" "Terminate only on a validated EMPTY page, never on a short one" — the rule that is invisible
|
|
against a faithful double.
|
|
|
|
On the real endpoint a short page IS the last page (measured: 50, 50, 14, `[]`), so an
|
|
implementation that stops at the first short page is indistinguishable from a correct one, and
|
|
every other test here would pass against it. The only way to observe the property is to model what
|
|
it actually guards: a TRUNCATED response. This stub serves 50 rows, then a short page of 10, then
|
|
a third page carrying the raced verdict.
|
|
|
|
A walk that treats the short page as exhaustion never reads page 3, misses the verdict, and leaves
|
|
the exemption green over a human rejection.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="verdict-after-short-page")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a raced verdict beyond a SHORT page was not found, so the walk stopped on a short page "
|
|
f"instead of on a validated empty one. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
assert "was overwritten" in (r.stdout + r.stderr), (
|
|
f"repaired, but not reported as a found verdict:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_PRE_WRITE_paging_failure_still_yields_a_usable_high_water_mark(tmp_path):
|
|
"""The fail-open a paging change can introduce while fixing one (ersatztv#763).
|
|
|
|
The high-water mark gates the post-write race check entirely: `max_id_before=-1` skips it, so a
|
|
human rejection landing in the write window is neither detected nor repaired. Before paging, only
|
|
a failure of the single page-1 request could reach that. Requiring a COMPLETE walk for the mark
|
|
would newly route a page-2 hiccup, an over-cap history, or one malformed id on a later page into
|
|
the same hole — a WIDER fail-open than the bug being fixed.
|
|
|
|
So a partial list still yields a mark. It can only be LOWER than the true maximum, which makes the
|
|
check more eager, never blinder.
|
|
|
|
Here page 2 fails on the PRE-write read only; the post-write read pages cleanly and carries a raced
|
|
verdict. With the mark salvaged from page 1 the verdict is above it and the exemption is repaired.
|
|
With `max_id_before=-1` the check never runs and the rejection stays green.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="premark-page2-error")
|
|
assert r.returncode == 0, r.stderr
|
|
log = r.stdout + r.stderr
|
|
assert "taking the high-water mark over the" in log, (
|
|
f"the partial-list fallback did not run, so the mark was not salvaged:\n{r.stdout[-900:]}"
|
|
)
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"the pre-write read could not be paged completely, the mark was abandoned, and the post-write "
|
|
f"race check was skipped — leaving a raced rejection green. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"mode,why",
|
|
[
|
|
("second-page-garbage", "a non-JSON page 2"),
|
|
("second-page-error", "an HTTP error on page 2"),
|
|
],
|
|
ids=["garbage", "transport-error"],
|
|
)
|
|
def test_an_UNREADABLE_history_page_2_also_repairs_rather_than_leaving_green(tmp_path, mode, why):
|
|
"""Found by mutating the branch and watching nothing go red — my own coverage gap, in the same
|
|
class the review had just flagged twice.
|
|
|
|
This is the fail-toward-SUCCESS path, so uncertainty must resolve to `pending`. "I could not read
|
|
the rest of the history" is not evidence that no verdict raced this write; treating it as such is
|
|
exactly how a forged green survives over a human rejection.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode=mode)
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
f"the exemption was left standing despite {why} — a raced verdict beyond page 1 would be "
|
|
f"buried. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success" and seq[1]["state"] == "pending", (
|
|
f"expected an exemption then a repair to pending; got {seq}"
|
|
)
|
|
# THE REASON IS PART OF THE FIX, so it is asserted. #761's stall was undiagnosable precisely
|
|
# because an uncertainty repair reported itself as an overwritten human verdict. Without these
|
|
# two the `raced_why` branch is unpinned — forcing it back to `human` reddens nothing.
|
|
log = r.stdout + r.stderr
|
|
assert "could not be read completely" in log, (
|
|
f"repaired, but not reported as an incomplete read:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "was overwritten" not in log, (
|
|
"an uncertainty repair is reported as an overwritten human verdict — the false ::error:: that "
|
|
f"made PR #761's stall undiagnosable:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
# --- ersatztv#849: post-write verification, the five routes ------------------------------------
|
|
#
|
|
# Each route left an exemption `success` — or a generic `pending` a later run re-derives into one —
|
|
# standing over a human `failure`. Each behavioural assertion below is paired with a MUTATION that
|
|
# disarms the shipped clause it names, so the proof shows THAT clause is what produces the outcome.
|
|
#
|
|
# BE PRECISE ABOUT WHAT THE MUTANTS ARE, because "restores the exact predecessor" is true of only
|
|
# some of them and claiming it of all would be the overclaim this repo treats as its own defect:
|
|
#
|
|
# * `test_MUTATION_a_SUCCESS_only_post_write_gate_...` and
|
|
# `test_MUTATION_restoring_the_default_operator_on_the_description_...` DO restore `origin/main`
|
|
# text verbatim; `test_MUTATION_restoring_the_SUCCESS_only_no_mark_downgrade_...` and
|
|
# `test_MUTATION_reading_a_malformed_FIELD_as_absent_...` restore text an EARLIER COMMIT ON THIS
|
|
# BRANCH shipped, which is where survivors were found.
|
|
# * `test_MUTATION_a_GENERIC_pending_...` restores the shape #742 attempted and WITHDREW, not
|
|
# `main` — which had no downgrade at all.
|
|
# * `test_MUTATION_restoring_the_SUCCESS_only_no_mark_downgrade_...` restores the predicate this
|
|
# branch itself shipped one commit earlier, which is where it survived the
|
|
# whole suite: the nearest existing proof mutated the DESCRIPTION the downgrade writes, not its
|
|
# SCOPE, and its fixture ran a succeeding enumeration, so `state=success` there and the
|
|
# `success`-only predecessor fired identically.
|
|
#
|
|
# TWO GUARDS HERE ARE OUTCOME-REDUNDANT AND WOULD OTHERWISE HIDE EACH OTHER: the `$own` exclusions
|
|
# and the no-op-repair skip both suppress the same duplicate POST, so mutating either alone leaves
|
|
# the post sequence unchanged. What the exclusions alone decide is the REPORT — without them a run
|
|
# counts its own row and tells a reviewer their verdict was overwritten when nothing raced it — so
|
|
# their proofs assert the LOG. That is the honest discriminator, not a weaker one.
|
|
# * the `if false` mutants disarm clauses that have NO predecessor, because the blocks they gate
|
|
# are new. Disarming the condition isolates the DECISION from the round-trip beside it, which
|
|
# deleting the block would not; they are counterfactual mutants and are sound as such.
|
|
#
|
|
# The count assertion in `_run_classify(mutate=...)` is what keeps every one of them bound: a clause
|
|
# that has since been reworded substitutes zero times and fails loudly rather than quietly measuring
|
|
# the unmutated body.
|
|
|
|
|
|
def test_a_verdict_racing_a_PENDING_write_is_repaired_to_the_repair_sentinel(tmp_path):
|
|
"""Route 2. The post-write check used to run only after an exemption `success`.
|
|
|
|
The reasoning was "a `pending` cannot turn a rejection green, so the only write that can cause
|
|
the damage is the exemption". The counter-example is ordinary: a docs-only PR hits a transient
|
|
enumeration failure, so the run writes the GENERIC `pending` — which masks a rejection landing in
|
|
its own write window exactly as a `success` would, and got no verification because of the state
|
|
test. The next run sees an ordinary machine `pending`, re-derives it into an exemption, and the
|
|
human's row is now below THAT run's high-water mark and invisible.
|
|
|
|
A failing enumeration is what selects the generic-`pending` path, which is why the stub exits 1.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="human-after-post",
|
|
)
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a human verdict landed in the write window of a generic `pending` and nothing verified it, "
|
|
f"so a later run can re-derive that pending into an exemption. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "pending" and seq[0]["description"] != REPAIR_DESC, (
|
|
f"expected the run's own generic pending first; got {seq}"
|
|
)
|
|
assert seq[1]["description"] == REPAIR_DESC, (
|
|
f"the raced pending was not repaired to the sticky sentinel: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_a_SUCCESS_only_post_write_gate_leaves_a_raced_PENDING_unrepaired(tmp_path):
|
|
"""The predecessor restored: `main`'s `state = success` conjunct on the post-write gate.
|
|
|
|
Without the pairing this is the shape ersatztv#787 catalogues as a test that passes for the wrong
|
|
reason — the fixture reaches the repair, but nothing shows the WIDENED gate is what took it there
|
|
rather than some other clause.
|
|
"""
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="human-after-post",
|
|
mutate=(
|
|
'if [ "$max_id_before" -ge 0 ]; then',
|
|
'if [ "$state" = "success" ] && [ "$max_id_before" -ge 0 ]; then',
|
|
),
|
|
)
|
|
seq = _posted_sequence(tmp_path / "mutant")
|
|
assert len(seq) == 1, (
|
|
"the mutant repaired anyway, so this fixture does not reach the repair through the widened "
|
|
f"gate and the test above proves nothing about it: {seq}"
|
|
)
|
|
assert seq[0]["state"] == "pending" and seq[0]["description"] != REPAIR_DESC, (
|
|
f"expected the unrepaired generic pending under the mutant; got {seq}"
|
|
)
|
|
|
|
|
|
def test_a_raced_PENDING_write_repairs_and_the_repair_SURVIVES_the_next_run(tmp_path):
|
|
"""The CHAINED form, which is the only shape that can assert a fixed point.
|
|
|
|
A single-hop test shows run N repairs. It cannot show the repair SURVIVES, and that is exactly
|
|
where route 2's damage lives — the burial arrives one event later, in run N+1, when a machine
|
|
`pending` is re-derived into an exemption. So run N's real output is fed in as run N+1's existing
|
|
status, with run N+1's enumeration now succeeding as a clean docs-only PR: the run that would
|
|
have granted the exemption is the one that must refuse.
|
|
"""
|
|
first, r1 = _run_classify(
|
|
tmp_path / "run1",
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="human-after-post",
|
|
)
|
|
seq1 = _posted_sequence(tmp_path / "run1")
|
|
assert seq1 and seq1[-1]["description"] == REPAIR_DESC, (
|
|
f"run 1 did not repair, so there is no chain to test: {seq1}\n{r1.stdout[-900:]}"
|
|
)
|
|
|
|
second, r2 = _run_classify(
|
|
tmp_path / "run2",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=seq1[-1]["description"], # <-- the chain: run N's own output
|
|
)
|
|
assert second is not None, f"run 2 posted nothing: {r2.stderr[-800:]}"
|
|
assert second["state"] == "pending", (
|
|
"run 2 re-derived run 1's repair into an exemption, burying the human verdict one event "
|
|
f"later than the write that raced it: {second}\n{r2.stdout[-900:]}"
|
|
)
|
|
assert second["description"] == seq1[-1]["description"], (
|
|
f"the repair is not a fixed point across the chain: {seq1[-1]['description']!r} -> {second['description']!r}"
|
|
)
|
|
|
|
|
|
def test_the_unverified_sentinel_is_a_FIXED_POINT_while_it_cannot_be_reconciled(tmp_path):
|
|
"""Route 1, chained. Withholding the exemption is only half the fix; the mark has to STICK.
|
|
|
|
#742's attempt withheld the exemption by writing a GENERIC `pending`, and a generic `pending` is
|
|
precisely what a later run re-derives into `success`. It moved which run posted the forged green
|
|
rather than stopping it, which is why it was withdrawn. So the property is a fixed point and only
|
|
a chain can assert it: run 1 fails to establish a mark and writes the sentinel; run 2 carries
|
|
run 1's own description, STILL cannot read the history to reconcile it, and must write the same
|
|
thing again rather than exempting a head nothing has ever verified.
|
|
"""
|
|
first, r1 = _run_classify(tmp_path / "run1", _emitting("docs/a.md"), history_mode="premark-page1-error")
|
|
assert first is not None and first["description"] == UNVERIFIED_DESC, (
|
|
f"run 1 did not write the sentinel: {first}\n{r1.stdout[-900:]}"
|
|
)
|
|
|
|
second, r2 = _run_classify(
|
|
tmp_path / "run2",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=first["description"], # <-- the chain
|
|
history_mode="premark-page1-error", # and still unreadable, so it cannot be reconciled away
|
|
)
|
|
assert second is not None, f"run 2 posted nothing: {r2.stderr[-800:]}"
|
|
assert second["state"] == "pending" and second["description"] == first["description"], (
|
|
"the unverified sentinel decayed: run 1's own output did not re-trigger the refusal, so run 2 "
|
|
f"exempted a head whose write was never verified ({first['description']!r} -> {second})"
|
|
f"\n{r2.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_a_GENERIC_pending_for_an_unverifiable_write_is_re_derived_into_an_exemption(tmp_path):
|
|
"""The predecessor restored: #742's withdrawn attempt, which wrote the GENERIC description.
|
|
|
|
This is the proof that the SENTINEL is the fix and not merely its packaging, and it needs both
|
|
arms of a chain because run 1 is indistinguishable either way — the mutant and the shipped code
|
|
post the same `pending` STATE, differing only in a description nothing has read yet.
|
|
|
|
Run 2 is where they diverge, and the fixture is the case that matters: a head that DOES carry a
|
|
buried verdict, below whatever high-water mark run 2 takes. The shipped chain recognises its own
|
|
sentinel, reconciles, finds the verdict and refuses. The mutant chain sees an ordinary machine
|
|
`pending`, re-derives it into a docs-only exemption, and the post-write check cannot save it —
|
|
the verdict predates the mark, so it is invisible and the green stands over it. That is route 1's
|
|
damage arriving one event later, which is exactly what "sticky" prevents and "generic" does not.
|
|
"""
|
|
generic = (
|
|
' state=pending\n desc="$UNVERIFIED_DESC"\nfi\n\n# LAST-MOMENT RE-READ',
|
|
' state=pending\n desc="Awaiting review verdict for ${SHA:0:7}"\nfi\n\n# LAST-MOMENT RE-READ',
|
|
)
|
|
|
|
# --- the mutant chain ---
|
|
m1, rm1 = _run_classify(
|
|
tmp_path / "mutant1",
|
|
_emitting("docs/a.md"),
|
|
history_mode="premark-page1-error",
|
|
mutate=generic,
|
|
)
|
|
assert m1 is not None and m1["state"] == "pending", (
|
|
f"the mutant did not withhold the exemption either, so this is not the chain under test: {m1}"
|
|
f"\n{rm1.stdout[-900:]}"
|
|
)
|
|
assert m1["description"] != UNVERIFIED_DESC, f"the mutation did not take — run 1 still wrote the sentinel: {m1}"
|
|
m2, rm2 = _run_classify(
|
|
tmp_path / "mutant2",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=m1["description"], # <-- the chain
|
|
history_mode="stale-human-already-present", # a verdict IS buried on this head
|
|
)
|
|
assert m2 is not None and m2["state"] == "success", (
|
|
"the generic-`pending` mutant was NOT re-derived into an exemption, so the shipped arm below "
|
|
f"is not discriminating on the description: {m2}\n{rm2.stdout[-900:]}"
|
|
)
|
|
|
|
# --- the shipped chain, same fixture ---
|
|
f1, rf1 = _run_classify(tmp_path / "fixed1", _emitting("docs/a.md"), history_mode="premark-page1-error")
|
|
assert f1 is not None and f1["description"] == UNVERIFIED_DESC, f"run 1: {f1}"
|
|
f2, rf2 = _run_classify(
|
|
tmp_path / "fixed2",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=f1["description"], # <-- the chain
|
|
history_mode="stale-human-already-present",
|
|
# Seeded on THIS arm only, and the asymmetry is the fixture being faithful rather than
|
|
# convenient: the shipped run 1 POSTed the sentinel, so its row is in the per-POST history;
|
|
# the mutant's run 1 POSTed a generic `pending`, so no sentinel row exists on that head.
|
|
history_extra=[_sentinel_row()],
|
|
)
|
|
assert f2 is not None and f2["state"] == "pending" and f2["description"] == REPAIR_DESC, (
|
|
"the shipped chain re-derived its own sentinel into an exemption over a buried verdict: "
|
|
f"{f2}\n{rf2.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def _sentinel_row(row_id=4000):
|
|
"""The row an earlier run POSTed when it wrote the unverified sentinel.
|
|
|
|
A fixture that puts the sentinel on the COMBINED endpoint without putting it in the per-POST
|
|
history describes a head that cannot exist: `/statuses/{sha}` returns one row per POST, so the
|
|
sentinel the combined endpoint is showing must be in there. Seeding it is not decoration — the
|
|
reconciliation now requires exactly this row as its witness, and without it these tests pass
|
|
against the impossible shape.
|
|
"""
|
|
return {
|
|
"id": row_id,
|
|
"context": "review-verdict/h10",
|
|
"status": "pending",
|
|
"creator": None,
|
|
"description": UNVERIFIED_DESC,
|
|
}
|
|
|
|
|
|
def test_the_unverified_sentinel_is_RECONCILED_AWAY_once_the_history_can_be_read(tmp_path):
|
|
"""The bound on the stall, and the reason this is a second sentinel rather than the repair one.
|
|
|
|
The repair sentinel records a fact that stays true — a verdict existed and was buried — so only a
|
|
human can clear it. "I could not read the history" expires: a run that CAN read it settles the
|
|
question. Without this the fix would trade route 1's fail-open for a permanent stall on a
|
|
transient API failure, which is the trade #742 was withdrawn for making.
|
|
|
|
Here the history is readable and carries no verdict row, so nothing was masked and the sentinel is
|
|
cleared — the docs-only exemption is granted on its own merits.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
# A REAL history: the sentinel's own row, plus an unrelated one so "non-empty" is not what is
|
|
# being tested. Without the sentinel row this fixture is the impossible shape and the test
|
|
# passed against a defect — the reconciliation used to clear over an EMPTY history, which
|
|
# cannot be true for a head whose combined endpoint is showing a row.
|
|
history_extra=[
|
|
{"id": 3900, "context": "ci/other", "status": "success", "creator": None, "description": "unrelated"},
|
|
_sentinel_row(),
|
|
],
|
|
)
|
|
assert posted is not None, f"nothing was posted: {r.stderr[-800:]}"
|
|
assert posted["state"] == "success", (
|
|
"a readable history carrying no verdict left the sentinel standing, so a transient failure "
|
|
f"costs a head its exemption permanently: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "Reconciled" in (r.stdout + r.stderr), (
|
|
f"the exemption was granted without the reconciliation running:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_reconciliation_UPGRADES_to_the_repair_sentinel_when_a_verdict_is_BURIED(tmp_path):
|
|
"""The other direction, and the one that must never resolve to an exemption.
|
|
|
|
A verdict masked by an unverified write is invisible on the COMBINED endpoint — that returns the
|
|
latest row per context, and the latest row is the sentinel. It is still in `/statuses/{sha}`,
|
|
which returns one row per POST, and that asymmetry is what makes reconciliation possible at all.
|
|
|
|
Finding one turns an open question into an established fact, so it upgrades to the repair
|
|
sentinel — clearable only by a human — rather than clearing.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
history_mode="stale-human-already-present",
|
|
history_extra=[_sentinel_row()],
|
|
)
|
|
assert posted is not None, f"nothing was posted: {r.stderr[-800:]}"
|
|
assert posted["state"] == "pending" and posted["description"] == REPAIR_DESC, (
|
|
f"a verdict buried under an unverified write was reconciled into an exemption: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "verdict row(s) underneath an unverified" in (r.stdout + r.stderr), (
|
|
f"the upgrade happened without saying why:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_reconciling_on_an_UNREADABLE_history_re_exempts_a_head(tmp_path):
|
|
"""The clause that must not be relaxed: only a COMPLETE walk may clear the sentinel.
|
|
|
|
An unreadable reconciliation is the same "cannot tell" every other branch here resolves to
|
|
pending. Treating it as "nothing buried" is the fail-open, and it is a tempting simplification
|
|
because the happy path looks identical.
|
|
|
|
THE MUTATION ISOLATES THE COMPLETENESS OPERAND, which needs a fixture where the OTHER operand
|
|
is satisfied. `reconcile-page2-error` serves page 1 — carrying the seeded sentinel, so the
|
|
witness is 1 — and fails page 2 of the reconciliation walk, so `ph_ok` is `no`. Dropping the
|
|
completeness operand therefore clears the sentinel over a list the job knows it did not finish
|
|
reading, which is where a buried verdict would be. A fixture with BOTH operands false, mutating
|
|
the whole condition to `if false`, disarms two guards at once and isolates neither.
|
|
|
|
That the cleared sentinel can be sitting on a real verdict is shown by
|
|
`test_the_reconciliation_UPGRADES_to_the_repair_sentinel_when_a_verdict_is_BURIED`, which is the
|
|
test that supplies one.
|
|
"""
|
|
seed = [_sentinel_row()]
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
history_mode="reconcile-page2-error",
|
|
history_extra=seed,
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"an incomplete reconciliation did not carry the sentinel forward: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
history_mode="reconcile-page2-error",
|
|
history_extra=seed,
|
|
mutate=(
|
|
'[ "$ph_ok" != yes ] || [ "$witness" -eq 0 ]; then',
|
|
'[ "$witness" -eq 0 ]; then',
|
|
),
|
|
)
|
|
assert mutant is not None and mutant["state"] == "success", (
|
|
"the mutant did not re-exempt, so the trust test is not what holds the sentinel and the "
|
|
f"assertion above proves nothing about it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_an_IMPOSSIBLE_EMPTY_history_does_NOT_reconcile_the_sentinel_away(tmp_path):
|
|
"""A complete-but-empty history is the one shape this read cannot legitimately return.
|
|
|
|
`ex_unverified=yes` means the COMBINED endpoint just returned the sentinel for this sha, and
|
|
`/statuses/{sha}` keeps one row per POST — so the sentinel it is showing MUST be in the history.
|
|
A complete walk that comes back empty therefore contradicts a write that demonstrably happened,
|
|
exactly as the post-write check already argues about its own read.
|
|
|
|
It is reachable rather than theoretical: `page_statuses` deliberately accepts an empty page 1 as
|
|
complete, because a head nothing has posted to genuinely has no statuses and the high-water mark
|
|
needs that answer. Clearing on it let the run classify normally and exempt a head whose sentinel
|
|
may have been sitting on a rejection.
|
|
|
|
The RECONCILED_AWAY test above must not use this fixture: the clear path would then be asserted
|
|
only against the impossible shape.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
) # no history_extra: the walk completes over ZERO rows
|
|
assert posted is not None, f"nothing was posted: {r.stderr[-800:]}"
|
|
assert posted["state"] == "pending" and posted["description"] == UNVERIFIED_DESC, (
|
|
"an impossible empty history reconciled the sentinel away, so the run exempted a head whose "
|
|
f"write was never verified: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "sentinel rows found=0" in (r.stdout + r.stderr), (
|
|
f"the sentinel was carried forward, but not because the witness was missing:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_reconciliation_WITNESS_clears_on_an_impossible_history(tmp_path):
|
|
"""Disarming the witness conjunct alone, so the empty-history refusal is isolated.
|
|
|
|
`ph_ok` is `yes` on this fixture — the walk really does complete — so the trust half of the
|
|
condition cannot be what refuses. Only the witness count can, and dropping it must produce the
|
|
exemption.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
mutate=(
|
|
'[ "$ph_ok" != yes ] || [ "$witness" -eq 0 ]; then',
|
|
'[ "$ph_ok" != yes ]; then',
|
|
),
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"the mutant did not clear the sentinel, so the witness conjunct is not what refuses on an "
|
|
f"empty history and the test above proves nothing about it: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_description_row_does_not_kill_the_post_write_count(tmp_path):
|
|
"""Route 4, the twin of the malformed-`creator` row already covered above.
|
|
|
|
`(.description // "") | startswith(...)` does not protect anything: `//` replaces `null` and
|
|
`false`, so a NUMERIC description survives it and `startswith` hard-errors on a number. jq exits
|
|
5, the count comes back unusable, and while the surrounding code now fails closed, the cost is
|
|
the whole count — a genuine verdict on another row is lost with it. Type-testing drops the
|
|
malformed row and keeps the real one countable, which is exactly what the `creator` guard beside
|
|
it already does.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), history_mode="malformed-description-beside-verdict")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2 and seq[1]["state"] == "pending", (
|
|
f"the raced verdict beside a malformed-description row was not repaired: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[1]["description"] == REPAIR_DESC, (
|
|
"the genuine verdict was lost with the malformed row, so the repair reports uncertainty "
|
|
f"rather than the verdict it should have counted: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_restoring_the_default_operator_on_the_description_loses_the_verdict(tmp_path):
|
|
"""The predecessor restored. `//` looks like a guard and is not one for a non-null wrong type.
|
|
|
|
The mutant still repairs — the surrounding unusable-count branch fails closed — so the STATE is
|
|
identical and only the reported reason separates them. That is the discriminator this proof
|
|
turns on: the shipped code counts the verdict, the mutant cannot see it and repairs on
|
|
uncertainty instead, which is the difference between telling a reviewer their verdict was
|
|
overwritten and telling them nothing could be checked.
|
|
"""
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
history_mode="malformed-description-beside-verdict",
|
|
mutate=(
|
|
' and ((.description | type) == "string")\n'
|
|
' and (.description | startswith("Review-verdict:")))',
|
|
' and (((.description // "") | startswith("Review-verdict:"))))',
|
|
),
|
|
)
|
|
seq = _posted_sequence(tmp_path / "mutant")
|
|
assert len(seq) == 2 and seq[1]["description"] != REPAIR_DESC, (
|
|
"the mutant still counted the verdict, so the type test is not what makes it countable and "
|
|
f"the test above proves nothing about it: {seq}"
|
|
)
|
|
|
|
|
|
def test_a_RETARGET_AFTER_the_POST_replaces_the_exemption(tmp_path):
|
|
"""Route 3. The fence narrows "writing while overtaken"; it never covered "overtaken after
|
|
writing", and that second window is the one that leaves a PERMANENT forged green.
|
|
|
|
`main` carries a real `failure` for head H. The PR is retargeted to a scratch base where H is
|
|
docs-only; run S classifies, derives `success`, and passes its final fence check. The PR is
|
|
retargeted BACK to `main` while S is paused before its POST. The successor run sees the
|
|
base-matching `failure`, short-circuits, and posts nothing. S resumes and posts its stale
|
|
`success`, which predates its own high-water mark — so the post-write check cannot see it — and
|
|
NO EVENT REMAINS to reclassify.
|
|
|
|
`moves:0,0,1` is that arrangement: quiet through classification and the pre-POST fence, moved by
|
|
the time the write has landed.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="moves:0,0,1")
|
|
assert r.returncode == 0, r.stderr
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
"a retarget after the POST left the exemption standing over a diff it no longer describes, "
|
|
f"with no event left to correct it. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success"
|
|
assert seq[1]["state"] == "pending" and seq[1]["description"] == UNVERIFIED_DESC, (
|
|
f"expected the exemption to be replaced by the reconcilable sentinel; got {seq}"
|
|
)
|
|
assert "retargeted, or its retarget count became unreadable, AFTER" in (r.stdout + r.stderr), (
|
|
f"the replacement happened without naming the post-write retarget:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_an_UNREADABLE_retarget_count_AFTER_the_POST_also_replaces_the_exemption(tmp_path):
|
|
""" "I cannot tell whether the base moved" must not resolve to leaving a green.
|
|
|
|
Reaching this point with a `success` means both earlier counts were TRUSTED — the pre-POST fence
|
|
refuses an exemption otherwise — so a third read that cannot be trusted is a fresh failure, not
|
|
the same one seen twice. The cost is bounded by the sentinel being reconcilable: the next run
|
|
that can read the history restores the exemption without a human.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="unreadable-after-post")
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 2, (
|
|
f"an unreadable post-write retarget count left the exemption green. Posts: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
assert seq[0]["state"] == "success"
|
|
assert seq[1]["description"] == UNVERIFIED_DESC, f"got {seq}"
|
|
|
|
|
|
def test_positive_control_a_QUIET_timeline_after_the_POST_leaves_the_exemption_alone(tmp_path):
|
|
"""The re-check must not fire on the ordinary path, or every exempt PR loses its exemption.
|
|
|
|
Stated as its own test rather than inferred from the other exemption tests passing: those would
|
|
also pass if the re-check ran and found nothing, and would NOT distinguish that from the re-check
|
|
being skipped. The single POST is what says it ran and stayed quiet.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="stable:2")
|
|
seq = _posted_sequence(tmp_path)
|
|
assert len(seq) == 1 and seq[0]["state"] == "success", (
|
|
f"a quiet timeline cost the PR its exemption: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
# THE SINGLE POST DOES NOT PROVE THE RECOUNT RAN — a job that skipped it entirely produces the
|
|
# identical result, so the assertion above is satisfied by the very regression it is meant to
|
|
# exclude. The stub counts REAL PAGES SERVED, one per walk at `STUB_TIMELINE_PAGES=1`, so three
|
|
# is the exempt path: before classifying, at the pre-POST fence, and after the POST.
|
|
walks = int((tmp_path / "timeline_reads.txt").read_text())
|
|
assert walks == 3, (
|
|
f"expected three timeline walks on the exemption path (before, pre-POST fence, post-POST "
|
|
f"re-check); the stub served {walks}, so the post-POST re-check did not run and the "
|
|
f"assertion above proves nothing about it"
|
|
)
|
|
|
|
|
|
def test_MUTATION_deleting_the_POST_POST_retarget_check_leaves_the_stale_green(tmp_path):
|
|
"""The predecessor restored: `main` had no post-POST re-check at all.
|
|
|
|
Disarming the re-check's own condition is the precise mutation — the walk still happens, so this
|
|
isolates the DECISION rather than the round-trip.
|
|
"""
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="moves:0,0,1",
|
|
mutate=(
|
|
' if [ "$rt_ok" != yes ] || [ "$rt_count" -ne "$retargets_before" ]; then',
|
|
" if false; then",
|
|
),
|
|
)
|
|
seq = _posted_sequence(tmp_path / "mutant")
|
|
assert len(seq) == 1 and seq[0]["state"] == "success", (
|
|
"the mutant did not leave the stale green, so the fixture reaches the replacement by some "
|
|
f"route other than the post-POST re-check: {seq}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_declining_to_replace_an_unreadable_combined_read_leaves_the_forged_green(tmp_path):
|
|
"""Route 5's predecessor restored: `exit 1` with nothing posted.
|
|
|
|
The behavioural half is asserted by the transport-failure and garbage tests above. This is what
|
|
binds those to the shipped clause: with the POST removed the job is red and silent, which is
|
|
exactly `main`'s behaviour and exactly what leaves an off-list `success` standing.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="transport-error",
|
|
mutate=(
|
|
' if repair_status_to "$UNVERIFIED_DESC"; then',
|
|
" if false; then",
|
|
),
|
|
)
|
|
assert r.returncode != 0
|
|
assert posted is None, (
|
|
"the mutant still posted, so the replacement does not go through `repair_status_to` and the "
|
|
f"route-5 tests are not bound to it: {posted}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_a_page_2_refusal_that_only_EXITS_leaves_the_head_unmarked(tmp_path):
|
|
"""The page-2 completeness probe refuses AND replaces (ersatztv#849).
|
|
|
|
It was excluded from the replacement on the reasoning that the probe fires when NO row for this
|
|
context was on page 1, so there is no green of any provenance to leave standing. That is
|
|
self-contradictory: the only reason page 2 is read is that the row MAY be beyond page 1, which
|
|
the probe's own message says.
|
|
|
|
`twopage` is a head whose status list runs past one page with no `h10` on either — the shape that
|
|
makes "no verdict exists" unestablishable.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="twopage")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a page-2 refusal left the head unmarked: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="twopage",
|
|
mutate=(
|
|
'replace_unknown_and_die "${CONTEXT} was not on page 1 of the statuses for ${SHA:0:7},'
|
|
" but page 2 carries ${more_len} more row(s) — the list is longer than one page, so a"
|
|
' verdict of ANY provenance may be sitting beyond it where this job cannot read it." ;;',
|
|
"exit 1 ;;",
|
|
),
|
|
)
|
|
assert mutant is None, (
|
|
"the mutant still posted, so this fixture does not reach the replacement through the page-2 "
|
|
f"refusal and the assertion above proves nothing about it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_an_untrusted_fence_that_only_ABSTAINS_leaves_the_declined_row_current(tmp_path):
|
|
"""The untrusted-fence branch writes rather than abstains (ersatztv#849).
|
|
|
|
It is reached only AFTER the classification declined to inherit whatever `h10` the head carries —
|
|
that is why it is re-deriving — so posting nothing leaves the declined row current, and no
|
|
retarget or push need have occurred, so no successor run is guaranteed. Its message used to say
|
|
the context "stays absent", which is true only of a head that had none.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), timeline_mode="unreadable")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"an untrusted fence left the head unmarked: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="unreadable",
|
|
mutate=(
|
|
'replace_unknown_state "Could not establish a trusted retarget/push count for PR #${PR}'
|
|
" (before=${retargets_before_ok}, after=${rt_ok}), so an exemption 'success' cannot be"
|
|
" shown to have been computed against the PR's current base, nor at a single head. NOTE a"
|
|
" later run only helps if the cause was transient — a PR whose timeline exceeds the page"
|
|
' cap will fail this way on every run, and needs a human verdict."',
|
|
":",
|
|
),
|
|
)
|
|
assert mutant is None, (
|
|
f"the mutant still posted, so the replacement does not come from that branch: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_ROW_ID_from_the_mid_run_comparison_overwrites_another_runs_sentinel(
|
|
tmp_path,
|
|
):
|
|
"""Two sentinels are byte-identical by design, so only the row id can tell them apart.
|
|
|
|
This run reconciles a pre-existing sentinel away — which is the case the guard must NOT fire on,
|
|
and the reason it cannot simply abstain whenever a sentinel is present — and then classifies
|
|
docs-only. Between the two reads another run replaces that sentinel with its own. The
|
|
description is unchanged, so the state/creator/description triple sees nothing; the id moved.
|
|
|
|
Without the id clause the run posts its exemption over a sentinel another run had just written,
|
|
which is a marker that something on this head is unchecked being replaced by a status a later run
|
|
re-derives.
|
|
"""
|
|
# THE SEEDED SENTINEL CARRIES THE ID THE COMBINED READ REPORTS AT THE FIRST READ (100), because
|
|
# the reconciliation witness now matches that id rather than the description. A seed with an
|
|
# unrelated id would make this run carry the sentinel forward instead of reconciling it, and the
|
|
# guard under test would never be reached.
|
|
seed = [
|
|
{"id": 3900, "context": "ci/other", "status": "success", "creator": None, "description": "unrelated"},
|
|
_sentinel_row(100),
|
|
]
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-replaced-mid-run",
|
|
history_extra=seed,
|
|
)
|
|
assert posted is None, (
|
|
f"a sentinel written by another run mid-classification was overwritten: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "written on" in (r.stdout + r.stderr), f"abstained, but silently:\n{r.stdout[-900:]}"
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-replaced-mid-run",
|
|
history_extra=seed,
|
|
mutate=(
|
|
'if [ -n "$ex_id" ] && [ -n "$pre_id" ] && [ "$ex_id" != "$pre_id" ]; then',
|
|
"if false; then",
|
|
),
|
|
)
|
|
assert mutant is not None and mutant["state"] == "success", (
|
|
"the mutant did not overwrite the sentinel, so the id clause is not what stops it and the "
|
|
f"assertion above proves nothing about it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_removing_the_repair_FLOOR_downgrades_the_repair_sentinel(tmp_path):
|
|
"""The repair may never write a description weaker than the one this run decided.
|
|
|
|
Widening the post-write gate to every write means the block now also runs after a carry-forward
|
|
write of `$REPAIR_DESC`. One transient post-write read then rewrote that head with the strictly
|
|
weaker, machine-clearable sentinel — reversing the ordering the classification chain states, and
|
|
depending on a later reconciliation to put it back.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
history_mode="postwrite-page1-error",
|
|
)
|
|
seq = _posted_sequence(tmp_path / "fixed")
|
|
assert seq and seq[-1]["description"] == REPAIR_DESC, (
|
|
f"the repair sentinel was downgraded on an unreadable post-write read: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
history_mode="postwrite-page1-error",
|
|
mutate=(' if [ "$desc" = "$REPAIR_DESC" ]; then repair_desc="$REPAIR_DESC"; fi', " :"),
|
|
)
|
|
mseq = _posted_sequence(tmp_path / "mutant")
|
|
assert mseq and mseq[-1]["description"] == UNVERIFIED_DESC, (
|
|
"the mutant did not downgrade, so the floor is not what preserves the repair sentinel and "
|
|
f"the assertion above proves nothing about it: {mseq}"
|
|
)
|
|
|
|
|
|
def test_an_OBSERVED_retarget_MARKS_a_row_this_run_declined(tmp_path):
|
|
"""Abstaining is a handoff only when there is nothing to hand off (ersatztv#849).
|
|
|
|
The arm is right not to post its CLASSIFICATION — computed against a base the PR may no longer
|
|
target — but when the head already carries a row this run DECLINED to inherit, posting nothing
|
|
leaves that row authoritative for the whole window until the successor finishes. And in the case
|
|
the head-arm's own message names, a PR's FIRST push, no successor is queued at all.
|
|
|
|
Here `mallory` is off `$H10_REVIEWERS`, so the existing `success` is declined rather than
|
|
inherited, and `moves:0,1` retargets the PR mid-classification.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
timeline_mode="moves:0,1",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
"a declined `success` was left authoritative while this run abstained on an observed "
|
|
f"retarget: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "was retargeted while this job was classifying" in (r.stdout + r.stderr), (
|
|
f"marked, but not by the retarget arm:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_an_OBSERVED_retarget_on_an_UNMARKED_head_still_posts_nothing(tmp_path):
|
|
"""The scoping, asserted rather than assumed.
|
|
|
|
On a head that carries nothing there is nothing to leave standing, so the arm must stay silent —
|
|
a write there would be noise on the commonest path in this job, and it is also what
|
|
`test_a_RETARGET_DURING_the_run_posts_NOTHING` above depends on.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), timeline_mode="moves:0,1")
|
|
assert posted is None, f"an empty head was marked on an observed retarget: {posted}"
|
|
|
|
|
|
def test_MUTATION_not_marking_the_declined_row_leaves_it_authoritative(tmp_path):
|
|
"""Disarming the scope test, which is the whole decision this helper makes."""
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
timeline_mode="moves:0,1",
|
|
mutate=('if [ -z "$ex_state" ]; then return 0; fi', "return 0"),
|
|
)
|
|
seq = _posted_sequence(tmp_path / "mutant")
|
|
assert seq == [], (
|
|
"the mutant still marked the head, so the scope test is not what produces the write and the "
|
|
f"test above proves nothing about it: {seq}"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_element_BESIDE_a_readable_row_still_refuses(tmp_path):
|
|
"""An element whose `.context` cannot be read cannot be shown to be a DIFFERENT context.
|
|
|
|
Dropping malformed neighbours once a target row was found reasons that a bad row beside a good
|
|
one is noise. It is not, and the rule is unconditional: the unreadable element may be a
|
|
mangled rendering of this head's own rejection, and the one-row-per-context invariant that would
|
|
rule that out is exactly what a schema-corrupt response has already broken.
|
|
|
|
This fixture reads as a POSITIVE control — a scalar beside an off-list `success` — and is the
|
|
failing case: the branch re-derived the off-list row and POSTed an
|
|
exemption where `origin/main` errored on the scalar and posted nothing. The assertion is
|
|
therefore inverted, not adjusted.
|
|
|
|
The cost is a stall on any head carrying a malformed element. That is the correct direction for a
|
|
required check: it withholds a green rather than granting one.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="malformed-row-beside-verdict")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a malformed element beside a readable row was ignored and the head judged: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode != 0, "an unreadable element must still fail the job"
|
|
|
|
|
|
def test_MUTATION_ignoring_a_malformed_element_when_a_row_was_found_greens_the_head(tmp_path):
|
|
"""The narrower scoping, restored as a mutant: refuse only when NO readable row was found."""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="malformed-row-beside-verdict",
|
|
mutate=(
|
|
'if [ "$unreadable" -gt 0 ]; then',
|
|
'if [ "$(printf \'%s\' "$row" | jq -r \'.context // ""\')" = "" ] && [ "$unreadable" -gt 0 ]; then',
|
|
),
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"the mutant did not green the head, so the unconditional refusal is not what prevents it: "
|
|
f"{posted}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_SUCCESS_wearing_the_repair_sentinel_text_is_not_treated_as_a_sentinel(tmp_path):
|
|
"""A sentinel is a `pending` STATE wearing that description, never the text alone.
|
|
|
|
Both flags were set from the description by itself — the same mistake the removed "this job's own
|
|
output" exclusion was removed FOR. A description is not provenance, and it is not state either.
|
|
|
|
Trace without the state test: a machine or off-list `success` carries `$REPAIR_DESC` verbatim,
|
|
`ex_repair` is set from the text, the mark's already-there test matches it, and the arm returns
|
|
without POSTing — leaving a green on an unreviewed head, on a first-push event with no successor
|
|
guaranteed.
|
|
"""
|
|
fixture = dict(
|
|
status_mode="existing:success",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
push_mode="moves:0,1",
|
|
)
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), **fixture)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
"a `success` wearing the sentinel's text was taken for a sentinel, so the mark claimed a "
|
|
f"human verdict was lost when nothing established one: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
**fixture,
|
|
mutate=(
|
|
'"$REPAIR_DESC"*) if [ "$ex_state" = pending ]; then ex_repair=yes; fi ;;',
|
|
'"$REPAIR_DESC"*) ex_repair=yes ;;',
|
|
),
|
|
)
|
|
assert mutant is not None and mutant["description"] == REPAIR_DESC, (
|
|
"the mutant did not mistake the impersonating row for a sentinel, so the flag's state test "
|
|
f"is not what stops it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_SUCCESS_wearing_the_sentinel_text_ARRIVING_MID_RUN_does_not_stop_the_run(tmp_path):
|
|
"""The same impersonation at the second read, where it exits the job entirely.
|
|
|
|
The mid-run guard treats a repair sentinel appearing between the reads as another run's repair and
|
|
abstains. A `success` wearing that text would therefore stop this run from writing anything, and
|
|
the green it was impersonating a sentinel with stays current.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
midrun_row=f"|success|{REPAIR_DESC}",
|
|
)
|
|
assert posted is not None, (
|
|
f"a `success` wearing the sentinel's text stopped the run from writing:\n{r.stdout[-900:]}"
|
|
)
|
|
assert posted["state"] == "success" and posted["description"].startswith("Exempt:"), (
|
|
f"expected the impersonating row to be re-derived, got {posted}"
|
|
)
|
|
|
|
|
|
def test_an_observed_retarget_does_NOT_downgrade_a_head_carrying_the_repair_sentinel(tmp_path):
|
|
"""The mark carries the strongest fact any snapshot shows, and declines a no-op write.
|
|
|
|
A head carrying `$REPAIR_DESC` is human-clearable only; the mark's default is the reconcilable
|
|
sentinel. Promoting on any snapshot that shows the repair fact, then refusing to write what is
|
|
already there, is what keeps an abstaining arm from weakening it — and the two halves are one
|
|
mechanism, so this pins them together.
|
|
"""
|
|
fixture = dict(
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
timeline_mode="moves:0,1",
|
|
)
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), **fixture)
|
|
assert posted is None, (
|
|
f"an abstaining arm rewrote a head that already carried the repair sentinel: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
**fixture,
|
|
mutate=('if [ "$desc" = "$REPAIR_DESC" ]; then mark_desc="$REPAIR_DESC"; fi', ":"),
|
|
)
|
|
assert mutant is not None and mutant["description"] == UNVERIFIED_DESC, (
|
|
"the mutant did not downgrade, so the promotion is not what protects the repair sentinel "
|
|
f"and the assertion above proves nothing about it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_FOREIGN_status_cannot_spoof_its_way_out_of_being_marked(tmp_path):
|
|
"""A description is not provenance, so there is no "this job's own output" exclusion.
|
|
|
|
The tempting exclusion is a prefix test on the generic `Awaiting review verdict …` text, to keep
|
|
the mark off the commonest head. It would be spoofable by exactly the writers it must not trust:
|
|
any workflow with `code: write` can POST a `creator: null` row and any repository writer can POST
|
|
one with a creator, either choosing that description. Masking a human `failure` with a lookalike
|
|
`pending` would then buy an abstention, and the successor would re-derive it as ordinary machine
|
|
output with the rejection below its own high-water mark.
|
|
|
|
So a row wearing this job's own description is marked like any other.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator="mallory",
|
|
status_desc="Awaiting review verdict for a9e3e23",
|
|
push_mode="moves:0,1",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a foreign row wearing this job's description escaped the mark: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_an_observed_PUSH_also_marks_a_row_this_run_did_not_inherit(tmp_path):
|
|
"""The HEAD arm, not just the base arm.
|
|
|
|
Two call sites of one helper, and only the retarget one had a fixture — the shape where a fix
|
|
lands on one caller and the other keeps the old behaviour unobserved. The head arm is the one
|
|
whose own message names the case with NO successor at all: a PR's first push.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
push_mode="moves:0,1",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"an off-list `success` was left authoritative while the head arm abstained: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert "head branch was pushed while this job was classifying" in (r.stdout + r.stderr), (
|
|
f"marked, but not by the head arm:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_HEAD_arms_mark_leaves_the_row_authoritative(tmp_path):
|
|
"""Disarms the head arm's call site ALONE, so the base arm's fixture cannot cover for it."""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
push_mode="moves:0,1",
|
|
mutate=(
|
|
'mark_declined_row_if_any "PR #${PR}\'s head branch was pushed while this job was classifying." || exit 1',
|
|
":",
|
|
),
|
|
)
|
|
assert posted is None, (
|
|
"the mutant still marked, so the head arm's own call site is not what does it and the test "
|
|
f"above proves nothing about it: {posted}"
|
|
)
|
|
|
|
|
|
def test_a_FAILED_mark_on_an_arm_FAILS_the_job(tmp_path):
|
|
"""The helper's result propagates from the ARM call sites too, not only the fence branch."""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
timeline_mode="moves:0,1",
|
|
post_fails=True,
|
|
)
|
|
assert posted is None, "the stub was supposed to reject every POST"
|
|
assert r.returncode != 0, (
|
|
"the arm reported a clean abstention while the row it meant to mark is still authoritative "
|
|
f"and every POST had failed:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_swallowing_the_marks_result_reports_a_clean_abstention(tmp_path):
|
|
"""The predecessor: the helper returning 0 whatever the write did."""
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="mallory",
|
|
timeline_mode="moves:0,1",
|
|
post_fails=True,
|
|
mutate=(
|
|
'if ! repair_status_to "$mark_desc"; then',
|
|
"if false; then",
|
|
),
|
|
)
|
|
assert rm.returncode == 0, (
|
|
"the mutant still failed the job, so the helper's `|| return 1` is not what propagates a "
|
|
f"failed write: rc={rm.returncode}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_reconciliation_witness_needs_the_CURRENT_row_not_an_OLDER_identical_one(tmp_path):
|
|
"""Two sentinels are byte-identical, so "a row with this description" is the wrong question.
|
|
|
|
The head's CURRENT sentinel has id 100; the history carries only an OLDER identical one at 999,
|
|
and the verdict buried under the current sentinel is not in this (stale) read either. Matching by
|
|
description is satisfied, the sentinel clears, and the run exempts a head whose write was never
|
|
verified. Matching the id this read reported is not.
|
|
"""
|
|
seed = [_sentinel_row(999)]
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-with-id:100",
|
|
history_extra=seed,
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"an older identical sentinel satisfied the witness: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-with-id:100",
|
|
history_extra=seed,
|
|
mutate=('if [ -n "$ex_id" ]; then', "if false; then"),
|
|
)
|
|
assert mutant is not None and mutant["state"] == "success", (
|
|
"the mutant did not clear the sentinel, so the id branch is not what refuses an older "
|
|
f"identical row and the assertion above proves nothing about it: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_STRING_id_on_the_current_row_takes_the_SCHEMA_FAULT_route(tmp_path):
|
|
"""The fault route is swept by FIELD, not written once for the field that produced it.
|
|
|
|
`.id` is consumed by the reconciliation witness, and a string where the schema says number is the
|
|
same class of unreadable as the corrupt `.creator` above. Feeding it to `--argjson` is a jq parse
|
|
error, which would make the witness unusable and carry the sentinel forward for a reason nobody
|
|
can see; routing it to the fault path says so instead.
|
|
|
|
The DESCRIPTION fallback in the witness is still live and still the documented behaviour — it is
|
|
reached when `.id` is legitimately ABSENT, which is every `existing:` fixture in this file, and
|
|
the id branch itself is pinned by the mutation in the test above.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode="sentinel-with-id:str-77")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a string id did not take the fault route: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode != 0, "an unreadable row must still fail the job"
|
|
assert "carries a type the schema does not allow" in (r.stdout + r.stderr), (
|
|
f"the sentinel was written, but not by the schema-fault route:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize("field", ["status", "creator", "description", "id"])
|
|
def test_EVERY_consumed_field_of_the_existing_row_takes_the_SCHEMA_FAULT_route(tmp_path, field):
|
|
"""Swept by FIELD, not written once for the field that produced it.
|
|
|
|
The `.creator` case is the one that was measured, and a route written for it alone leaves the
|
|
other three to the reader's assumption — which is the per-field gap this repo keeps re-learning.
|
|
Each of the four is read by a decision: `.creator` and `.description` by the provenance test,
|
|
`.status` by both short-circuits, `.id` by the reconciliation witness.
|
|
"""
|
|
posted, r = _run_classify(tmp_path, _emitting("docs/a.md"), status_mode=f"malformed-field:{field}")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a corrupt .{field} did not take the fault route: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode != 0, "an unreadable row must still fail the job"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"field,shipped,mutated",
|
|
[
|
|
("status", '(.status | type) == "null" then "" else $f end', 'true then "" else "" end'),
|
|
(
|
|
"description",
|
|
'(.description | type) == "null" then "" else $f end',
|
|
'true then "" else "" end',
|
|
),
|
|
],
|
|
)
|
|
def test_MUTATION_a_malformed_field_read_as_ABSENT_stops_reaching_the_fault_route(tmp_path, field, shipped, mutated):
|
|
"""The two fields whose type test had no proof of its own.
|
|
|
|
`.creator`'s is proved separately (it is the one whose failure greens a rejection outright). For
|
|
these two the mutant's damage is quieter and still real: the row reads as stateless or
|
|
descriptionless, so the provenance tests see nothing to inherit and the run re-derives a head it
|
|
cannot actually read.
|
|
"""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode=f"malformed-field:{field}",
|
|
mutate=(f"elif {shipped}", f"elif {mutated}"),
|
|
)
|
|
assert posted is None or posted["description"] != UNVERIFIED_DESC, (
|
|
f"the mutant still took the fault route for .{field}, so its own type test is not what sends "
|
|
f"it there: {posted}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_description_in_the_history_does_not_lose_the_verdict_beside_it(tmp_path):
|
|
"""The reconciliation's `buried` filter, twin of the post-write one that already had a proof.
|
|
|
|
`(.description | type) == "string"` is what keeps a numeric description from hard-erroring
|
|
`startswith`. Without it the whole count comes back unusable, the sentinel is carried forward,
|
|
and the genuine verdict on the next row — the one reconciliation exists to find — is never
|
|
counted, so the head never gets its human-only marker.
|
|
"""
|
|
seed = [
|
|
_sentinel_row(),
|
|
{
|
|
"id": 4100,
|
|
"context": "review-verdict/h10",
|
|
"status": "failure",
|
|
"creator": {"login": "timothy"},
|
|
"description": 7,
|
|
},
|
|
{
|
|
"id": 4200,
|
|
"context": "review-verdict/h10",
|
|
"status": "failure",
|
|
"creator": {"login": "timothy"},
|
|
"description": "Review-verdict: BLOCKED @ a9e3e23 (base: main)",
|
|
},
|
|
]
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
history_extra=seed,
|
|
)
|
|
assert posted is not None and posted["description"] == REPAIR_DESC, (
|
|
f"the verdict beside a malformed row was lost with it: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
history_extra=seed,
|
|
mutate=(
|
|
'| select(((.description | type) == "string")\n'
|
|
' and (.description | startswith("Review-verdict:")))] | length\') || buried=""',
|
|
'| select((.description // "") | startswith("Review-verdict:"))] | length\') || buried=""',
|
|
),
|
|
)
|
|
assert mutant is not None and mutant["description"] != REPAIR_DESC, (
|
|
"the mutant still upgraded, so the description type test is not what keeps the verdict "
|
|
f"countable: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_combined_read_RETRIES_a_transient_failure(tmp_path):
|
|
"""A retry that never retries is a silent degradation, and this one is load-bearing.
|
|
|
|
It is what keeps the route-5 replacement attached to a PERSISTENT failure: without it a single
|
|
blip costs a head whatever verdict it carries, which is the trade the comment there explicitly
|
|
says the retry buys down. Against a stub that fails EVERY attempt a retrying read and a one-shot
|
|
read are indistinguishable, which is how it shipped unexercised.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="flaky-combined")
|
|
assert posted is not None and posted["state"] == "success", (
|
|
f"a transient combined-read failure was not retried: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="flaky-combined",
|
|
mutate=(" for try in 1 2; do\n json=$(gh", " for try in 1; do\n json=$(gh"),
|
|
)
|
|
assert mutant is not None and mutant["description"] == UNVERIFIED_DESC, (
|
|
"the mutant did not take the replacement path, so the retry is not what absorbs the blip: "
|
|
f"{mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_the_sentinel_write_RETRIES_a_transient_POST_failure(tmp_path):
|
|
"""`repair_status_to`'s second attempt, on the one write whose failure leaves the gate unmarked."""
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="unreadable",
|
|
post_fails="first",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a transient POST failure lost the sentinel write: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="unreadable",
|
|
post_fails="first",
|
|
# The SECOND attempt only. The block is byte-identical to the first, so the clause is
|
|
# that second copy TOGETHER WITH the `return 1` after it, which is what makes it unique.
|
|
mutate=(
|
|
" if gh -X POST -H 'Content-Type: application/json' -d \"$body\" \\\n"
|
|
' "$BASE_URL/repos/$REPO/statuses/$SHA" >/dev/null 2>&1; then\n'
|
|
" return 0\n"
|
|
" fi\n"
|
|
" return 1",
|
|
" return 1",
|
|
),
|
|
)
|
|
assert mutant is None, (
|
|
"the mutant still wrote the sentinel, so the second attempt is not what absorbs the blip: "
|
|
f"{mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_run_whose_OWN_write_is_the_sentinel_does_not_abstain_on_it(tmp_path):
|
|
"""The mid-run sentinel guard exempts a run that is itself writing a sentinel.
|
|
|
|
Without that exemption the run abstains on a row it was about to replace with an equivalent one,
|
|
posts nothing, and the head keeps whichever sentinel got there first — which is not wrong in
|
|
state but IS the deadlock shape: replacing a sentinel with a sentinel loses nothing, and refusing
|
|
to is how a fixed point stops converging.
|
|
|
|
The fixture is the case that needs it: the head's sentinel is replaced mid-run (ids 100 -> 200,
|
|
byte-identical text), and the seed does NOT name id 100, so reconciliation cannot clear it and
|
|
this run's own write is the sentinel too.
|
|
"""
|
|
seed = [_sentinel_row(4321)]
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-replaced-mid-run",
|
|
history_extra=seed,
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"the run abstained on a row its own write would have matched: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="sentinel-replaced-mid-run",
|
|
history_extra=seed,
|
|
mutate=(
|
|
' && [ "$desc" != "$REPAIR_DESC" ] && [ "$desc" != "$UNVERIFIED_DESC" ]; then',
|
|
" ; then",
|
|
),
|
|
)
|
|
assert mutant is None, (
|
|
"the mutant did not abstain, so the sentinel exemptions on that guard are not what lets a "
|
|
f"sentinel-writing run through: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_FAILED_repair_write_FAILS_the_job(tmp_path):
|
|
"""The repair is the write whose failure leaves an unverified status standing.
|
|
|
|
`post_fails="after-first"` is the only arrangement that reaches it: with every POST failing the
|
|
job dies on its own classification write and never gets here, so a job that ignores the repair's
|
|
result and one that acts on it look identical.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="human-after-post",
|
|
post_fails="after-first",
|
|
)
|
|
assert r.returncode != 0, f"a repair that never landed reported a clean run:\n{r.stdout[-900:]}"
|
|
assert "COULD NOT REPAIR" in (r.stdout + r.stderr), (
|
|
f"the job went red, but not because the repair failed:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="human-after-post",
|
|
post_fails="after-first",
|
|
mutate=(' if ! repair_status_to "$repair_desc"; then', " if false; then"),
|
|
)
|
|
assert rm.returncode == 0, (
|
|
"the mutant still failed the job, so the repair's result is not what reddens it: "
|
|
f"rc={rm.returncode}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_a_repair_that_leaves_state_at_success_re_enters_the_post_POST_check(tmp_path):
|
|
"""`state=pending` after the repair is what keeps the post-POST check off a repaired head.
|
|
|
|
Without it the block still sees `success`, walks the timeline a third time, and — on a retarget
|
|
it then observes — replaces the REPAIR sentinel with the weaker reconcilable one. That is the
|
|
same ordering inversion the floor beside it exists to prevent, reached by a different route.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
history_mode="human-after-post",
|
|
timeline_mode="moves:0,0,1",
|
|
)
|
|
seq = _posted_sequence(tmp_path / "fixed")
|
|
assert len(seq) == 2 and seq[-1]["description"] == REPAIR_DESC, (
|
|
f"expected the exemption then the repair, and nothing after it: {seq}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
history_mode="human-after-post",
|
|
timeline_mode="moves:0,0,1",
|
|
mutate=(
|
|
' echo "Repaired ${CONTEXT} to pending on ${SHA:0:7} (${repair_desc})."\n state=pending',
|
|
' echo "Repaired ${CONTEXT} to pending on ${SHA:0:7} (${repair_desc})."',
|
|
),
|
|
)
|
|
mseq = _posted_sequence(tmp_path / "mutant")
|
|
assert len(mseq) == 3 and mseq[-1]["description"] == UNVERIFIED_DESC, (
|
|
"the mutant did not re-enter the post-POST check, so the state update is not what keeps it "
|
|
f"out and the assertion above proves nothing about it: {mseq}"
|
|
)
|
|
|
|
|
|
def test_a_FAILED_post_POST_replacement_FAILS_the_job(tmp_path):
|
|
"""The post-POST replacement is the write that takes back a green already published."""
|
|
posted, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="moves:0,0,1",
|
|
post_fails="after-first",
|
|
)
|
|
assert r.returncode != 0, f"a stale exemption was left standing and the job reported clean:\n{r.stdout[-900:]}"
|
|
assert "COULD NOT REPLACE" in (r.stdout + r.stderr), (
|
|
f"the job went red, but not because the replacement failed:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="moves:0,0,1",
|
|
post_fails="after-first",
|
|
mutate=(' if ! repair_status_to "$UNVERIFIED_DESC"; then', " if false; then"),
|
|
)
|
|
assert rm.returncode == 0, (
|
|
"the mutant still failed the job, so the replacement's result is not what reddens it: "
|
|
f"rc={rm.returncode}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_a_post_POST_replacement_that_leaves_state_at_success_stops_asking_for_a_verdict(
|
|
tmp_path,
|
|
):
|
|
"""`state=pending` after the replacement is what makes the job ask for a verdict.
|
|
|
|
The closing notice — the line that names the command a reviewer runs — is keyed on `$state`. Left
|
|
at `success`, the head is pending and nothing on screen says so.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), timeline_mode="moves:0,0,1")
|
|
assert "needs an H10 review verdict" in (r.stdout + r.stderr), (
|
|
f"the head was replaced with the sentinel but the run never asked for a verdict:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="moves:0,0,1",
|
|
mutate=(
|
|
' echo "Replaced ${CONTEXT} with the unverified-write sentinel on ${SHA:0:7}."\n state=pending',
|
|
' echo "Replaced ${CONTEXT} with the unverified-write sentinel on ${SHA:0:7}."',
|
|
),
|
|
)
|
|
assert "needs an H10 review verdict" not in (rm.stdout + rm.stderr), (
|
|
"the mutant still asked for a verdict, so the state update is not what produces the notice"
|
|
)
|
|
|
|
|
|
def test_a_reviewers_verdict_ARRIVING_MID_RUN_is_not_buried_by_an_abstaining_arm(tmp_path):
|
|
"""The refusals must judge the row the POST replaces, not the one the run started with.
|
|
|
|
All three read `$pre_*`, the FIRST read, while the write replaces whatever is current — so a
|
|
reviewer's verdict arriving BETWEEN the two reads slipped past every one of them: the base
|
|
mismatch clears `ex_attributable` so the mid-run abstain declines, `pre_creator` is empty so the
|
|
allow-list loop declines, and the arm marks a row nobody evaluated. The static case was covered;
|
|
this is the one that moves.
|
|
|
|
Recovery is not free either: the next run's reconciliation counts that `Review-verdict:` row as
|
|
buried and upgrades to the human-only sentinel — exactly the cost the refusal exists to avoid.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
midrun_row="timothy|failure|Review-verdict: BLOCKED @ a9e3e23 (base: probe/scratch)",
|
|
push_mode="moves:0,1",
|
|
)
|
|
assert posted is None, (
|
|
f"a reviewer's verdict that landed mid-run was buried by an abstaining arm: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_deciding_the_arms_refusals_on_the_FIRST_read_buries_it(tmp_path):
|
|
"""Restores the narrower form: the allow-list veto reading only the opening snapshot."""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="appears-on-read:2",
|
|
midrun_row="timothy|failure|Review-verdict: BLOCKED @ a9e3e23 (base: probe/scratch)",
|
|
push_mode="moves:0,1",
|
|
mutate=(
|
|
'if [ "$rv" = "$ex_creator" ]; then return 0; fi',
|
|
'if [ "$rv" = "$pre_creator" ]; then return 0; fi',
|
|
),
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
"the mutant did not bury it, so consulting the CURRENT read is not what protects it and the "
|
|
f"test above proves nothing about it: {posted}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_positive_control_a_machine_SUCCESS_is_still_marked(tmp_path):
|
|
"""The scoping is on the DESCRIPTION, not on `creator: null`.
|
|
|
|
Excluding every machine-written row would exclude a `success` posted by another workflow — which
|
|
is precisely the row this marking exists for, and the reason the exclusion is not written the
|
|
obvious way.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator=None,
|
|
status_desc="Exempt: docs-only change (no code, no protected path)",
|
|
push_mode="moves:0,1",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a machine `success` was left authoritative while the arm abstained: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_UNREADABLE_status_elements_cannot_license_a_no_verdict_conclusion(tmp_path):
|
|
"""Dropping a malformed element is safe only once the target row has been FOUND.
|
|
|
|
`select(type == "object")` was added so a malformed NEIGHBOUR could not kill the step. When it
|
|
drops every element, `first // {}` yields `{}`, all `ex_*` read empty, and the job concludes NO
|
|
VERDICT EXISTS — so a docs-only PR walks straight to the exemption. `origin/main` raised jq error
|
|
5 on the scalar and `set -e` aborted before any POST, so this was an input on which the branch
|
|
posted a green that `main` did not; if the scalar is a mangled rendering of the head's human
|
|
`failure`, that rejection is greened.
|
|
|
|
The asymmetry is the rule: a malformed row beside one we DID read is noise; a malformed row where
|
|
we found nothing is the only evidence there was.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="scalar-statuses-only")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"an all-unreadable status list was read as 'no verdict exists': {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode != 0, "an unreadable list must still fail the job"
|
|
|
|
|
|
def test_MUTATION_concluding_absence_over_unreadable_elements_greens_the_head(tmp_path):
|
|
"""Disarms the absence guard alone; the type-safe filter above it is untouched."""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="scalar-statuses-only",
|
|
mutate=('if [ "$unreadable" -gt 0 ]; then', "if false; then"),
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"the mutant did not green the head, so the absence guard is not what prevents it and the "
|
|
f"test above proves nothing about it: {posted}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_FAILED_sentinel_write_on_the_fence_path_FAILS_the_job(tmp_path):
|
|
"""The write helper reports whether it wrote, and the fence caller acts on it.
|
|
|
|
Its first version ended the failure arm with a successful `echo`, so it returned 0 after BOTH
|
|
POST attempts failed and the caller's `exit 0` beside it reported an abstention that had not
|
|
happened — while whatever the head carried stayed authoritative.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), timeline_mode="unreadable", post_fails=True)
|
|
assert posted is None, "the stub was supposed to reject every POST"
|
|
assert r.returncode != 0, (
|
|
"the job reported a clean abstention while the head was left unmarked and its POSTs had all "
|
|
f"failed:\n{r.stdout[-900:]}"
|
|
)
|
|
assert "COULD NOT WRITE THE UNVERIFIED SENTINEL" in (r.stdout + r.stderr), (
|
|
f"the job went red, but not because the sentinel write failed:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_generic_PENDING_with_no_mark_also_becomes_the_sentinel(tmp_path):
|
|
"""The no-mark downgrade covers every re-derivable write, not only the exemption.
|
|
|
|
The damaging PR is one that IS exemptible and got the generic `pending` only from a transient
|
|
enumeration failure. Its description carries no marker, the post-write check does not run without
|
|
a mark, so a verdict landing in the write window is buried and the NEXT run re-derives that
|
|
`pending` into the exemption with the human row below its own mark.
|
|
|
|
The fixture is that PR: the enumerator exits 1 (generic `pending`) AND the status history cannot
|
|
be read (no mark).
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="premark-page1-error",
|
|
)
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
"a generic `pending` nothing could verify was posted with its re-derivable description "
|
|
f"intact: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_a_MALFORMED_creator_FIELD_on_the_existing_row_is_unknown_state_not_an_absent_one(tmp_path):
|
|
"""A wrong TYPE is not an absent value, and reading it as one is a licence to re-derive.
|
|
|
|
Type-testing the four consumed fields and resolving a failure to `""` means, for `.creator`,
|
|
"no creator", i.e. unattributable, i.e. re-derive — so a head carrying a human `failure`
|
|
with a corrupt creator is greened. `origin/main` died on `.creator.login` BEFORE writing
|
|
anything, which is fail-closed, so this was a direction regression rather than a residual.
|
|
|
|
The rationale that produced it came from #763, whose site is the POST-WRITE filter: there, dying
|
|
leaves a green already published, so dropping the row is the safe direction. Here the alternative
|
|
is dying before any write. The deferral rationale did not transfer.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="malformed-creator-field")
|
|
assert posted is not None and posted["description"] == UNVERIFIED_DESC, (
|
|
f"a corrupt creator on a human rejection produced {posted}, not the sentinel\n{r.stdout[-900:]}"
|
|
)
|
|
assert r.returncode != 0, "an unreadable row must still fail the job"
|
|
|
|
|
|
def test_an_ID_that_appears_on_only_ONE_read_is_not_a_replacement(tmp_path):
|
|
"""One response omitting `id` beside one that includes it is not evidence of a mid-run write.
|
|
|
|
The row, its state, its creator and its description are all unchanged; only the SERVER's
|
|
reporting differs. Treating that as a replacement makes the run abstain — leaving current a row
|
|
the classification had already declined to inherit, which is the direction that costs something.
|
|
"""
|
|
posted, r = _run_classify(tmp_path / "fixed", _emitting("docs/a.md"), status_mode="id-appears-on-second-read")
|
|
assert posted is not None, f"an asymmetric id report was read as a mid-run replacement:\n{r.stdout[-900:]}"
|
|
assert posted["state"] == "success" and posted["description"].startswith("Exempt:"), (
|
|
f"expected the off-list row to be re-derived into the docs-only exemption; got {posted}"
|
|
)
|
|
|
|
|
|
def test_an_observed_retarget_does_NOT_bury_an_ALLOWLISTED_reviewers_verdict(tmp_path):
|
|
""" "Declined" is decided against THIS event's base, so it is not a judgement about the row.
|
|
|
|
A genuine verdict recorded for another base is declined here and is still the right answer for
|
|
the base it names — the successor run for that base short-circuits on it. Burying it costs a
|
|
manual re-post on an ordinary retarget-onto-the-reviewed-base flow. An OFF-list row is what the
|
|
marking exists for and is still marked, which the sibling test above asserts.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:success",
|
|
status_creator="timothy",
|
|
status_desc="Review-verdict: MERGEABLE @ a9e3e23 (base: probe/scratch)",
|
|
timeline_mode="moves:0,1",
|
|
)
|
|
assert posted is None, (
|
|
f"a reviewer's verdict for another base was buried by an abstaining run: {posted}\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_an_observed_retarget_does_NOT_bury_the_repair_sentinel(tmp_path):
|
|
"""The arms may mark, but never with a weaker description than the head already carries.
|
|
|
|
`$REPAIR_DESC` is human-clearable only; `replace_unknown_state` writes the machine-clearable one.
|
|
A head carrying the repair sentinel has a non-empty `pre_state`, so the bare scope test fired on
|
|
it and inverted the ordering the repair site's own floor exists to protect — one mechanism, three
|
|
writers, and only two had the rule.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path,
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
timeline_mode="moves:0,1",
|
|
)
|
|
assert posted is None, f"the repair sentinel was overwritten by an abstaining run: {posted}"
|
|
|
|
|
|
def test_MUTATION_comparing_ids_WITHOUT_requiring_both_makes_the_run_abstain(tmp_path):
|
|
"""Disarming the presence guards alone, leaving the inequality."""
|
|
mutant, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="id-appears-on-second-read",
|
|
mutate=(
|
|
'if [ -n "$ex_id" ] && [ -n "$pre_id" ] && [ "$ex_id" != "$pre_id" ]; then',
|
|
'if [ "$ex_id" != "$pre_id" ]; then',
|
|
),
|
|
)
|
|
assert mutant is None, (
|
|
"the mutant did not abstain, so the both-present requirement is not what keeps an asymmetric "
|
|
f"id report from reading as a replacement: {mutant}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_no_op_repair_SKIP_re_posts_what_is_already_there(tmp_path):
|
|
"""The skip, isolated: a repair that would write what this run already wrote is not a repair.
|
|
|
|
The fixture reaches it the ordinary way — a carry-forward `$REPAIR_DESC` write whose post-write
|
|
walk then fails, so the floor pins `repair_desc` to the description just POSTed.
|
|
"""
|
|
_run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
history_mode="postwrite-page1-error",
|
|
)
|
|
assert len(_posted_sequence(tmp_path / "fixed")) == 1, (
|
|
f"the shipped code re-posted: {_posted_sequence(tmp_path / 'fixed')}"
|
|
)
|
|
|
|
_run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
history_mode="postwrite-page1-error",
|
|
mutate=('if [ "$raced" -gt 0 ] && [ "$repair_desc" = "$desc" ]; then', "if false; then"),
|
|
)
|
|
mseq = _posted_sequence(tmp_path / "mutant")
|
|
assert len(mseq) == 2 and mseq[0] == mseq[1], (
|
|
"the mutant did not duplicate the row, so the skip is not what suppresses it and the "
|
|
f"assertion above proves nothing about it: {mseq}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_OWN_exclusion_on_the_UNVERIFIED_arm_reports_a_phantom_other_run(tmp_path):
|
|
"""The twin, on the arm that counts the reconcilable sentinel.
|
|
|
|
A run carrying the unverified sentinel forward POSTs it, then its own row sits above the mark. If
|
|
the arm does not exclude it, the job reports that ANOTHER run recorded an unverified write on this
|
|
head — a second run that does not exist.
|
|
"""
|
|
_, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
)
|
|
assert "another run recorded an unverified write" not in (r.stdout + r.stderr), (
|
|
f"the shipped code reported a phantom second run:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=UNVERIFIED_DESC,
|
|
mutate=(
|
|
'select((.creator == null) and ((.description // "") == $ud)\n'
|
|
' and ((.description // "") != $own))] | length\')',
|
|
'select((.creator == null) and ((.description // "") == $ud))] | length\')',
|
|
),
|
|
)
|
|
assert "another run recorded an unverified write" in (rm.stdout + rm.stderr), (
|
|
"the mutant did not report a phantom run, so the `$own` exclusion on the unverified arm is "
|
|
f"not what prevents it:\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_dropping_the_OWN_row_exclusion_reports_a_race_that_did_not_happen(tmp_path):
|
|
"""`--arg own "$desc"` keeps this job from counting the row it has just written.
|
|
|
|
The two exclusions are OUTCOME-redundant with the no-op skip above — drop one and the other
|
|
still suppresses the duplicate POST — which is exactly the shape where two guards hide each
|
|
other. What the exclusion alone decides is the REPORT, and after the skip learned to keep the
|
|
human `::error::` that report is a false alarm: a run whose own carry-forward row is counted
|
|
tells a reviewer their verdict was overwritten when nothing raced it.
|
|
|
|
So this asserts the log, not the post sequence, and that is the honest discriminator rather than
|
|
a weaker one. The head carries `$REPAIR_DESC`, the history is otherwise empty (mark 0), and this
|
|
run's own POST lands above the mark.
|
|
"""
|
|
_, r = _run_classify(
|
|
tmp_path / "fixed",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
)
|
|
# `::error::A human` and not the bare phrase "was overwritten": the classification's own REASON
|
|
# string for a carry-forward run also contains that phrase, so matching it would report the
|
|
# shipped code as failing on text it is supposed to print.
|
|
assert "::error::A human" not in (r.stdout + r.stderr), (
|
|
f"the shipped code reported a race against its own row:\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="existing:pending",
|
|
status_creator=None,
|
|
status_desc=REPAIR_DESC,
|
|
mutate=(
|
|
'or ((.creator == null) and ((.description // "") == $rd)\n'
|
|
' and ((.description // "") != $own))',
|
|
'or ((.creator == null) and ((.description // "") == $rd))',
|
|
),
|
|
)
|
|
assert "::error::A human" in (rm.stdout + rm.stderr), (
|
|
"the mutant did not report a false race, so the `$own` exclusion on the repair-sentinel arm "
|
|
f"is not what prevents it:\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_ignoring_the_write_result_reports_a_clean_abstention(tmp_path):
|
|
"""The predecessor: `replace_unknown_state` followed by an unconditional `exit 0`."""
|
|
_, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
timeline_mode="unreadable",
|
|
post_fails=True,
|
|
# MUTATES THE CALLER'S REACTION, not the `if` itself: dropping the `if` keyword leaves a
|
|
# dangling `then`/`fi` and the step dies on a syntax error, which is a red for the wrong
|
|
# reason. Turning the failure exit into a clean one is exactly the predecessor's OUTCOME and
|
|
# isolates the decision.
|
|
mutate=(
|
|
" # THE SENTINEL WRITE FAILED, so nothing marked this head and whatever it carries is still\n"
|
|
" # authoritative. Exiting 0 here would report an abstention that did not happen.\n"
|
|
" exit 1",
|
|
" exit 0",
|
|
),
|
|
)
|
|
assert rm.returncode == 0, (
|
|
"the mutant still failed the job, so the caller is not what turns a failed write into a red "
|
|
f"run and the test above proves nothing about it: rc={rm.returncode}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_reading_a_malformed_FIELD_as_absent_greens_a_rejection(tmp_path):
|
|
"""The weaker form restored: type-test, then fall back to the empty string."""
|
|
posted, rm = _run_classify(
|
|
tmp_path / "mutant",
|
|
_emitting("docs/a.md"),
|
|
status_mode="malformed-creator-field",
|
|
mutate=(
|
|
'elif (.creator | type) == "null" then "" else $f end\')',
|
|
'else "" end\')',
|
|
),
|
|
)
|
|
assert posted is not None and posted["state"] == "success", (
|
|
"the mutant did not green the rejection, so the schema-fault route is not what prevents it "
|
|
f"and the test above proves nothing about it: {posted}\n{rm.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
def test_MUTATION_restoring_the_SUCCESS_only_no_mark_downgrade_leaves_a_re_derivable_pending(tmp_path):
|
|
"""The exact predecessor from this branch's own previous commit, restored.
|
|
|
|
This is the mutation the suite was missing: the nearest proof mutated the DESCRIPTION
|
|
the downgrade writes, not its SCOPE, and its fixture ran a succeeding enumeration — so
|
|
`state=success` there and the `success`-only predicate fired identically. Nothing reached the
|
|
downgrade with `state=pending`, and the predecessor survived the whole suite.
|
|
"""
|
|
posted, r = _run_classify(
|
|
tmp_path / "mutant",
|
|
"#!/usr/bin/env bash\n" + DOCS_ONLY + "exit 1\n",
|
|
history_mode="premark-page1-error",
|
|
mutate=(
|
|
'[ "$max_id_before" -lt 0 ] && [ "$desc" != "$REPAIR_DESC" ]; then',
|
|
'[ "$state" = "success" ] && [ "$max_id_before" -lt 0 ]; then',
|
|
),
|
|
)
|
|
assert posted is not None and posted["description"] != UNVERIFIED_DESC, (
|
|
"the `success`-only predecessor still wrote the sentinel, so this fixture does not reach the "
|
|
f"downgrade with state=pending and the test above proves nothing about its scope: {posted}"
|
|
f"\n{r.stdout[-900:]}"
|
|
)
|
|
|
|
|
|
# --- Workflow token scope (ersatztv#748) -------------------------------------------------------
|
|
#
|
|
# Both assertions guard a property whose violation is SILENT and, for the first, unrecoverable.
|
|
# `permissions:` is exhaustive: dropping a unit does not degrade, it 403s the `curl -sf` under
|
|
# `set -euo pipefail`, so `review-verdict/h10` is never posted and EVERY merge to `main` blocks —
|
|
# including the PR that would repair it, with no force-merge escape since ersatztv#743. And because
|
|
# `review-verdict.yml` is base-resolved, the PR making that edit CANNOT catch it: its own run uses
|
|
# the definition already on `main`. A test on the tree is the only place this is checkable at review
|
|
# time. Measured basis for the unit list: docs/ci-cd.md -> "Workflow token scope".
|
|
|
|
|
|
def _declared_permissions(path, job=None):
|
|
import yaml
|
|
|
|
doc = yaml.safe_load(path.read_text()) or {}
|
|
if job is None:
|
|
return doc.get("permissions")
|
|
return ((doc.get("jobs") or {}).get(job) or {}).get("permissions")
|
|
|
|
|
|
def test_the_verdict_gate_job_declares_exactly_the_units_its_api_calls_need():
|
|
"""The gate job's block is load-bearing in BOTH directions, so this pins the whole mapping.
|
|
|
|
Not just `code: write`. A missing read unit is quieter but still wrong: without `issues: read`
|
|
the retarget fence cannot establish a trusted count and the exemption is withheld, and without
|
|
`pull-requests: read` the enumeration degrades to `complete=no` and every exempt PR falls to a
|
|
generic `pending`. Those are recoverable, unlike a missing `code: write`, but they silently turn
|
|
the two exemption classes off — the ersatztv#751 failure shape, arrived at from a different
|
|
direction.
|
|
|
|
Equality, not containment: an EXTRA unit is a finding too. This job holds the only repo-write in
|
|
the repo's CI, so widening it (`contents: write`, a stray `packages:`) is exactly the drift the
|
|
scoping in ersatztv#697/#748 exists to prevent, and it would land unremarked as "just adding a
|
|
permission".
|
|
|
|
WHAT THIS DOES NOT CATCH, stated because the test's name over-promises: it TRANSCRIBES the unit
|
|
set, it does not DERIVE it from the job's API calls. Add a new `gh` call to the workflow that
|
|
needs an undeclared unit and this test stays green while the gate breaks in production — the
|
|
failure this test exists for, arriving from the other side. Deriving the mapping would mean
|
|
parsing every URL out of a 1300-line shell body and knowing Gitea's endpoint-to-unit table, which
|
|
is a different and much larger change. So: when you add an API call to this job, the unit list
|
|
here is a checklist to revisit by hand, not a net that will tell you. (Same honesty as
|
|
`test_the_verdict_workflow_has_NO_expression_delimiter_in_any_run_body` above.)
|
|
"""
|
|
wf = REPO_ROOT / ".gitea" / "workflows" / "review-verdict.yml"
|
|
assert _declared_permissions(wf) is None, (
|
|
"review-verdict.yml grew a TOP-LEVEL permissions: block. Only the job-level one on "
|
|
"set-verdict-status was probed (docs/ci-cd.md -> 'Workflow token scope'); adding a "
|
|
"top-level default changes what other jobs in this file would inherit and is unmeasured."
|
|
)
|
|
assert _declared_permissions(wf, "set-verdict-status") == {
|
|
"code": "write", # POST /statuses/{sha} — Gitea has no `statuses` scope, so this IS it
|
|
"issues": "read", # GET /issues/{n}/timeline — the ersatztv#706 retarget fence
|
|
"pull-requests": "read", # scripts/pr-changed-files.sh -> /pulls/{n}, /pulls/{n}/files
|
|
}, (
|
|
"the review-verdict gate job's permissions: block no longer matches the unit set that was "
|
|
"measured against its API calls. Dropping `code: write` makes review-verdict/h10 UNWRITABLE "
|
|
"and blocks every merge with no force-merge escape (ersatztv#743), and this workflow is "
|
|
"base-resolved so the PR changing it cannot detect that. Re-run the scratch-base probe "
|
|
"(docs/ci-cd.md -> Review-verdict gate) before changing this."
|
|
)
|
|
|
|
|
|
def test_every_tracked_workflow_declares_a_permissions_block():
|
|
"""Otherwise the convention is prose only, and a NEW workflow is the case that breaks it.
|
|
|
|
A workflow added without `permissions:` inherits the owner-level Actions default, which is
|
|
`permissive` today — i.e. a full read/write repository token, which is status-capable and can
|
|
therefore forge `review-verdict/h10` (`ci.actions-credential-scoping`). That is silent: nothing
|
|
reddens, the new file simply holds more than it needs.
|
|
|
|
Population comes from the GIT INDEX via `_workflow_files()`, not a directory listing — same
|
|
reason as the two completeness claims above (ersatztv#806/#778).
|
|
|
|
NO EXEMPTIONS, deliberately. `ci-image.yml` briefly needed one: editing that file re-pointed
|
|
`ci-image-pin`'s `expected` at the editing commit and reddened a BLOCKING job, and its own
|
|
`paths:` made the edit publish an image. ersatztv#744 took that path out of both
|
|
(`ci.toolchain-image-publish-is-a-dispatch`), so this asserts over the whole derived population
|
|
with nothing carved out — which is the form ersatztv#835 asked for. If a future file seems to
|
|
need an exemption, that is a finding about the file, not about this test.
|
|
"""
|
|
import yaml
|
|
|
|
workflows = _workflow_files()
|
|
# Anti-vacuity. `tracked_children` matches direct children of one directory, so a renamed or
|
|
# moved `.gitea/workflows/` yields an EMPTY population and every completeness claim below passes
|
|
# having examined nothing — the #778 shape this test's own docstring invokes. `_git_ls_files`'s
|
|
# assert covers the whole index, not this subset, so it does not catch it. Measured: after
|
|
# `git mv .gitea/workflows .gitea/wf-renamed` this test passed with 0 files before this guard.
|
|
assert workflows, (
|
|
"no tracked workflows found under .gitea/workflows — this test examined NOTHING and would "
|
|
"have passed vacuously. The directory was renamed/moved, or the glob no longer matches."
|
|
)
|
|
|
|
missing = []
|
|
for wf in workflows:
|
|
doc = yaml.safe_load(wf.read_text()) or {}
|
|
jobs = (doc.get("jobs") or {}).values()
|
|
# Top-level absent is fine only if EVERY job declares its own — one undeclared job still
|
|
# inherits the owner default, so `any` would pass a file that is half-covered.
|
|
declares = doc.get("permissions") is not None or (
|
|
bool(jobs) and all((job or {}).get("permissions") is not None for job in jobs)
|
|
)
|
|
|
|
if not declares:
|
|
missing.append(wf.name)
|
|
assert not missing, (
|
|
f"these workflows declare no permissions: block at either level: {missing}. Every workflow "
|
|
"in this repo declares one (ersatztv#748), with NO exemption since #744 landed, so a new "
|
|
"file cannot silently inherit the "
|
|
"owner-level default. Read-only (`permissions: {code: read}` at top level) is the "
|
|
"convention; declare write only with a stated reason at the declaration."
|
|
)
|