Extend the dropped-step guard to docker-build.yml's required jobs, where a dropped step is fail-OPEN #756

Closed
opened 2026-08-06 21:30:32 +02:00 by timothy · 2 comments
Owner

Split out of #751, which fixed the silent-green class only in .gitea/workflows/review-verdict.yml. Raised by cold review of that fix.

The asymmetry

A step the runner declines to interpolate is dropped, and the job concludes success (#751, ci.workflow-run-body-no-expressions). What that costs depends on which required check the job writes:

Job On a dropped step Direction
review-verdict.yml / set-verdict-status review-verdict/h10 is absent → merge blocked fail-CLOSED
docker-build.yml / test (Build & test (.NET)) required context goes green having done no work fail-OPEN
docker-build.yml / migrations (EF migration integrity) required context goes green having done no work fail-OPEN

Those three are the required contexts on main. #751 guarded the one where the failure mode is safe and left the two where it is dangerous — because that is where the live bug was, not because the others were checked and found fine.

This repo has already been bitten by a required check going green without running the suite (#684, the docs-only fast path). That was a different mechanism with the same shape and the same cost.

Currently latent, which is exactly when it is cheap

Neither test nor migrations has an expression delimiter in a run: body today (build, api-docs and format do, so the file is not delimiter-free and the class is reachable by an ordinary edit). Latent, not absent.

Proposed

Extend the start-marker pattern to the two required docker-build.yml jobs: each writes a marker as its first act, and a trailing if: always() step fails the job when it is missing. Notes from doing it in #751:

  • The guard body must be expression-free, or the mechanism it guards against can drop the guard too — and that absence is silent as well.
  • Assert execution started, not completed, or legitimate early exits become red.
  • RUNNER_TEMP is /tmp on these runners, not a private per-job dir, so key the marker on GITHUB_RUN_ID/GITHUB_RUN_ATTEMPT — otherwise a stale marker makes the guard pass on a dropped run.
  • Prove it behaviourally (execute the guard body with and without the marker), not just structurally: a bare exit 1 substring is satisfiable by dead code.
  • These jobs run in containers on the build lane, unlike the small lane used in #751 — re-measure that a later step still runs after a drop there rather than assuming it transfers.

Also worth considering, since it is the cheaper and more general half: a repo-wide check that no run: body in a job writing a required context contains an expression delimiter. #751 applied that absolutely to review-verdict.yml only.

Done-when

  • Both required docker-build.yml jobs fail when a step is dropped, verified BEHAVIOURALLY (guard body executed with and without the marker)
  • Verified that a later step still runs after a drop on the build lane — not assumed from the small lane measurement in #751
  • ci.workflow-run-body-no-expressions updated: its rule: currently states this is NOT yet a repo-wide property and points here
  • docs/ci-cd.md updated
  • Adversarial review passed
Split out of #751, which fixed the silent-green class **only in `.gitea/workflows/review-verdict.yml`**. Raised by cold review of that fix. ## The asymmetry A step the runner declines to interpolate is **dropped, and the job concludes `success`** (#751, `ci.workflow-run-body-no-expressions`). What that costs depends on which required check the job writes: | Job | On a dropped step | Direction | |---|---|---| | `review-verdict.yml` / `set-verdict-status` | `review-verdict/h10` is **absent** → merge blocked | fail-CLOSED | | `docker-build.yml` / `test` (`Build & test (.NET)`) | required context goes **green having done no work** | **fail-OPEN** | | `docker-build.yml` / `migrations` (`EF migration integrity`) | required context goes **green having done no work** | **fail-OPEN** | Those three are the required contexts on `main`. #751 guarded the one where the failure mode is *safe* and left the two where it is *dangerous* — because that is where the live bug was, not because the others were checked and found fine. This repo has already been bitten by a required check going green without running the suite (#684, the docs-only fast path). That was a different mechanism with the same shape and the same cost. ## Currently latent, which is exactly when it is cheap Neither `test` nor `migrations` has an expression delimiter in a `run:` body today (`build`, `api-docs` and `format` do, so the file is not delimiter-free and the class is reachable by an ordinary edit). Latent, not absent. ## Proposed Extend the start-marker pattern to the two required `docker-build.yml` jobs: each writes a marker as its first act, and a trailing `if: always()` step fails the job when it is missing. Notes from doing it in #751: - The guard body must be **expression-free**, or the mechanism it guards against can drop the guard too — and that absence is silent as well. - Assert execution **started**, not completed, or legitimate early exits become red. - `RUNNER_TEMP` is `/tmp` on these runners, not a private per-job dir, so key the marker on `GITHUB_RUN_ID`/`GITHUB_RUN_ATTEMPT` — otherwise a stale marker makes the guard *pass* on a dropped run. - Prove it **behaviourally** (execute the guard body with and without the marker), not just structurally: a bare `exit 1` substring is satisfiable by dead code. - These jobs run in containers on the build lane, unlike the `small` lane used in #751 — re-measure that a later step still runs after a drop there rather than assuming it transfers. Also worth considering, since it is the cheaper and more general half: a repo-wide check that no `run:` body in a job writing a **required** context contains an expression delimiter. #751 applied that absolutely to `review-verdict.yml` only. ## Done-when - [x] Both required `docker-build.yml` jobs fail when a step is dropped, verified BEHAVIOURALLY (guard body executed with and without the marker) - [x] Verified that a later step still runs after a drop **on the build lane** — not assumed from the `small` lane measurement in #751 - [x] `ci.workflow-run-body-no-expressions` updated: its `rule:` currently states this is NOT yet a repo-wide property and points here - [x] `docs/ci-cd.md` updated - [x] Adversarial review passed
timothy added the ci-cdpriority: highsecurity labels 2026-08-06 21:30:33 +02:00
Author
Owner

Claiming this (Claude Code orchestrator session, 2026-08-10). Checked for prior work first: no open PR references #756, git ls-remote --heads origin '*756*' is empty, no prior comments. Will extend the #751 start-marker guard to test + migrations in docker-build.yml, re-measuring the drop behaviour on the build lane rather than assuming it transfers from small.

Claiming this (Claude Code orchestrator session, 2026-08-10). Checked for prior work first: no open PR references #756, `git ls-remote --heads origin '*756*'` is empty, no prior comments. Will extend the #751 start-marker guard to `test` + `migrations` in `docker-build.yml`, re-measuring the drop behaviour on the build lane rather than assuming it transfers from `small`.
Author
Owner

Closing record

Outcome: Shipped in PR #768 (squashed to 884ac8a7e). Both required docker-build.yml jobs now fail when a step is dropped, via per-STEP execution markers (scripts/ci-step-ran.sh) plus a trailing guard; and the raw ${{ opener is banned in test, migrations and build.

Root cause: #751 fixed the silent-green class only where it was fail-CLOSED (review-verdict.yml, an absent required status blocks the merge). The same runner behaviour in docker-build.yml's test/migrations is fail-OPEN — a required check reports green having done no work — and was left because that is where the live bug was, not because these were checked.

Decisions/conventions changed: Added ci.required-job-step-execution-markers. Updated ci.workflow-run-body-no-expressions, whose rule: named this issue as its open residual and separately asserted that build interpolates legitimately and "a drop there fails no gate" — both now false.

Reusable knowledge:

  • Two of the worst defects were in the TESTS, not the code. A fixture that never sent the guard its production gate value (false) hid a one-token fail-open in the central predicate — mutate = "true" to -n and all 30 tests stayed green while the guard reported "All 2 expected step(s) executed" on every ordinary run. And a positive control that passed whether or not the mechanism worked. A green suite was never the signal; the surviving mutation was.
  • Five commits carried Decisions-Edit: yes and git parsed none of them. Refs #756 has no colon, so it is not a trailer, and one bad line voids the whole block. It would have surfaced as a red decisions lifecycle — the one red this repo is standing-instructed to dismiss as a known flake. Verify with git log --format='%(trailers:key=Decisions-Edit,valueonly)', never by reading the message.
  • Prefer making the system REPORT over inferring. Three questions here were settled by printed evidence and two inference-based arguments about the same variable were both wrong: grepping a job log for a variable NAME proves nothing (logs do not dump the environment), and "the warning never appears" proves nothing when the warning goes to stderr and stderr capture is itself unestablished. Adding a stdout Marker identity: line answered it in one run.
  • A $( ) helper's STDOUT is its return value. Printing a notice inside marker_path embedded it in the path.
  • A guard for a multi-step job must NOT use if: always(). With a dozen steps, a genuine early failure skips the rest, so always() cries wolf on every red build. Default success() works because guard-skipped implies job-red.

Verification: 510 tests; 30 mutations killed across two harnesses; decisions-validate clean. Measured live on the build lane in BOTH directions — probe #765 (test: step dropped, guard caught it, the only failure in the job, so the pre-change outcome would have been green; migrations untouched: All 6 expected step(s) executed, green) and probe #766 (a failing continue-on-error step does NOT skip the guard: All 12 expected step(s) executed, green). Both probe PRs closed unmerged, branches deleted. Five cold review rounds across two model families (Codex GPT-5.x, Claude Opus).

Deferred: #767 — the build delimiter ban is enforced only by script-tests, which is on: pull_request and not a required context, so it is review-time rather than fail-closed on the tag/release path. Not closed here because Smoke's if: carries a publish-ref condition the guard's always/gated buckets do not model. An earlier draft also claimed measuring it would require cutting a real release tag; that was wrong and is retracted on #767 — build runs on every push to main.

Docs updated: docs/ci-cd.md (new "Dropped-step guard on the required jobs" section, plus corrections to the #751 section), docs/decisions/records/ci/required-job-step-execution-markers.md (new), docs/decisions/records/ci/workflow-run-body-no-expressions.md, docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Shipped in PR #768 (squashed to `884ac8a7e`). Both required `docker-build.yml` jobs now fail when a step is dropped, via per-STEP execution markers (`scripts/ci-step-ran.sh`) plus a trailing guard; and the raw `${{` opener is banned in `test`, `migrations` and `build`. **Root cause:** #751 fixed the silent-green class only where it was fail-CLOSED (`review-verdict.yml`, an absent required status blocks the merge). The same runner behaviour in `docker-build.yml`'s `test`/`migrations` is fail-OPEN — a required check reports green having done no work — and was left because that is where the live bug was, not because these were checked. **Decisions/conventions changed:** Added `ci.required-job-step-execution-markers`. Updated `ci.workflow-run-body-no-expressions`, whose `rule:` named this issue as its open residual and separately asserted that `build` interpolates legitimately and "a drop there fails no gate" — both now false. **Reusable knowledge:** - **Two of the worst defects were in the TESTS, not the code.** A fixture that never sent the guard its production gate value (`false`) hid a one-token fail-open in the central predicate — mutate `= "true"` to `-n` and all 30 tests stayed green while the guard reported "All 2 expected step(s) executed" on every ordinary run. And a positive control that passed whether or not the mechanism worked. A green suite was never the signal; the surviving mutation was. - **Five commits carried `Decisions-Edit: yes` and git parsed none of them.** `Refs #756` has no colon, so it is not a trailer, and one bad line voids the whole block. It would have surfaced as a red `decisions lifecycle` — the one red this repo is standing-instructed to dismiss as a known flake. Verify with `git log --format='%(trailers:key=Decisions-Edit,valueonly)'`, never by reading the message. - **Prefer making the system REPORT over inferring.** Three questions here were settled by printed evidence and two inference-based arguments about the same variable were both wrong: grepping a job log for a variable NAME proves nothing (logs do not dump the environment), and "the warning never appears" proves nothing when the warning goes to stderr and stderr capture is itself unestablished. Adding a stdout `Marker identity:` line answered it in one run. - **A `$( )` helper's STDOUT is its return value.** Printing a notice inside `marker_path` embedded it in the path. - **A guard for a multi-step job must NOT use `if: always()`.** With a dozen steps, a genuine early failure skips the rest, so `always()` cries wolf on every red build. Default `success()` works because guard-skipped implies job-red. **Verification:** 510 tests; 30 mutations killed across two harnesses; `decisions-validate` clean. Measured live on the build lane in BOTH directions — probe #765 (`test`: step dropped, guard caught it, the only failure in the job, so the pre-change outcome would have been green; `migrations` untouched: `All 6 expected step(s) executed`, green) and probe #766 (a failing `continue-on-error` step does NOT skip the guard: `All 12 expected step(s) executed`, green). Both probe PRs closed unmerged, branches deleted. Five cold review rounds across two model families (Codex GPT-5.x, Claude Opus). **Deferred:** #767 — the `build` delimiter ban is enforced only by `script-tests`, which is `on: pull_request` and not a required context, so it is review-time rather than fail-closed on the tag/release path. Not closed here because `Smoke`'s `if:` carries a publish-ref condition the guard's always/gated buckets do not model. An earlier draft also claimed measuring it would require cutting a real release tag; that was wrong and is retracted on #767 — `build` runs on every push to `main`. **Docs updated:** `docs/ci-cd.md` (new "Dropped-step guard on the required jobs" section, plus corrections to the #751 section), `docs/decisions/records/ci/required-job-step-execution-markers.md` (new), `docs/decisions/records/ci/workflow-run-body-no-expressions.md`, `docs/decisions/README.md` (regenerated).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#756