fix(767): gate the release path on the delimiter ban with a prerequisite job #770

Merged
timothy merged 1 commits from fix/767-delimiter-ban-fail-closed into main 2026-08-13 20:04:53 +02:00
Owner

fixes #767

The delimiter ban that keeps build's Smoke + IPTV E2E from being silently dropped was enforced only by a pytest in script-tests — on: pull_request, and not a required context. Nothing re-checked it on a v* tag push, which is exactly when the candidate image is published and DeployStack jazz-media promotes it. A delimiter reaching main would drop Smoke on the tag build, publish an unsmoked candidate, and report green.

A scan job now runs the PyYAML-based ban test, and build lists it in needs:. A red scan skips build outright, so the image is never built — fail-closed by dependency, not by assertion.

Measured end to end, without cutting a release

Two workflow_dispatch runs on scratch branches (a non-main/non-v* ref runs build but publishes nothing, so the release path is exercised without a release):

scan evidence in the job log build
Poisoned Smoke body (run 1928) failure 1 failed, 64 passed — the ban test, naming Smoke + IPTV E2E: 'steps.meta.outputs.short' skipped
Clean control (run 1929) success 65 passed; guard reported All 2 expected step(s) executed: deps ban ran, success

Both directions, each for the right reason. The control was checked for a vacuous green — 13s is fast, so the log was pulled rather than trusting the tick.

What is deliberately not cited as evidence: "no image was published" holds on a scratch ref regardless of the gate (Build and push has push: false there), so it could not have come out the other way. The discriminating observation is scan red → build skipped.

Three kinds of check, because each single kind was defeated

Seven adversarial review rounds, each finding a real defect in the one before. They rhyme, and that is the reusable part: every fix was right about its instance and wrong about its class.

Attempt How it was defeated
A bespoke stdlib scanner as a step inside build A guard step cannot protect the job it publishes from; "my body has no opener so I can't be dropped" is circular when only the PR-only test enforces that. Its hand parser also had ~10 false negatives. Deleted.
Shape assertions about the invocation echoing the command, flags past a \ continuation, exit 0, if false; then … fi, set +e, PYTEST_ADDOPTS in step env:, plus sibling-module mutations no shape check could reach
Reconstructing only the step env: The same disarm one tier up (job env:) still worked
Trusting needs: as the gate always() in build's if: downgrades the edge to mere ordering
Static reconstruction of any tier $GITHUB_ENV writes inject into later steps at runtime — banned rather than modelled
Anything living inside pytest A repo-root pytest.ini/conftest.py deselects the ban test and every test guarding it

So the gate now rests on three different kinds of check, chosen so no single channel takes all three down:

  1. the ban test itself, run by the scan job;
  2. an execution probe that runs that command against a poisoned copy, with the workflow/job/step env: tiers layered;
  3. scripts/ci-prove-ban-detects.sh — not a test. It poisons the workflow in the real checkout, re-runs the ban test, and vouches only for that test's build parametrisation failing.

That last one is deliberately outside pytest, because pytest's own configuration outranks anything inside it. It runs in the real checkout because an isolated copy does not inherit the repo-root config a disarm would live in — the first version of the script made exactly that mistake and cheerfully reported "the gate is enforcing" against four separate disarms.

Its three refusal branches are exhaustive, and each has its own positive control: exit 0 → the ban is not enforcing; any exit ≠ 1 → cannot prove anything (a total deselect exits 5, and "non-zero means it noticed" was a live bug here); exit 1 without the [build] failure → failed for the wrong reason (also a live bug). For a while that script was the one guard exercised only on the happy path — the exact failure it argues about everything else.

Two disarms review reported are not real and are recorded as such: --ignore= and collect_ignore do not suppress a file pytest is given explicitly as an argument. But refuting two variants of a channel did not clear the channel — the working attacks through that same pytest-config channel were simply never tried, which is what a later round found.

Also corrected

Claims this PR's subject makes or breaks, all found by review while verifying the above:

  • build's needs: was cited stale in three places, one a decision record (breadcrumb rule).
  • "the docs-only path still builds an image" — it does not; Build and push carries the docs_only gate too.
  • The small lane's 10g cap attributed to build alone, where the record names two heavy jobs.
  • A corroboration from script-tests that cannot hold: it is on: pull_request and has never run on the tag path this job now sits on.
  • "the one job that installs from the network" — test reaches nuget.org/npm on a cache miss.
  • "an interrupted run cannot leave a poisoned tree behind" — SIGKILL falsifies it; now qualified, with why it stays harmless stated rather than assumed.
  • The marker-path rationale is carved out for scan: it has no container:, so the rerun residual the container jobs no longer have still applies to it.

Residual, stated

Not claimed: that a step can never fail to run for a reason other than the interpolation drop. scan's own steps carry #756 markers and a trailing assert, which moves the terminal assumption rather than removing it. Per-step markers on build itself remain a genuine smaller residual.

scan is also the first small-lane job on the tag path since #535 moved them off, and the only lane member whose failure skips the release build rather than merely reddening a status. That is safe only because server-management#639 took build off the lane; if the wedging returns, move scan to ubuntu-latest — never weaken the needs: edge.

fixes #767 The delimiter ban that keeps `build`'s `Smoke + IPTV E2E` from being silently dropped was enforced **only** by a pytest in `script-tests` — `on: pull_request`, and not a required context. Nothing re-checked it on a `v*` tag push, which is exactly when the candidate image is published and `DeployStack jazz-media` promotes it. A delimiter reaching `main` would drop `Smoke` on the tag build, publish an unsmoked candidate, and report green. **A `scan` job now runs the PyYAML-based ban test, and `build` lists it in `needs:`.** A red `scan` skips `build` outright, so the image is never built — fail-closed by dependency, not by assertion. ## Measured end to end, without cutting a release Two `workflow_dispatch` runs on scratch branches (a non-`main`/non-`v*` ref runs `build` but publishes nothing, so the release path is exercised without a release): | | `scan` | evidence in the job log | `build` | |---|---|---|---| | Poisoned `Smoke` body (run **1928**) | **failure** | `1 failed, 64 passed` — the ban test, naming `Smoke + IPTV E2E: 'steps.meta.outputs.short'` | **skipped** | | Clean control (run **1929**) | **success** | `65 passed`; guard reported `All 2 expected step(s) executed: deps ban` | **ran, success** | Both directions, each for the right reason. The control was checked for a *vacuous* green — 13s is fast, so the log was pulled rather than trusting the tick. What is deliberately **not** cited as evidence: "no image was published" holds on a scratch ref regardless of the gate (`Build and push` has `push: false` there), so it could not have come out the other way. The discriminating observation is `scan` red → `build` skipped. ## Three kinds of check, because each single kind was defeated Seven adversarial review rounds, each finding a real defect in the one before. They rhyme, and that is the reusable part: **every fix was right about its instance and wrong about its class.** | Attempt | How it was defeated | |---|---| | A bespoke stdlib scanner as a step *inside* `build` | A guard step cannot protect the job it publishes from; "my body has no opener so I can't be dropped" is circular when only the PR-only test enforces that. Its hand parser also had ~10 false negatives. **Deleted.** | | Shape assertions about the invocation | `echo`ing the command, flags past a `\` continuation, `exit 0`, `if false; then … fi`, `set +e`, `PYTEST_ADDOPTS` in step `env:`, plus sibling-module mutations no shape check could reach | | Reconstructing only the **step** `env:` | The same disarm one tier up (job `env:`) still worked | | Trusting `needs:` as the gate | `always()` in `build`'s `if:` downgrades the edge to mere ordering | | Static reconstruction of any tier | `$GITHUB_ENV` writes inject into later steps at **runtime** — banned rather than modelled | | Anything living inside pytest | A repo-root `pytest.ini`/`conftest.py` deselects the ban test **and every test guarding it** | So the gate now rests on three different **kinds** of check, chosen so no single channel takes all three down: 1. **the ban test** itself, run by the `scan` job; 2. **an execution probe** that runs that command against a poisoned *copy*, with the workflow/job/step `env:` tiers layered; 3. **`scripts/ci-prove-ban-detects.sh`** — *not* a test. It poisons the workflow in the **real checkout**, re-runs the ban test, and vouches only for that test's `build` parametrisation failing. That last one is deliberately outside pytest, because pytest's own configuration outranks anything inside it. It runs in the real checkout because an isolated copy does not inherit the repo-root config a disarm would live in — the first version of the script made exactly that mistake and cheerfully reported "the gate is enforcing" against four separate disarms. Its three refusal branches are exhaustive, and **each has its own positive control**: `exit 0` → the ban is not enforcing; any exit ≠ 1 → cannot prove anything (a total deselect exits **5**, and "non-zero means it noticed" was a live bug here); exit 1 without the `[build]` failure → failed for the wrong reason (also a live bug). For a while that script was the one guard exercised only on the happy path — the exact failure it argues about everything else. Two disarms review reported are **not real** and are recorded as such: `--ignore=` and `collect_ignore` do not suppress a file pytest is given explicitly as an argument. But refuting two variants of a channel did not clear the channel — the *working* attacks through that same pytest-config channel were simply never tried, which is what a later round found. ## Also corrected Claims this PR's subject makes or breaks, all found by review while verifying the above: - `build`'s `needs:` was cited **stale** in three places, one a decision record (breadcrumb rule). - "the docs-only path still builds an image" — it does not; `Build and push` carries the docs_only gate too. - The `small` lane's 10g cap attributed to `build` alone, where the record names **two** heavy jobs. - A corroboration from `script-tests` that cannot hold: it is `on: pull_request` and has never run on the tag path this job now sits on. - "the one job that installs from the network" — `test` reaches nuget.org/npm on a cache miss. - "an interrupted run cannot leave a poisoned tree behind" — **SIGKILL falsifies it**; now qualified, with why it stays harmless stated rather than assumed. - The marker-path rationale is **carved out** for `scan`: it has no `container:`, so the rerun residual the container jobs no longer have still applies to it. ## Residual, stated Not claimed: that a step can never fail to run for a reason other than the interpolation drop. `scan`'s own steps carry #756 markers and a trailing assert, which *moves* the terminal assumption rather than removing it. Per-step markers on `build` itself remain a genuine smaller residual. `scan` is also the first `small`-lane job on the tag path since #535 moved them off, and the only lane member whose failure skips the release build rather than merely reddening a status. That is safe only because server-management#639 took `build` off the lane; if the wedging returns, move `scan` to `ubuntu-latest` — never weaken the `needs:` edge.
Author
Owner

Review-verdict: MERGEABLE @ 1242f9f

Eight adversarial rounds (rounds 1-8); rounds 1-5 found real defects, 6-8 returned MERGEABLE. Fail-closed behaviour measured live on runs 1928 (poisoned: scan failed, build SKIPPED) and 1929 (control: scan green, build ran). Final head differs from the round-8 sha by a COMMIT MESSAGE ONLY - tree byte-identical (cb56ae0e) - correcting an overclaim that all three refusal branches had positive controls; the third does not, is fail-closed anyway, and is now recorded as such. Round-8 nits 2-5 deferred as follow-ups, not fixed.

Review-verdict: MERGEABLE @ 1242f9f Eight adversarial rounds (rounds 1-8); rounds 1-5 found real defects, 6-8 returned MERGEABLE. Fail-closed behaviour measured live on runs 1928 (poisoned: scan failed, build SKIPPED) and 1929 (control: scan green, build ran). Final head differs from the round-8 sha by a COMMIT MESSAGE ONLY - tree byte-identical (cb56ae0e) - correcting an overclaim that all three refusal branches had positive controls; the third does not, is fail-closed anyway, and is now recorded as such. Round-8 nits 2-5 deferred as follow-ups, not fixed.
timothy added 1 commit 2026-08-13 19:18:55 +02:00
fix(767): gate the release path on the delimiter ban with a prerequisite job
review-verdict/h10 Review-verdict: MERGEABLE @ f420c66 (base: main)
Build ErsatzTV Image / Delimiter ban (release path) (pull_request) Successful in 19s
PR Gates / Docs update reminder (pull_request) Successful in 17s
PR Gates / CI image pin matches docker/ci (pull_request) Successful in 18s
PR Gates / decisions lifecycle (pull_request) Successful in 26s
Review verdict / Set review-verdict status (pull_request_target) Successful in 10s
PR Gates / Script tests (pytest) (pull_request) Successful in 2m6s
Build ErsatzTV Image / Build & test (.NET) (pull_request) Successful in 9m25s
Build ErsatzTV Image / EF migration integrity (SQLite + MySql) (pull_request) Successful in 7m2s
Build ErsatzTV Image / Build & push image (amd64) (pull_request) Skipped
Build ErsatzTV Image / Functional E2E (curl + UI contracts) (pull_request) Successful in 6m30s
Build ErsatzTV Image / API docs in sync (OpenAPI + endpoint index) (pull_request) Successful in 9s
Build ErsatzTV Image / Formatting (changed .cs conform to .editorconfig) (pull_request) Successful in 5s
f420c668e6
The ban that keeps `build`'s `Smoke + IPTV E2E` from being silently dropped was enforced
only by a pytest in `script-tests` — `on: pull_request`, and not a required context. Nothing
re-checked it on a `v*` tag push, which is exactly when the candidate image is published and
`DeployStack jazz-media` promotes it. A delimiter that reached `main` would drop `Smoke` on
the tag build, publish an unsmoked candidate, and report green.

A `scan` job now runs the PyYAML-based ban test, and `build` lists it in `needs:`. A red
`scan` skips `build` outright, so the image is never built.

MEASURED, both directions, without cutting a release: two `workflow_dispatch` runs on scratch
branches. Poisoned `Smoke` (run 1928) -> `scan` FAILED on the ban test, naming the payload,
and `Build & push` was SKIPPED. Clean control (run 1929) -> `scan` green (`65 passed`, guard
reported `All 2 expected step(s) executed`) and `build` RAN to success. "No image was
published" is deliberately NOT cited as evidence: on a scratch ref `push:` is false anyway,
so that conjunct could not have come out the other way.

SEVEN REVIEW ROUNDS, each finding a real defect in the one before. They rhyme, and that is the
reusable part: every fix was right about its instance and wrong about its class.

  * A bespoke stdlib scanner as a step inside `build`. A guard STEP cannot protect the job it
    lives in, and "my body has no opener so I cannot be dropped" is circular when only the
    PR-only test enforces that. Its hand parser also had ~10 false NEGATIVES. Deleted.
  * Shape assertions about the invocation — disarmed by `echo`ing the command, flags past a
    `\` continuation, `exit 0`, `if false; then .. fi`, `set +e`, `PYTEST_ADDOPTS` in the step
    `env:`, and mutations in the sibling module no shape check could reach.
  * Reconstructing only the STEP env tier; the same disarm one tier up still worked.
  * Trusting `needs:` as the gate — `always()` in `build`'s `if:` downgrades it to ordering.
  * `$GITHUB_ENV` writes inject into later steps at RUNTIME, invisible to static checks.
  * A repo-root `pytest.ini`/`conftest.py` deselects the ban test AND every test guarding it.

So the gate rests on three different KINDS of check, so that no single channel takes all three
down: the ban test; an execution probe running that command against a poisoned COPY with the
workflow/job/step `env:` tiers layered; and `scripts/ci-prove-ban-detects.sh`, which is NOT a
test — it poisons the workflow in the REAL checkout, re-runs the ban test, and vouches ONLY
for that test's `build` parametrisation failing. Any other outcome is a refusal, not a pass:
"non-zero means it noticed" reported OK on a collection error, and the deselection disarm this
exists to catch exits 5 rather than 1. Both weaker readings were live bugs here.

The two refusal branches a disarm actually lands on each have their own positive control,
because for a while the proof script was the one guard exercised only on the happy path — the
exact failure it argues about everything else. The third (pytest passing outright) has none:
neutering it is still fail-CLOSED, since the run then falls through to the exit-code branch and
refuses, so it only buys a better message. Recorded rather than claimed away — an earlier draft
of this very paragraph said all three were controlled, which was false.

Two disarms review reported are NOT real and are recorded as such: `--ignore=` and
`collect_ignore` do not suppress a file pytest is given explicitly. But refuting two variants
of a channel did not clear the channel; the working attacks through that same pytest-config
channel were simply never tried, which is what a later round found.

Also corrected: three stale citations of `build`'s `needs:` including one in a decision record;
a header claim that the docs-only path still builds an image (it does not); the `small` lane's
10g cap attributed to `build` alone where the record names two heavy jobs; a corroboration
from `script-tests` that cannot hold since it is `on: pull_request` and has never run on the
tag path; an "only job that installs from the network" claim ignoring `test`'s cache misses;
and "an interrupted run cannot leave a poisoned tree behind", which SIGKILL falsifies. The
marker-path rationale is carved out for `scan`, which has no `container:` and so keeps the
rerun residual the container jobs no longer have.

Refs: #767
Decisions-Edit: yes
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
timothy force-pushed fix/767-delimiter-ban-fail-closed from 1242f9f9b8 to f420c668e6 2026-08-13 19:18:55 +02:00 Compare
Author
Owner

Review-verdict: MERGEABLE @ f420c66

Round 8 returned MERGEABLE on tree cb56ae0e with 4 cosmetic nits + 1 overclaim. All 5 now fixed rather than deferred (they were cheap; deferring a line-wrap was the wrong call). Delta since the reviewed tree: two test assertions tightened from disjunctions to the specific branch each was written for, one line rewrapped, two doc/record wordings corrected, and a commit-message overclaim retracted. The two tightened assertions were re-mutation-verified: disabling the grep branch alone reddens the wrong-test test; disabling the exit-code branch alone reddens the deselect test. 538 pass, validators clean.

Review-verdict: MERGEABLE @ f420c66 Round 8 returned MERGEABLE on tree cb56ae0e with 4 cosmetic nits + 1 overclaim. All 5 now fixed rather than deferred (they were cheap; deferring a line-wrap was the wrong call). Delta since the reviewed tree: two test assertions tightened from disjunctions to the specific branch each was written for, one line rewrapped, two doc/record wordings corrected, and a commit-message overclaim retracted. The two tightened assertions were re-mutation-verified: disabling the grep branch alone reddens the wrong-test test; disabling the exit-code branch alone reddens the deselect test. 538 pass, validators clean.
timothy closed this pull request 2026-08-13 19:39:56 +02:00
timothy reopened this pull request 2026-08-13 19:40:00 +02:00
timothy merged commit cfdab63a3a into main 2026-08-13 20:04:53 +02:00
timothy deleted branch fix/767-delimiter-ban-fail-closed 2026-08-13 20:04:54 +02:00
Sign in to join this conversation.