Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7fcb5e9b28 |
@@ -643,6 +643,61 @@ jobs:
|
||||
# server. Its exit status is Playwright's.
|
||||
scripts/e2e-ui.sh
|
||||
|
||||
# THE DELIMITER BAN, RE-CHECKED ON THE RELEASE PATH ITSELF (ersatztv#767).
|
||||
#
|
||||
# The ban that keeps `build`'s `Smoke + IPTV E2E` from being silently dropped was enforced only by
|
||||
# `test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body` in the
|
||||
# `script-tests` job of pr-checks.yml — `on: pull_request`, and NOT a required context. So the ban
|
||||
# was REVIEW-TIME only: nothing re-checked it on a `v*` tag push, which is precisely when the
|
||||
# candidate image is published and `DeployStack jazz-media` promotes it.
|
||||
#
|
||||
# WHY A JOB AND NOT A STEP INSIDE `build`. A step cannot protect the thing it shares a job with:
|
||||
# `build` is what publishes, so a guard step there fails OPEN if the runner drops it, and "my body
|
||||
# has no opener so I cannot be dropped" is circular when the only thing enforcing that property is
|
||||
# the same PR-only test being backstopped. As a `needs:` of `build`, a red here means `build` never
|
||||
# runs at all — the image is not built, let alone pushed. Fail-closed by dependency, not by
|
||||
# assertion.
|
||||
#
|
||||
# WHY IT RUNS THE REAL PYTEST rather than a bespoke scanner. The first cut of #767 hand-parsed the
|
||||
# workflow YAML in stdlib Python, to avoid provisioning PyYAML on `build`'s bare runner. Two
|
||||
# independent reviews found ~10 false NEGATIVES in that parser within one round (flow mappings
|
||||
# `{run: …}`, a quoted `"run":` key, aliases, multiline quoted scalars) — i.e. it was strictly
|
||||
# WEAKER than the check it was meant to backstop, in the one direction that matters for a security
|
||||
# gate. Running the existing PyYAML-based test needs no second implementation of "what is a `run:`
|
||||
# body" and therefore has no drift surface. `small` is git-only, so Python is provisioned here the
|
||||
# same way `script-tests` does it.
|
||||
#
|
||||
# This job's OWN steps carry #756 markers and a trailing assert, so a drop inside THIS job is
|
||||
# caught too. That terminates the regress at the same axiom the sibling guards already rest on —
|
||||
# to fail open you must now drop the pytest step AND the assert step, rather than either one.
|
||||
scan:
|
||||
name: Delimiter ban (release path)
|
||||
runs-on: small
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.x'
|
||||
- name: Install test dependencies
|
||||
run: |
|
||||
"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark deps
|
||||
python3 -m pip install --disable-pip-version-check --quiet pytest pyyaml
|
||||
# The ban test plus the structural tests that hold this job's own shape. NOT the whole
|
||||
# scripts/tests suite: that is `script-tests`'s job, it needs jq/git preflights, and an
|
||||
# unrelated pytest regression must not be able to block a release.
|
||||
- name: Run the delimiter-ban tests
|
||||
run: |
|
||||
"${GITHUB_WORKSPACE:-.}/scripts/ci-step-ran.sh" mark ban
|
||||
PYTHONPATH=. python3 -m pytest scripts/tests/test_ci_dropped_step_guard.py scripts/tests/test_ci_release_path_scan_job.py -q
|
||||
# No `if:` — see the sibling guards in `test`/`migrations` for why the default `success()` is
|
||||
# the wanted condition. Both keys are `--always`: every step in this job is unconditional.
|
||||
- name: Assert every expected step executed (ersatztv#756)
|
||||
run: >-
|
||||
scripts/ci-step-ran.sh assert
|
||||
--always deps ban
|
||||
|
||||
build:
|
||||
name: Build & push image (amd64)
|
||||
# Moved back off `small` (server-management#639). This is the one HEAVY job that
|
||||
@@ -659,7 +714,10 @@ jobs:
|
||||
# was queueing behind has drained. Real builds (main/tags) get the full
|
||||
# ubuntu-latest allotment: 4 CPUs / 10g on ci-runner (.127).
|
||||
runs-on: ubuntu-latest
|
||||
needs: [test, migrations]
|
||||
# `scan` (ersatztv#767) re-checks the delimiter ban on the release path. As a `needs:` its red
|
||||
# SKIPS this job outright, so a delimiter in `Smoke + IPTV E2E` can no longer reach the point
|
||||
# where an image is published and never booted.
|
||||
needs: [test, migrations, scan]
|
||||
if: github.event_name != 'pull_request'
|
||||
steps:
|
||||
- name: Checkout
|
||||
@@ -740,15 +798,15 @@ jobs:
|
||||
# body delimiter-free the class is unreachable here — held by
|
||||
# test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body.
|
||||
#
|
||||
# This job gets the ban but NOT (yet) the per-step markers — and NOT because "it is not a
|
||||
# required context, so the markers would buy little". That is the same family as the
|
||||
# "smaller cost (no required context lies)" dismissal that
|
||||
# `ci.required-job-step-execution-markers` records as FALSE about this very step. Markers
|
||||
# here would buy strictly MORE than the ban does: the ban is enforced only by `script-tests`,
|
||||
# which is `on: pull_request` and not a required check, so nothing re-checks it when a release
|
||||
# is actually cut, whereas markers would be a runtime, fail-closed gate on the tag path itself.
|
||||
# It is deferred because this step's `if:` carries a publish-ref condition the guard's
|
||||
# always/gated buckets do not model. Tracked as ersatztv#767.
|
||||
# The ban IS re-checked on the release path now (ersatztv#767): the `scan` job above runs the
|
||||
# PyYAML-based ban test and is a `needs:` of this job, so a delimiter here means `build` never
|
||||
# runs and no image is published. Do not re-add the note that once stood here saying the ban is
|
||||
# "review-time only, tracked as #767" — that was true before the `scan` job existed.
|
||||
#
|
||||
# This step still carries no per-step markers, and that is a genuine (smaller) residual rather
|
||||
# than a dismissal: markers would additionally catch a drop caused by something OTHER than a
|
||||
# delimiter. Adding them needs a bucket modelling this step's publish-ref `if:`, which the
|
||||
# guard's always/gated buckets do not express. The delimiter class itself is covered.
|
||||
- name: Smoke + IPTV E2E (assert key endpoints)
|
||||
if: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')) && steps.detect.outputs.docs_only != 'true' }}
|
||||
env:
|
||||
|
||||
+42
-16
@@ -629,23 +629,49 @@ Two independent mechanisms hold it, and neither is redundant:
|
||||
jazz-media` promotes exactly that image. Its two payloads moved into the step's `env:`, so the ban
|
||||
cost nothing.
|
||||
|
||||
**Be precise about what that buys, because it is less than it looks.** The ban is enforced by the
|
||||
`script-tests` job, which lives in `pr-checks.yml` (`on: pull_request`) and is **not** a required
|
||||
context. So it is a *review-time* check on the PR that would introduce a delimiter — not a
|
||||
fail-closed gate on the release path. `pr-checks.yml` does not run on a `v*` tag push at all, and
|
||||
`build` has no markers and no guard, so a delimiter that ever reached `main` would still drop
|
||||
`Smoke` on the tag build and go green. `main` being PR-only (#743) means such a change must pass
|
||||
through a PR where `script-tests` reddens, but a red on a non-required check does not block the
|
||||
merge server-side. Strictly better than before, not closed. Tracked as **ersatztv#767**.
|
||||
**The ban is re-checked on the release path itself (ersatztv#767).** It used to be enforced only by
|
||||
the `script-tests` job, which lives in `pr-checks.yml` (`on: pull_request`) and is **not** a
|
||||
required context — a *review-time* check on the PR that would introduce a delimiter, not a gate on
|
||||
the release. `pr-checks.yml` does not run on a `v*` tag push at all, so a delimiter that ever
|
||||
reached `main` would still drop `Smoke` on the tag build and go green; `main` being PR-only (#743)
|
||||
meant such a change had to pass through a PR where `script-tests` reddens, but a red on a
|
||||
non-required check does not block the merge server-side.
|
||||
|
||||
The reason it was deferred is that `Smoke`'s `if:` carries a publish-ref condition the guard's
|
||||
always/gated buckets do not model — that is the whole of it. An earlier draft here also claimed
|
||||
measuring such a guard "would mean cutting a real release tag", and that was simply **wrong**:
|
||||
`build` runs on every push to `main` (`if: github.event_name != 'pull_request'`) and `Smoke`'s ref
|
||||
condition admits `refs/heads/main`, so the runs-side is exercised by any ordinary merge — including
|
||||
the one that lands this change — and the legitimately-skipped side by a `workflow_dispatch` on any
|
||||
other ref. Left recorded because a false cost estimate is exactly what talks the next person out of
|
||||
a tracked follow-up. `functional-e2e` is delimiter-free too but is deliberately **not** banned: it is
|
||||
There is now a **`scan` job** (`Delimiter ban (release path)`) that runs the PyYAML-based ban test,
|
||||
and **`build` lists it in `needs:`**. That single edge is the fail-closed property: a red `scan`
|
||||
means `build` is skipped outright, so the image is never built, let alone pushed.
|
||||
|
||||
**Why a job and not a step inside `build`.** A step cannot protect the job it lives in. `build` is
|
||||
what publishes, so a guard step there fails **open** if the runner drops it — and the defence
|
||||
("the guard's own body has no opener, so it cannot be dropped") is circular when the only thing
|
||||
enforcing that property is the same PR-only test being backstopped. This was the first design and
|
||||
two independent reviews rejected it for exactly that.
|
||||
|
||||
**Why it runs the real pytest and not a bespoke scanner.** The same first cut hand-parsed the
|
||||
workflow YAML in stdlib Python, to avoid provisioning PyYAML on `build`'s bare runner. Review found
|
||||
~10 **false negatives** in that parser in one round — flow mappings (`{run: …}`), a quoted
|
||||
`"run":` key, aliases, multiline quoted scalars — making it strictly *weaker* than the check it
|
||||
backstopped, in the only direction that matters for a security gate. Running the existing test
|
||||
needs no second definition of "what is a `run:` body", so it has no drift surface at all. `scan`
|
||||
runs on `small` and provisions Python the same way `script-tests` does.
|
||||
|
||||
The wiring is held by `scripts/tests/test_ci_release_path_scan_job.py` — `build` depends on it, it
|
||||
carries **no job-level `if:`** (one that excluded the tag push would restore the hole; one that
|
||||
skipped the job would skip `build` too), no step is `continue-on-error`, it actually invokes the
|
||||
ban test, and every one of its own `run:` bodies is delimiter-free. Its steps also carry #756
|
||||
markers and a trailing assert, so a drop *inside this job* is caught as well.
|
||||
|
||||
What this does **not** claim: that no step can ever fail to run for a reason other than the
|
||||
interpolation drop. It moves the terminal assumption — to fail open you must now drop the pytest
|
||||
step **and** the assert step, rather than either one alone.
|
||||
|
||||
Measuring a guard on this path does **not** require cutting a release, and an earlier draft here
|
||||
claiming it would was simply wrong: `build` runs on every push to `main`
|
||||
(`if: github.event_name != 'pull_request'`), and a `workflow_dispatch` on any other ref runs the
|
||||
job while `Build and push` publishes nothing (its `push:` is gated on `main`/`v*`). That is how
|
||||
#767 was verified — see the decision record for the run ids.
|
||||
|
||||
`functional-e2e` is delimiter-free too but is deliberately **not** banned: it is
|
||||
advisory by declaration, and the rule is "ban where a drop is consequential", not "ban wherever it
|
||||
is currently free". `api-docs` and `format` keep one `github.base_ref` each in a detect step and
|
||||
gate nothing that ships.
|
||||
|
||||
@@ -56,7 +56,7 @@ the link for rationale. Superseded/retired history lives in `archive/`. Regenera
|
||||
| `ci.monitor-armed-at-pr-open` | Arm a CI monitor on the PR head sha the moment the PR opens, polling the commit-status endpoint — not at the end of the work. | 2026-07-21 | [link](records/ci/monitor-armed-at-pr-open.md) |
|
||||
| `ci.no-host-health-gating` | Push when your work is validated — never SSH to bumblebee to sample load/RAM first, and never hand-schedule around other sessions' runs. | 2026-07-21 | [link](records/ci/no-host-health-gating.md) |
|
||||
| `ci.peak-anon-measurement` | The `test` job's headline memory figure is a sampled high-water mark of cgroup `anon`, produced by `scripts/ci-peak-anon.sh`; `memory.peak` and the end-of-job `anon`/`file` split are kept only as a cache-inflated reference. | 2026-07-19 | [link](records/ci/peak-anon-measurement.md) |
|
||||
| `ci.required-job-step-execution-markers` | A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`'s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job's LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step's `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships. | 2026-08-10 | [link](records/ci/required-job-step-execution-markers.md) |
|
||||
| `ci.required-job-step-execution-markers` | A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`'s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job's LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step's `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships. The ban is enforced on the RELEASE PATH itself, not only in review (#767): a `scan` job runs the PyYAML-based ban test and `build` lists it in `needs:`, so a delimiter means `build` never runs and no image is published. A guard STEP inside `build` was tried first and is wrong — a step cannot protect the job it publishes from, and "my body has no opener so I cannot be dropped" is circular when only the PR-only test enforces that. The pytest in `script-tests` remains, but it is `on: pull_request` and not a required context, so it alone left the tag path unchecked. | 2026-08-10 | [link](records/ci/required-job-step-execution-markers.md) |
|
||||
| `ci.root-screenshot-guard` | The Husky `pre-commit` hook refuses a staged root-level `*.png` (belt-and-suspenders with the `.gitignore` rule); nested `*.png` real assets are unaffected. | 2026-07-12 | [link](records/ci/root-screenshot-guard.md) |
|
||||
| `ci.runner-placement` | No persistent Roslyn compiler server survives a CI build (`UseSharedCompilation=false` etc., runner env + Dockerfile `ENV`); every `services:` container gets its own explicit `--memory`/`--memory-swap`/`--cpus` cap (it does not inherit the job container's). | 2026-07-17 | [link](records/ci/runner-placement.md) |
|
||||
| `ci.script-tests-job` | The `scripts/tests/` pytest suite runs on every PR as a dedicated `script-tests` job in `pr-checks.yml` (`runs-on: small`, `setup-python` + `pip install pytest`, `PYTHONPATH=. python3 -m pytest scripts/tests -q`), unconditionally rather than behind a `scripts/**` path filter, and **never as a step inside `decisions-guard`** — a job whose reds a standing rule instructs sessions to ignore must never host a gate whose reds are real. Any new CI gate must be reachable by a failure that is unambiguously attributable to it. | 2026-07-26 | [link](records/ci/script-tests-job.md) |
|
||||
|
||||
@@ -5,9 +5,9 @@ status: active
|
||||
since: '2026-08-10'
|
||||
supersedes: none
|
||||
superseded-by: none
|
||||
rule: 'A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`''s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job''s LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step''s `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships.'
|
||||
signals: 'required check green but no work done, step never ran but job green, Build & test green in seconds, EF migration integrity green without replaying, missing Run Main step marker, Unable to interpolate expression format(, dropped step docker-build, ci-step-ran.sh, marker file, expression delimiter in a required job · paths: `.gitea/workflows/docker-build.yml`, `scripts/ci-step-ran.sh`, `scripts/tests/test_ci_dropped_step_guard.py` · issues: #756, #751, #684'
|
||||
mechanics: '`scripts/ci-step-ran.sh` owns the marker path so it exists ONCE and the write and the read cannot diverge. It is keyed on `GITHUB_JOB`/`GITHUB_RUN_ID` — REQUIRED, refusing rather than falling back to a reusable name — plus `GITHUB_RUN_ATTEMPT`. All three REFUSE rather than falling back to a reusable name. The third was warn-and-default until its presence was measured: grepping a log for the variable NAME proves nothing, and inferring it from the absence of a stderr warning proves nothing either (stderr capture was itself unestablished), so `assert` was made to print `Marker identity: job=… run=… attempt=… (from the runner)` on STDOUT and the answer was read off run 1916 for both required jobs. That line is retained as standing evidence. Do NOT justify the keying with #751''s "RUNNER_TEMP is /tmp, not a private per-job dir": that was measured on a job with no `container:` and does not transfer — these jobs get a fresh container, which is the primary protection, and the keying is defence in depth. Held by `scripts/tests/test_ci_dropped_step_guard.py`: static (marker set derived from the workflow equals the guard''s expectations, bucket matches each step''s `if:`, guard is last / has no `if:` / is not advisory / has no delimiter) and behavioural (the guard''s real command line executed against markers written by the steps'' real marker lines, dropping each key in turn).'
|
||||
rule: 'A step the runner declines to interpolate is DROPPED and the job still concludes `success` (`ci.workflow-run-body-no-expressions`). In `review-verdict.yml` that is fail-CLOSED — the required status is absent and the merge is blocked. In `docker-build.yml`''s `test` and `migrations` it is fail-OPEN: those are the other two required contexts on `main`, so the check reports green having done no work. So in those two jobs every `run:` step that is not `continue-on-error: true` calls `"$GITHUB_WORKSPACE/scripts/ci-step-ran.sh" mark <key>` as its FIRST act, and the job''s LAST step calls `ci-step-ran.sh assert --always <keys> --gated <keys>`, which fails the job when an expected key was never recorded. PER STEP, not per job: a marker written by the first step only proves the job started, while the drop that costs something is `Test` or the migration replay. The guard carries NO `if:` — the default `success()` is the wanted condition, because a genuine failure in an early step legitimately skips every later one and an `always()` guard would announce a false "these steps never executed" on every ordinary red build; the invariant that makes the omission safe is that the guard is skipped only when an earlier step FAILED, which already fails the job, so guard-skipped implies job-red and every path to a green job runs the guard. Separately and independently, no `${{` OPENER may appear in any `run:` body of those two jobs OR of `build` — the drop mechanism requires the opener, so banning it makes the class unreachable rather than merely caught, and an UNCLOSED opener triggers the same rewrite as a well-formed pair. Pass values in through the step''s `env:`, which is interpolated per value. The two halves have DIFFERENT scopes on purpose: markers cover the required pair, while the ban also covers `build`, whose `Smoke + IPTV E2E` step runs AFTER the image is pushed, so a drop there publishes a release candidate that was never booted and that `DeployStack jazz-media` then promotes. `functional-e2e` is delimiter-free but deliberately excluded (advisory by declaration), and `api-docs`/`format` keep one `github.base_ref` each and gate nothing that ships. The ban is enforced on the RELEASE PATH itself, not only in review (#767): a `scan` job runs the PyYAML-based ban test and `build` lists it in `needs:`, so a delimiter means `build` never runs and no image is published. A guard STEP inside `build` was tried first and is wrong — a step cannot protect the job it publishes from, and "my body has no opener so I cannot be dropped" is circular when only the PR-only test enforces that. The pytest in `script-tests` remains, but it is `on: pull_request` and not a required context, so it alone left the tag path unchecked.'
|
||||
signals: 'required check green but no work done, step never ran but job green, Build & test green in seconds, EF migration integrity green without replaying, missing Run Main step marker, Unable to interpolate expression format(, dropped step docker-build, ci-step-ran.sh, marker file, expression delimiter in a required job · paths: `.gitea/workflows/docker-build.yml`, `scripts/ci-step-ran.sh`, `scripts/tests/test_ci_dropped_step_guard.py`, `scripts/tests/test_ci_release_path_scan_job.py` · issues: #756, #751, #684, #767'
|
||||
mechanics: '`scripts/ci-step-ran.sh` owns the marker path so it exists ONCE and the write and the read cannot diverge. It is keyed on `GITHUB_JOB`/`GITHUB_RUN_ID` — REQUIRED, refusing rather than falling back to a reusable name — plus `GITHUB_RUN_ATTEMPT`. All three REFUSE rather than falling back to a reusable name. The third was warn-and-default until its presence was measured: grepping a log for the variable NAME proves nothing, and inferring it from the absence of a stderr warning proves nothing either (stderr capture was itself unestablished), so `assert` was made to print `Marker identity: job=… run=… attempt=… (from the runner)` on STDOUT and the answer was read off run 1916 for both required jobs. That line is retained as standing evidence. Do NOT justify the keying with #751''s "RUNNER_TEMP is /tmp, not a private per-job dir": that was measured on a job with no `container:` and does not transfer — these jobs get a fresh container, which is the primary protection, and the keying is defence in depth. Held by `scripts/tests/test_ci_dropped_step_guard.py`: static (marker set derived from the workflow equals the guard''s expectations, bucket matches each step''s `if:`, guard is last / has no `if:` / is not advisory / has no delimiter) and behavioural (the guard''s real command line executed against markers written by the steps'' real marker lines, dropping each key in turn). The release-path `scan` job (#767) runs the existing PyYAML-based ban test rather than a second implementation, so there is no drift surface; `scripts/tests/test_ci_release_path_scan_job.py` holds the WIRING instead — that `build` needs it, that it carries no job-level `if:` (one excluding the tag push restores the hole, one skipping the job skips `build` too), that no step is advisory, that it actually invokes the ban test, and that its own run bodies are delimiter-free. Its steps carry markers and a trailing assert of their own, verified by the same drop-each-key-in-turn behavioural pattern.'
|
||||
---
|
||||
|
||||
**Why per step, when #756 proposed per job.** A job-start marker answers "did this job begin", which
|
||||
@@ -75,12 +75,42 @@ ban. The claim not to repeat is the draft's dichotomy ("give up interpolation or
|
||||
remains uncovered is `api-docs` and `format`, whose one `github.base_ref` each sits in a detect step
|
||||
that gates nothing that ships, and `functional-e2e`, which is advisory by declaration.
|
||||
|
||||
**And the `build` ban is review-time, not fail-closed — do not read it as more.** It is enforced by
|
||||
`script-tests`, which is `on: pull_request` and is NOT a required context, so nothing re-checks it
|
||||
when a release is actually cut and `build` carries no markers or guard. A delimiter that reached
|
||||
`main` would still drop `Smoke` on the tag build and report green. Closing that needs a third
|
||||
bucket modelling `Smoke`'s publish-ref condition — deferred to ersatztv#767 rather than shipped
|
||||
unmeasured onto the release path, since every other claim in this record is backed by a live probe.
|
||||
Do NOT repeat the cost estimate an earlier draft gave for it ("would require pushing a real `v*`
|
||||
tag"): `build` runs on every push to `main` and `Smoke`'s ref condition admits `refs/heads/main`, so
|
||||
both directions are measurable without cutting a release.
|
||||
**The `build` ban is now fail-closed on the release path (ersatztv#767 — this was the open
|
||||
residual).** It used to be enforced only by `script-tests`, which is `on: pull_request` and is not a
|
||||
required context, so nothing re-checked it when a release was actually cut: a delimiter that reached
|
||||
`main` would still drop `Smoke` on the tag build and report green. A `scan` job now runs the
|
||||
PyYAML-based ban test and `build` lists it in `needs:`, so a delimiter means `build` never runs and
|
||||
no image is published.
|
||||
|
||||
**Two designs were tried, and the first one's failures are the reusable part.** The first put a
|
||||
bespoke stdlib scanner in `build` itself as an unconditional step before `Build and push`. Two
|
||||
independent reviews rejected it on two counts, both easy to re-invent:
|
||||
|
||||
- **A guard step cannot protect the job it lives in.** `build` publishes, so a guard step there is
|
||||
fail-OPEN if the runner drops it. The defence offered — "the guard's own body has no opener, so it
|
||||
cannot be dropped" — is circular, because the only thing enforcing that property was the same
|
||||
PR-only, non-required test being backstopped. A `needs:` edge is not circular: a red job skips its
|
||||
dependents by construction.
|
||||
- **A hand-written parser was strictly weaker than the check it backstopped.** It hand-parsed YAML to
|
||||
avoid provisioning PyYAML on `build`'s bare runner, and review found ~10 false NEGATIVES in one
|
||||
round (flow mappings, a quoted `"run":` key, aliases, multiline quoted scalars). For a security
|
||||
gate only false negatives matter, so this was worse than useless — it looked like enforcement. Do
|
||||
not re-attempt a bespoke scanner to save provisioning a dependency; run the real test.
|
||||
|
||||
**Why this needs no third marker bucket.** The deferral assumed the answer had to be markers on
|
||||
`build`, requiring a bucket that models `Smoke`'s publish-ref `if:`. It does not: the delimiter class
|
||||
is a *static* property of the workflow text, so a job that reads the text catches it without
|
||||
modelling any `if:`. The marker buckets are unchanged. Per-step markers on `build` remain a genuine
|
||||
smaller residual — they would catch a drop caused by something other than a delimiter.
|
||||
|
||||
**What this does not claim.** That no step can ever fail to run for another reason. The `scan` job's
|
||||
own steps carry markers and a trailing assert, which moves the terminal assumption rather than
|
||||
removing it: to fail open you must now drop the pytest step AND the assert step, not either alone.
|
||||
|
||||
**Measured, not assumed.** See the closing record on ersatztv#767 for the run ids of the poisoned and
|
||||
control dispatches. The arrangement: a `workflow_dispatch` on a scratch branch whose `Smoke` body
|
||||
carries a deliberate delimiter must redden `scan` and leave `build` skipped, and the same dispatch
|
||||
without the poison must pass. Note that "no image was published" is NOT part of the evidence — on a
|
||||
scratch ref `Build and push` has `push: false` regardless, so that conjunct could not have come out
|
||||
the other way; the discriminating observation is `scan` red and `build` skipped. Do NOT repeat the
|
||||
cost estimate an earlier draft gave ("would require pushing a real `v*` tag").
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
"""The `scan` job — the delimiter ban made fail-CLOSED on the release path (ersatztv#767).
|
||||
|
||||
WHAT THIS IS PROTECTING. #756 brought `build` into the delimiter ban, because a dropped
|
||||
`Smoke + IPTV E2E` publishes a release candidate that was never booted and reports the job green.
|
||||
But the ban was enforced ONLY by `test_the_delimiter_banned_jobs_have_NO_expression_delimiter_in_any_run_body`
|
||||
in `script-tests` — `on: pull_request`, not a required context. Nothing re-checked it on a `v*` tag
|
||||
push, which is exactly when the candidate is published.
|
||||
|
||||
WHY A JOB AND NOT A STEP IN `build`, and why this file is structural. The first cut of #767 put a
|
||||
bespoke stdlib scanner in `build` itself. Two independent reviews killed it on two counts, and both
|
||||
are worth keeping written down because both are easy to re-invent:
|
||||
|
||||
* A guard step inside `build` cannot protect `build`. If the runner drops it, the job carries on
|
||||
and publishes — fail-OPEN. The defence offered was "the guard's own body has no opener, so it
|
||||
cannot be dropped", but the only thing enforcing THAT was the same PR-only test being
|
||||
backstopped. Circular. As a `needs:` of `build`, a red here means `build` never runs at all.
|
||||
* The bespoke scanner hand-parsed YAML (to avoid provisioning PyYAML on `build`'s bare runner) and
|
||||
had ~10 false NEGATIVES within one review round — flow mappings, a quoted `"run":` key, aliases,
|
||||
multiline quoted scalars. It was strictly WEAKER than the check it backstopped, in the only
|
||||
direction that matters. The fix was to delete it and run the real PyYAML-based test, which needs
|
||||
no second definition of "what is a `run:` body" and so has no drift surface.
|
||||
|
||||
So the detection logic is not retested here — it lives in `test_ci_dropped_step_guard.py` and this
|
||||
job runs that file. What this file holds is the WIRING, which is what makes the ban fail-closed:
|
||||
the job exists, `build` depends on it, nothing can skip it, its own steps cannot be silently
|
||||
dropped, and it actually invokes the ban test.
|
||||
|
||||
WHAT THIS DOES NOT CLAIM. That no step can ever fail to run for a reason other than the
|
||||
interpolation drop. This job's own steps carry #756 markers and a trailing assert, so the regress
|
||||
terminates where the sibling guards' does — to fail open you must now drop the pytest step AND the
|
||||
assert step, not either one. The end-to-end behaviour (a poisoned `Smoke` body reddens `scan` and
|
||||
`build` never runs) is a LIVE measurement recorded on the issue, not something a static test here
|
||||
can establish.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
WORKFLOW = REPO_ROOT / ".gitea" / "workflows" / "docker-build.yml"
|
||||
SCRIPT = REPO_ROOT / "scripts" / "ci-step-ran.sh"
|
||||
|
||||
_DOC = yaml.safe_load(WORKFLOW.read_text())
|
||||
_OPENER = re.compile(r"\$\{\{")
|
||||
_MARK = re.compile(r'ci-step-ran\.sh"?\s+mark\s+(\S+)')
|
||||
|
||||
JOB = "scan"
|
||||
BAN_TEST_FILE = "scripts/tests/test_ci_dropped_step_guard.py"
|
||||
|
||||
|
||||
def _job():
|
||||
assert JOB in _DOC["jobs"], f"the `{JOB}` job is gone — the release path is unguarded again"
|
||||
return _DOC["jobs"][JOB]
|
||||
|
||||
|
||||
def _steps():
|
||||
return _job()["steps"]
|
||||
|
||||
|
||||
def _run_steps():
|
||||
return [s for s in _steps() if s.get("run")]
|
||||
|
||||
|
||||
def _guard():
|
||||
"""The trailing assert step, located by CONTENT — never by index, so that
|
||||
`test_the_guard_is_the_LAST_step` is not true by construction."""
|
||||
hits = [s for s in _run_steps() if "ci-step-ran.sh assert" in s["run"]]
|
||||
assert len(hits) == 1, f"expected exactly 1 assert step in `{JOB}`, found {len(hits)}"
|
||||
return hits[0]
|
||||
|
||||
|
||||
def _marked():
|
||||
out = []
|
||||
for s in _run_steps():
|
||||
m = _MARK.search(s["run"])
|
||||
if m:
|
||||
out.append((s, m.group(1)))
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# WIRING — the properties that make the ban fail-closed
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_build_DEPENDS_on_the_scan_job():
|
||||
"""This single edge is the whole fail-closed property.
|
||||
|
||||
Without it the scan is advisory: it could go red while `build` publishes anyway.
|
||||
"""
|
||||
needs = _DOC["jobs"]["build"]["needs"]
|
||||
needs = [needs] if isinstance(needs, str) else needs
|
||||
assert JOB in needs, f"`build` no longer needs `{JOB}` — a red scan would not stop a release"
|
||||
|
||||
|
||||
def test_the_scan_job_has_NO_job_level_if():
|
||||
"""Two failure modes at once, in opposite directions.
|
||||
|
||||
An `if:` that excludes the tag push would leave the release path unguarded — the exact hole
|
||||
#767 closed. An `if:` that skipped it for any other reason would SKIP `build` too (a skipped
|
||||
dependency skips its dependents), breaking every release. Neither is wanted: it always runs.
|
||||
"""
|
||||
job = _job()
|
||||
assert "if" not in job, f"`{JOB}` must carry no job-level `if:`, found {job.get('if')!r}"
|
||||
|
||||
|
||||
def test_the_scan_job_actually_invokes_the_ban_test():
|
||||
"""Otherwise the job is an expensive no-op that reports green.
|
||||
|
||||
Asserted against the file path the ban test really lives in, so renaming that file without
|
||||
updating the workflow is a red here rather than a silently unguarded release path.
|
||||
"""
|
||||
assert (REPO_ROOT / BAN_TEST_FILE).is_file()
|
||||
assert any(BAN_TEST_FILE in s["run"] for s in _run_steps()), (
|
||||
f"no step in `{JOB}` runs {BAN_TEST_FILE}"
|
||||
)
|
||||
|
||||
|
||||
def test_no_step_in_the_scan_job_is_advisory():
|
||||
"""`continue-on-error: true` would make the whole gate a no-op while every other test here
|
||||
stayed green — it is the cheapest way to accidentally disarm this."""
|
||||
offenders = [s.get("name") for s in _steps() if s.get("continue-on-error")]
|
||||
assert not offenders, f"advisory step(s) in `{JOB}`: {offenders}"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"step_name",
|
||||
[s.get("name", "?") for s in yaml.safe_load(WORKFLOW.read_text())["jobs"][JOB]["steps"]
|
||||
if s.get("run")],
|
||||
)
|
||||
def test_every_run_body_in_the_scan_job_is_delimiter_free(step_name):
|
||||
"""The guard must not be vulnerable to the defect it guards against.
|
||||
|
||||
Not a proof that it always runs — a construction argument about ONE mechanism, the same axiom
|
||||
the sibling guards rest on. It is asserted per step so a failure names which step regressed.
|
||||
"""
|
||||
step = next(s for s in _run_steps() if s.get("name", "?") == step_name)
|
||||
assert not _OPENER.search(step["run"]), (
|
||||
f"step {step_name!r} of `{JOB}` contains an expression delimiter; the runner would rewrite "
|
||||
"the whole body and DROP the step while reporting success (ersatztv#751). Pass values "
|
||||
"through `env:`, which is interpolated per value."
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# THE JOB'S OWN DROPPED-STEP GUARD
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_every_consequential_step_marks_itself():
|
||||
"""Every `run:` step except the guard records that it executed."""
|
||||
marked = {s.get("name") for s, _ in _marked()}
|
||||
expected = {s.get("name") for s in _run_steps() if s is not _guard()}
|
||||
assert marked == expected, f"unmarked step(s) in `{JOB}`: {expected - marked}"
|
||||
|
||||
|
||||
def test_the_guard_expectations_match_the_markers_exactly():
|
||||
"""The set the guard waits for IS the set the steps write — derived from the workflow, not
|
||||
restated here, so adding a step without a marker is a red."""
|
||||
argv = _guard()["run"].split()
|
||||
assert "--always" in argv, argv
|
||||
always = argv[argv.index("--always") + 1:]
|
||||
assert "--gated" not in argv, "every step in this job is unconditional; there is nothing to gate"
|
||||
assert sorted(always) == sorted(k for _, k in _marked())
|
||||
|
||||
|
||||
def test_the_guard_is_the_LAST_step():
|
||||
assert _steps()[-1] is _guard(), "the assert must run after the steps it checks"
|
||||
|
||||
|
||||
def test_the_guard_has_no_if():
|
||||
"""Same reasoning as the sibling guards: the default `success()` is wanted, because a genuine
|
||||
early failure legitimately skips later steps and already fails the job."""
|
||||
assert "if" not in _guard()
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# BEHAVIOURAL — the guard's REAL command line, against the steps' REAL marker lines
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _mark_line(step) -> str:
|
||||
"""The step's own marker line, verbatim from the workflow — never rebuilt in Python, so a
|
||||
drift between the workflow and the script cannot hide behind a test that composed its own."""
|
||||
return next(ln for ln in step["run"].splitlines() if _MARK.search(ln)).strip()
|
||||
|
||||
|
||||
def _env(tmp_path, **extra):
|
||||
env = {
|
||||
"PATH": os.environ["PATH"],
|
||||
"GITHUB_WORKSPACE": str(REPO_ROOT),
|
||||
"RUNNER_TEMP": str(tmp_path),
|
||||
"GITHUB_JOB": JOB,
|
||||
"GITHUB_RUN_ID": "424242",
|
||||
"GITHUB_RUN_ATTEMPT": "7",
|
||||
}
|
||||
env.update(extra)
|
||||
return {k: v for k, v in env.items() if v is not None}
|
||||
|
||||
|
||||
def _run(script: str, env):
|
||||
return subprocess.run(["bash", "-c", script], cwd=REPO_ROOT, env=env,
|
||||
capture_output=True, text=True)
|
||||
|
||||
|
||||
def test_the_guard_PASSES_when_every_step_ran(tmp_path):
|
||||
env = _env(tmp_path)
|
||||
for step, _ in _marked():
|
||||
assert _run(_mark_line(step), env).returncode == 0
|
||||
res = _run(_guard()["run"], env)
|
||||
assert res.returncode == 0, res.stderr
|
||||
|
||||
|
||||
@pytest.mark.parametrize("dropped", [k for _, k in _marked()])
|
||||
def test_the_guard_FAILS_when_a_step_was_dropped(tmp_path, dropped):
|
||||
"""The positive control. Drop each key in turn — the guard must go red and NAME it.
|
||||
|
||||
A guard only ever exercised on the happy path is indistinguishable from one that passes
|
||||
unconditionally, which is the failure this whole mechanism exists to remove.
|
||||
"""
|
||||
env = _env(tmp_path)
|
||||
for step, key in _marked():
|
||||
if key != dropped:
|
||||
assert _run(_mark_line(step), env).returncode == 0
|
||||
res = _run(_guard()["run"], env)
|
||||
assert res.returncode != 0, f"guard passed despite '{dropped}' never running: {res.stdout}"
|
||||
# BOTH streams: the script's `::error::` lands on stdout here while other diagnostics go to
|
||||
# stderr, and a test that picked the wrong one would assert on an empty string and pass for the
|
||||
# wrong reason on any message change.
|
||||
assert dropped in (res.stdout + res.stderr), (res.stdout, res.stderr)
|
||||
|
||||
|
||||
def test_the_guard_REFUSES_to_pass_with_no_expectations(tmp_path):
|
||||
"""`assert` with an empty expectation set would report success having checked nothing."""
|
||||
res = _run(f"{SCRIPT} assert --always", _env(tmp_path))
|
||||
assert res.returncode != 0
|
||||
Reference in New Issue
Block a user