diff --git a/docs/ci-cd.md b/docs/ci-cd.md index 0bbb5b0eb..176281373 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -9,7 +9,7 @@ server-management#481. | Host | IP | Role | |---|---|---| -| **jazz** | 192.168.1.29 | **Docker host for the media transcoders** — prod `ersatztv` (8409), `ersatztv-test` (8410), Jellyfin. Release scans (`security-scan.sh`) and the prod-copy `migration-smoke.sh` run here. | +| **jazz** | 192.168.1.29 | **Docker host for the media transcoders** — prod `ersatztv` (8409), `ersatztv-test` (8410), Jellyfin. The prod-copy `migration-smoke.sh` runs here. The release scan (`security-scan.sh`) boots its own throwaway target, so it needs a docker host that meets its requirements (Security scanning → Where/when), not this one. | | **bumblebee** | 192.168.1.99 | CI runners (`bumblebee-runner`, `small-runner`), plus every other Docker stack. All the memory/lane measurements below were taken here. | | **ci-runner** | VM 127 (pve4) | The other `ubuntu-latest` CI runner. | @@ -116,8 +116,12 @@ commit in the range (see the `decisions.md` header) — routine lifecycle metada > protection is never left off) — and record the outcome on `release.tag-protection-v-star`, > which is waiting for it. 2. Wait for tag CI to build `:prod` + the immutable `:` + `:` images. Run - `scripts/security-scan.sh` on jazz against **the immutable `:` image**, not a - moving tag, and triage every ZAP/semgrep finding. + `scripts/security-scan.sh` against **the immutable `:` image**, not a moving tag, from a + checkout of the tagged commit on a docker host whose non-interactive `PATH` has semgrep: check with + `ssh command -v semgrep` first (bumblebee passed that check on 2026-10-27, + server-management#1072). A host without semgrep fails the gate with exit 2; the only alternative is + `ETV_SCAN_SKIP_SEMGREP=1` plus a separate semgrep run against the tagged commit, recorded as such. + Triage every ZAP/semgrep finding. Requirements: Security scanning → Where/when. 3. Only after the candidate passes, manually `DeployStack jazz-media` and observe its pre-deploy output. Prod's compose deliberately follows floating `:prod` (Timothy's 2026-07-11 decision), so no CI push or pin bump is needed. @@ -2660,19 +2664,21 @@ running product from outside our C#/review stack. `/api/v1` operation, injecting `X-Api-Key` on every request via a ZAP replacer rule so it reaches the `[RequiresAuthentication]` + `RequireKeyForReads` surface (not just the `/app` shell an unauthenticated spider sees). Then a **semgrep** SAST cross-check (`p/security-audit` + `p/secrets` + `p/csharp`). The - container is torn down on exit. **The SAST half is required**: a host without - semgrep on `PATH` fails the script up front (exit 2, naming the tool) before the image is pulled, and the - only skip is the explicit opt-out `ETV_SCAN_SKIP_SEMGREP=1` (DAST only; a gate run that sets it has not - run the SAST half, so run semgrep separately and say so in the release record). A semgrep that is installed but exits non-zero (bad config, - registry unreachable) also fails the script, since findings alone exit 0. `--selftest` pins the - missing, opted-out, present and failing cases through the script's own exit status (ersatztv#931). A non-interactive ssh shell needs semgrep on - its `PATH` (`~/.zshenv`, not `.zshrc`). -- **Where/when.** Runs on a **docker host** with registry access (bumblebee, which has semgrep on its non-interactive `PATH`; a host without semgrep fails the gate, so jazz needs it installed first), like - `migration-smoke.sh`: `scripts/security-scan.sh [IMAGE] [PORT]` (defaults `…:latest` / `8411`). It is a - **manual release-gate**, deliberately **not** a per-PR CI job — it needs docker + a booted image, takes - several minutes, and is noisy (expect to tune, not take raw). The continuous layer is the per-PR - white-box gates + the weekly `dependency-scan`; this is the per-release black-box pass. Re-run it each - release and before any change to the exposure posture. + container is torn down on exit. **The SAST half is required** (ersatztv#931): with no semgrep on `PATH` + the script exits 2 naming the tool, before the image is pulled. The only skip is the explicit opt-out + `ETV_SCAN_SKIP_SEMGREP=1` (any other value, `0` included, does not skip). A run that sets it is DAST + only and has not passed the SAST half: run semgrep separately and say so in the release record. A + semgrep that is installed but exits non-zero (bad config, registry unreachable) also fails the script, + since findings alone exit 0. +- **Where/when.** Runs on a **docker host** with registry access, from a checkout of the commit under test + (semgrep scans the checkout the script lives in), whose **non-interactive** `PATH` has semgrep: the + `PATH` that `ssh command -v semgrep` sees, which for zsh comes from `~/.zshenv`, not `.zshrc`. A + host without it fails the gate by construction; the alternative is the opt-out plus a separate SAST + run. Like `migration-smoke.sh`: `scripts/security-scan.sh [IMAGE] [PORT]` (defaults `…:latest` / + `8411`). It is a **manual release-gate**, deliberately **not** a per-PR CI job — it needs docker + a + booted image, takes several minutes, and is noisy (expect to tune, not take raw). The continuous layer + is the per-PR white-box gates + the weekly `dependency-scan`; this is the per-release black-box pass. + Re-run it each release and before any change to the exposure posture. - **Exit-code contract (ersatztv#338).** `zap-api-scan.py`'s raw exit code is NOT a simple pass/fail — it conflates a clean run with a warnings-only run unless you know its wrapper contract: **0** clean (no FAIL or WARN alerts), **2** WARN-only (triage required, but **not** release-blocking), **1** FAIL (at @@ -2680,14 +2686,19 @@ running product from outside our C#/review stack. hung post-scan cleanup (the report written before the hang is still usable — triage it), any other code means the scanner/tool itself errored (not a scan result at all). `scripts/security-scan.sh` encodes this in `classify_zap_exit()` and prints an unambiguous `==> ZAP result: ...` line; the script's own exit status reflects that classification (0 for - clean/WARN, 1 for FAIL/timeout/tool-error) rather than ZAP's raw code, so a warnings-only run no longer - reads as a failed scan. Found when the v26.8.0 release scan (#335) returned raw exit 2 for a report with - `FAIL-NEW: 0` and two known/expected warning classes — the shell result looked like a failure though the - release gate had actually passed. Run `scripts/security-scan.sh --selftest` for a docker-free regression - check of the classification logic. - **The script's own exit 2 is not ZAP's WARN**: it means the SAST half did not run (semgrep missing - before the scan, or present and erroring after it) and is release-blocking. + TIMEOUT|TOOL ERROR> ...` line. The script's **own** exit is not ZAP's raw code, so a warnings-only run + no longer reads as a failed scan. Found when the v26.8.0 release scan (#335) returned raw exit 2 for a + report with `FAIL-NEW: 0` and two known/expected warning classes — the shell result looked like a + failure though the release gate had actually passed. Every non-zero script exit is release-blocking: + **0** ZAP clean or WARN-only and the SAST half ran clean (or was opted out); **1** ZAP FAIL, timeout or + tool error, or the throwaway target never came up or its key was unreadable; **2** — *not* ZAP's WARN — + a scanner half did not run: semgrep missing (before the scan) or erroring (after it), or docker + missing; if ZAP already classified non-zero, that status stands and the semgrep error prints beside + it; **any other** code is a setup command (e.g. `docker pull`) failing under `set -e`. Run + `scripts/security-scan.sh --selftest` for a docker-free regression check: it pins every ZAP + classification, and runs the script itself with shimmed tools (ZAP reporting clean) to pin the + semgrep gate — missing → exit 2 before docker is checked, erroring → 2, succeeding → 0 with semgrep + run, opt-out → 0 with semgrep not run (`=0` still runs it). It exercises no real ZAP or semgrep. - **Triage.** Triage each WARN/FAIL finding false-positive vs real. Real, in-scope, go-live-blocking findings get fixed (e.g. the security headers from the #319 baseline; the Microsoft.OpenApi pin above); LAN-expected noise (Private-IP disclosure) is revisited only for genuine remote exposure. nuclei