From 940af13a5fb690be7bf0f4a2ce81ce12e200d306 Mon Sep 17 00:00:00 2001 From: Timothy Date: Mon, 13 Jul 2026 08:09:30 +0200 Subject: [PATCH 1/3] =?UTF-8?q?fix(deps):=20#314=20=E2=80=94=20pin=20Micro?= =?UTF-8?q?soft.OpenApi=202.7.5=20(GHSA-v5pm-xwqc-g5wc,=20High)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Microsoft.AspNetCore.OpenApi 10.0.2 + Scalar.AspNetCore pull Microsoft.OpenApi 2.0.0 transitively — High severity (stack overflow parsing a circular $ref; fixed in 2.7.5). Direct-pin 2.7.5 in CPM + a direct ErsatzTV PackageReference so the override actually resolves (the #8 SQLitePCLRaw transitive-override pattern). Build clean (transformers compile unchanged against 2.7.5), v1.json byte-identical (no OpenAPI-output change), ErsatzTV + ErsatzTV.Tests both vuln-clear, 198 OpenAPI/contract/security tests green. refs #314 #8 Co-Authored-By: Claude Opus 4.8 (1M context) --- Directory.Packages.props | 4 ++++ ErsatzTV/ErsatzTV.csproj | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/Directory.Packages.props b/Directory.Packages.props index 59e56e359..436d5cfa7 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -35,6 +35,10 @@ + + diff --git a/ErsatzTV/ErsatzTV.csproj b/ErsatzTV/ErsatzTV.csproj index a3a4d2c18..c94ffb3de 100644 --- a/ErsatzTV/ErsatzTV.csproj +++ b/ErsatzTV/ErsatzTV.csproj @@ -37,6 +37,10 @@ + + all -- 2.47.3 From 366d5d316ce860703d9f7ff1ff3d295743827ea0 Mon Sep 17 00:00:00 2001 From: Timothy Date: Mon, 13 Jul 2026 08:28:13 +0200 Subject: [PATCH 2/3] =?UTF-8?q?feat(ci):=20#314=20=E2=80=94=20repeatable?= =?UTF-8?q?=20black-box=20security=20scan=20(authenticated=20ZAP=20+=20sem?= =?UTF-8?q?grep)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/security-scan.sh: boots a THROWAWAY container from the image under test (fresh config volume; never the deployed prod/test container — the active scan attacks write endpoints), reads its machine key, and runs an authenticated OWASP ZAP api-scan that imports /openapi/v1.json (all 160 /api/v1 ops) and injects X-Api-Key on every request via a replacer rule so it reaches the [RequiresAuthentication] + RequireKeyForReads surface — then a semgrep SAST cross-check. Wrapped in `timeout` because zap-api-scan can hang in post-scan cleanup after the report is already written. docs/ci-cd.md: new 'Security scanning' section (out-of-ecosystem black-box gate, run on the docker host per-release like migration-smoke, not a per-PR CI job) + the Microsoft.OpenApi 2.7.5 pin note in dependency management. refs #314 #197 Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/ci-cd.md | 36 +++++++++++++- scripts/security-scan.sh | 100 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 2 deletions(-) create mode 100755 scripts/security-scan.sh diff --git a/docs/ci-cd.md b/docs/ci-cd.md index 2fb86c248..3aff5bee5 100644 --- a/docs/ci-cd.md +++ b/docs/ci-cd.md @@ -167,8 +167,11 @@ opt-out (`ManagePackageVersionsCentrally=false`). (ersatztv#14) `Directory.Build.props` demotes low/moderate/high advisories (NU1901-1903) to warnings and promotes NU1904 (critical) to an error in **every** project via `WarningsAsErrors`. The advisories that prompted this were resolved in ersatztv#8 (NCalcSync→6.x; SQLitePCLRaw -bundle 3.x). The NU1901-1903 demotion is **kept by design**: criticals (NU1904) still hard- -block, while low/moderate/high advisories surface as warnings + via the weekly scan and +bundle 3.x) and ersatztv#314 (**Microsoft.OpenApi 2.0.0→2.7.5**, GHSA-v5pm-xwqc-g5wc High — +direct-pinned in `ErsatzTV.csproj` over the 2.0.0 that `Microsoft.AspNetCore.OpenApi` + +`Scalar.AspNetCore` pull transitively; the SQLitePCLRaw override pattern; regenerates the +OpenAPI doc byte-identically). The NU1901-1903 demotion is **kept by design**: criticals (NU1904) +still hard-block, while low/moderate/high advisories surface as warnings + via the weekly scan and Renovate security PRs, rather than breaking unrelated PRs the moment a new transitive advisory drops. @@ -206,6 +209,35 @@ Core `[9.0.x,10)`) are respected — no v10 jump. PR volume is throttled (`prCon to drain faster. `workflow_dispatch` defaults to a safe **dry run**. Cross-repo rollout tracked in server-management#484. (server-management#484) +## Security scanning — black-box DAST + SAST (`scripts/security-scan.sh`, ersatztv#314) + +Every other security check we run is **in-ecosystem / white-box** — SonarAnalyzer, NetArchTest, the +adversarial fork + Codex review passes, the api-docs/format/decisions CI gates, `dotnet list package +--vulnerable` — so they share our blind spots. `scripts/security-scan.sh` is the **out-of-ecosystem, +black-box** complement and a **#197 exit criterion** (HARD GATE before remote exposure): it drives the +running product from outside our C#/review stack. + +- **What it does.** Boots a **throwaway** container from the image under test (fresh empty config volume; + never the deployed prod/test container — the authenticated *active* scan sends attack payloads to write + endpoints), reads the generated machine key, and runs an **authenticated OWASP ZAP** API scan + (`zap-api-scan.py`) that imports the static `/openapi/v1.json` so it exercises **every** declared + `/api/v1` operation, injecting `X-Api-Key` on every request via a ZAP replacer rule so it reaches the + `[RequiresAuthentication]` + `RequireKeyForReads` surface (not just the `/app` shell an unauthenticated + spider sees). Then a **semgrep** SAST cross-check (`p/security-audit` + `p/secrets` + `p/csharp`). The + container is torn down on exit. +- **Where/when.** Runs on the **docker host** (bumblebee — the Mac has no docker), like + `migration-smoke.sh`: `scripts/security-scan.sh [IMAGE] [PORT]` (defaults `…:latest` / `8411`). It is a + **manual release-gate**, deliberately **not** a per-PR CI job — it needs docker + a booted image, takes + several minutes, and is noisy (expect to tune, not take raw). The continuous layer is the per-PR + white-box gates + the weekly `dependency-scan`; this is the per-release black-box pass. Re-run it each + release and before any change to the exposure posture. +- **Triage.** ZAP exits non-zero on any FAIL-level alert; the tooling is noisy, so triage each finding + false-positive vs real. Real, in-scope, go-live-blocking findings get fixed (e.g. the security headers + from the #319 baseline; the Microsoft.OpenApi pin above); LAN-expected noise (Private-IP disclosure) is + revisited only for genuine remote exposure. nuclei (template-based CVE fingerprinting) is an optional + third pass — deferred while its template fetch is blocked in the runner env (pre-seed a template volume + to add it); ZAP covers the DAST baseline and semgrep the SAST, so it is not on the critical path. + ## Static analysis & formatting **Analyzer packs** — `Directory.Build.targets` references **Roslynator**, **SonarAnalyzer.CSharp**, diff --git a/scripts/security-scan.sh b/scripts/security-scan.sh new file mode 100755 index 000000000..0e71b6a6d --- /dev/null +++ b/scripts/security-scan.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# scripts/security-scan.sh — out-of-ecosystem, black-box security scan of a release image (ersatztv#314). +# +# WHY: every OTHER security check we run is in-ecosystem / white-box (SonarAnalyzer, NetArchTest, the +# adversarial fork + Codex review passes, the api-docs/format/decisions CI gates, `dotnet list package +# --vulnerable`). They reason about our source the way we do, so they share our blind spots. This drives +# the RUNNING product from outside our C#/review stack — OWASP ZAP as an authenticated DAST against every +# `/api/v1` endpoint, plus semgrep as a SAST cross-check — to catch the classes structural review misses: +# missing/incorrect security headers, an endpoint that forgot its auth attribute, verb tampering, +# injection reflections, known-CVE fingerprints. It is a #197 exit criterion — a HARD GATE before remote +# exposure — and should be re-run each release (see docs/ci-cd.md → Security scanning). +# +# It runs on the docker host (bumblebee) against a THROWAWAY container booted from the image under test — +# NEVER the deployed prod (or test) container: the authenticated active scan sends attack payloads to +# write endpoints (creating/deleting rows, triggering playout builds), so it must target a disposable +# instance with a fresh empty config volume. The container is torn down on exit. +# +# Usage: +# scripts/security-scan.sh [IMAGE] [PORT] +# IMAGE container image to scan (default: 192.168.1.95:3000/timothy/ersatztv:latest) +# PORT host port for the throwaway app (default: 8411) +# Env: +# ETV_SCAN_OUT host dir for reports (default: /tmp/etv-scan-out) +# ETV_SCAN_SKIP_SEMGREP=1 skip the SAST pass (DAST only) +# +# Reports land in $ETV_SCAN_OUT: zap-api-report.{html,md,json}. Exit code is ZAP's: +# 0 = no FAIL-level alerts, non-zero = at least one FAIL (triage it — this tooling is noisy; expect to +# tune the ignore list in .zap/ rather than take the raw report at face value). + +set -euo pipefail + +IMAGE="${1:-192.168.1.95:3000/timothy/ersatztv:latest}" +PORT="${2:-8411}" +OUT="${ETV_SCAN_OUT:-/tmp/etv-scan-out}" +TARGET_NAME="etv-secscan-target" +ZAP_IMAGE="ghcr.io/zaproxy/zaproxy:stable" + +command -v docker >/dev/null || { echo "error: docker not found — run this on the docker host (bumblebee)" >&2; exit 2; } + +CFG="$(mktemp -d /tmp/etv-secscan-cfg-XXXXXX)" +cleanup() { + docker rm -f "$TARGET_NAME" >/dev/null 2>&1 || true + rm -rf "$CFG" 2>/dev/null || true +} +trap cleanup EXIT + +echo "==> pulling $IMAGE" +docker pull "$IMAGE" >/dev/null + +echo "==> booting throwaway scan target ($TARGET_NAME) on :$PORT with a fresh config volume" +docker rm -f "$TARGET_NAME" >/dev/null 2>&1 || true +docker run -d --name "$TARGET_NAME" -p "$PORT:8409" -v "$CFG":/config "$IMAGE" >/dev/null + +echo -n "==> waiting for the app to serve /app/ " +ready=0 +for _ in $(seq 1 60); do + if [ "$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:$PORT/app/" 2>/dev/null)" = "200" ]; then + ready=1; echo "ready"; break + fi + echo -n "."; sleep 3 +done +[ "$ready" = 1 ] || { echo "FAILED: app never became ready"; docker logs "$TARGET_NAME" 2>&1 | tail -30; exit 1; } + +# The machine key is generated at /config/api.key (0600, root-owned on the host mount) — read it from +# inside the container. ZAP injects it as X-Api-Key on every request via a replacer rule so the scan +# reaches the authenticated (`RequiresAuthentication` + RequireKeyForReads) surface, not just the /app shell. +KEY="$(docker exec "$TARGET_NAME" cat /config/api.key 2>/dev/null)" +[ -n "$KEY" ] || { echo "FAILED: could not read /config/api.key"; exit 1; } +echo "==> got machine key (${#KEY} chars); it will be injected as X-Api-Key on every request" + +mkdir -p "$OUT"; chmod 777 "$OUT" + +# zap-api-scan.py imports the OpenAPI spec (served as a static file at /openapi/v1.json) so it exercises +# every declared /api/v1 operation — not just what a spider finds — then runs passive + active scanners. +# The reports are written when the scan finishes but BEFORE the wrapper exits; zap-api-scan has been seen +# to hang in post-scan cleanup (draining the passive queue / holding a connection to a streaming endpoint), +# so the whole run is wrapped in `timeout` — a hung wrapper is killed and the already-written report is +# still used. ETV_SCAN_TIMEOUT (default 45m) bounds it. +echo "==> running authenticated OWASP ZAP API scan (this can take several minutes)" +set +e +timeout "${ETV_SCAN_TIMEOUT:-45m}" docker run --rm --network host -v "$OUT":/zap/wrk:rw "$ZAP_IMAGE" \ + zap-api-scan.py -t "http://localhost:$PORT/openapi/v1.json" -f openapi \ + -r zap-api-report.html -w zap-api-report.md -J zap-api-report.json \ + -z "replacer.full_list(0).description=apikey;replacer.full_list(0).enabled=true;replacer.full_list(0).matchtype=REQ_HEADER;replacer.full_list(0).matchstr=X-Api-Key;replacer.full_list(0).regex=false;replacer.full_list(0).replacement=$KEY" +zap_rc=$? +[ "$zap_rc" = 124 ] && echo "NOTE: ZAP wrapper hit the ${ETV_SCAN_TIMEOUT:-45m} timeout (usually a post-scan cleanup hang) — the report written before the hang is still valid; triage it." +set -e +# A timed-out `docker run` leaves the ZAP container; reap any stragglers so a re-run is clean. +docker ps --filter ancestor="$ZAP_IMAGE" -q | xargs -r docker kill >/dev/null 2>&1 || true +echo "==> ZAP report: $OUT/zap-api-report.html (+ .md/.json). Exit code: $zap_rc" + +if [ "${ETV_SCAN_SKIP_SEMGREP:-0}" != "1" ] && command -v semgrep >/dev/null; then + echo "==> semgrep SAST cross-check (repo root)" + REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" + semgrep scan --config p/security-audit --config p/secrets --config p/csharp \ + --exclude web/node_modules --exclude design-system --metrics off --quiet \ + "$REPO_ROOT" || echo "(semgrep reported findings — triage in the report above)" +fi + +exit "$zap_rc" -- 2.47.3 From a042623a53cfc5cf24469ddbcd7dda61e208fe84 Mon Sep 17 00:00:00 2001 From: Timothy Date: Mon, 13 Jul 2026 08:36:08 +0200 Subject: [PATCH 3/3] =?UTF-8?q?fix(ci):=20#314=20review=20=E2=80=94=20reap?= =?UTF-8?q?=20ZAP=20container=20in=20the=20cleanup()=20trap;=20warn=20on?= =?UTF-8?q?=20missing=20semgrep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cold review of PR #331: - Move the ZAP-container reap from an inline post-scan line into cleanup() (the EXIT trap) so a SIGINT/timeout kill mid-scan can't leave it running. - semgrep absent now prints a stderr WARN instead of silently skipping the SAST pass (no false confidence for a release gate). refs #314 Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/security-scan.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/security-scan.sh b/scripts/security-scan.sh index 0e71b6a6d..013534d38 100755 --- a/scripts/security-scan.sh +++ b/scripts/security-scan.sh @@ -39,6 +39,9 @@ command -v docker >/dev/null || { echo "error: docker not found — run this on CFG="$(mktemp -d /tmp/etv-secscan-cfg-XXXXXX)" cleanup() { + # Reap the ZAP scanner container too — in the trap (not just inline after the scan) so an interrupt + # (SIGINT/SIGTERM) or a `timeout` kill mid-scan doesn't leave it running. + docker ps --filter ancestor="$ZAP_IMAGE" -q | xargs -r docker kill >/dev/null 2>&1 || true docker rm -f "$TARGET_NAME" >/dev/null 2>&1 || true rm -rf "$CFG" 2>/dev/null || true } @@ -85,11 +88,15 @@ timeout "${ETV_SCAN_TIMEOUT:-45m}" docker run --rm --network host -v "$OUT":/zap zap_rc=$? [ "$zap_rc" = 124 ] && echo "NOTE: ZAP wrapper hit the ${ETV_SCAN_TIMEOUT:-45m} timeout (usually a post-scan cleanup hang) — the report written before the hang is still valid; triage it." set -e -# A timed-out `docker run` leaves the ZAP container; reap any stragglers so a re-run is clean. -docker ps --filter ancestor="$ZAP_IMAGE" -q | xargs -r docker kill >/dev/null 2>&1 || true +# (The ZAP container + target + temp dir are reaped by cleanup() on the EXIT trap — every path, incl. +# a `timeout` kill or Ctrl-C mid-scan.) echo "==> ZAP report: $OUT/zap-api-report.html (+ .md/.json). Exit code: $zap_rc" -if [ "${ETV_SCAN_SKIP_SEMGREP:-0}" != "1" ] && command -v semgrep >/dev/null; then +if [ "${ETV_SCAN_SKIP_SEMGREP:-0}" = "1" ]; then + echo "==> semgrep SAST skipped (ETV_SCAN_SKIP_SEMGREP=1)" +elif ! command -v semgrep >/dev/null; then + echo "WARN: semgrep not installed — SAST cross-check SKIPPED (install semgrep for full coverage)" >&2 +else echo "==> semgrep SAST cross-check (repo root)" REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" semgrep scan --config p/security-audit --config p/secrets --config p/csharp \ -- 2.47.3