ci: distinguish ZAP warning exit 2 from FAIL in security-scan #338

Closed
opened 2026-07-13 20:36:26 +02:00 by timothy · 2 comments
Owner

Found during the v26.8.0 exact-image release scan for #335.

scripts/security-scan.sh and docs/ci-cd.md currently say a non-zero ZAP exit means a FAIL-level alert. In practice zap-api-scan.py returned 2 for a report with FAIL-NEW: 0 and two warning classes only (the known #330 CORP Low plus expected SPA fallback content types). The report was triaged and the release gate passed, but the shell result looks like a failed scan.

Done-when

  • Document the actual ZAP wrapper exit contract (0 clean, 1 FAIL, 2 WARN/triage, 3+ tool error, 124 wrapper timeout as applicable).
  • Make the script print an unambiguous final classification that distinguishes security FAIL from warnings requiring triage and scanner/tool failure.
  • Preserve the report-first timeout behavior and cleanup trap.
  • Add a lightweight regression check for the exit-classification logic.
  • Review passes.

Refs #314 #330 #335

Found during the v26.8.0 exact-image release scan for #335. `scripts/security-scan.sh` and `docs/ci-cd.md` currently say a non-zero ZAP exit means a FAIL-level alert. In practice `zap-api-scan.py` returned **2** for a report with `FAIL-NEW: 0` and two warning classes only (the known #330 CORP Low plus expected SPA fallback content types). The report was triaged and the release gate passed, but the shell result looks like a failed scan. ## Done-when - [ ] Document the actual ZAP wrapper exit contract (0 clean, 1 FAIL, 2 WARN/triage, 3+ tool error, 124 wrapper timeout as applicable). - [ ] Make the script print an unambiguous final classification that distinguishes security FAIL from warnings requiring triage and scanner/tool failure. - [ ] Preserve the report-first timeout behavior and cleanup trap. - [ ] Add a lightweight regression check for the exit-classification logic. - [ ] Review passes. Refs #314 #330 #335
timothy added the ci-cdpriority: lowsecurity labels 2026-07-13 20:36:26 +02:00
timothy added the in-progress label 2026-07-18 16:30:48 +02:00
Author
Owner

🤖 Claiming as part of a batched small-issue sweep this session (parallel worktrees, one PR per item). Will post a close comment per the Task Completion Protocol when its PR merges.

🤖 Claiming as part of a batched small-issue sweep this session (parallel worktrees, one PR per item). Will post a close comment per the Task Completion Protocol when its PR merges.
Author
Owner

✅ Merged via PR #452 (squash). Root cause: security-scan.sh passed ZAP's raw exit code straight through, so a warnings-only run (exit 2) failed the job identically to a real FAIL (exit 1). Done: added classify_zap_exit() (0=PASS, 2=WARN→exit 0, 1/124/other=FAIL), a docker-free --selftest regression harness (6 cases), and updated the exit-code contract in docs/ci-cd.md. Files: scripts/security-scan.sh, docs/ci-cd.md. bash -n + shellcheck + selftest green.

✅ Merged via PR #452 (squash). **Root cause:** `security-scan.sh` passed ZAP's raw exit code straight through, so a warnings-only run (exit 2) failed the job identically to a real FAIL (exit 1). **Done:** added `classify_zap_exit()` (0=PASS, 2=WARN→exit 0, 1/124/other=FAIL), a docker-free `--selftest` regression harness (6 cases), and updated the exit-code contract in `docs/ci-cd.md`. **Files:** `scripts/security-scan.sh`, `docs/ci-cd.md`. bash -n + shellcheck + selftest green.
timothy removed the in-progress label 2026-07-18 18:22:19 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#338