CI monitors: record that skipped is not red (build skips on every PR) #592

Closed
opened 2026-07-25 11:53:17 +02:00 by timothy · 1 comment
Owner

Surfaced while landing #436/#583 on 2026-07-25.

What happened

My CI monitor enumerated per-context statuses and treated anything != "success" as red. On two fully green PRs it announced "NOT all green" because Build & push image (amd64) reports skipped.

Nothing was blocked — Gitea's combined /status endpoint already treats skipped as non-blocking and reported overall=success, and both PRs merged cleanly through the consent gate. But the monitor is the thing an agent reads to decide whether to investigate, so a false red costs a diagnosis cycle every time.

Two corrections worth recording

  1. The image job is skipped on every PR, not because of docs-only. It is if:-gated at the JOB level on github.event_name != 'pull_request' — images are built only on push-to-main and tags. I initially misattributed the skip to #416's docs-only mechanism, which is a plausible-sounding wrong diagnosis: docs-only deliberately gates individual STEPS precisely so required jobs still report success (ci.docs-only-skip-steps). decisions.md already states "build is skipped on every PR" from the branch-protection angle; what was missing is the monitor-authoring consequence.
  2. skipped / failure / cancelled are three different things. skipped = deliberately not applicable, settled, not red. failure = a real red (check ci.killed-job-triage first). cancelled = no verdict (ci.cancelled-is-not-a-verdict). A monitor that collapses them misreports.

Change

Amended ci.monitor-armed-at-pr-open (docs/decisions/workflow-process.md) — the record that already tells you to arm a monitor — with the classification rules, a preference for gating on the combined .state, and a working enumeration filter.

The snippet needed fixing during authoring, which is itself the lesson: my first draft used select(.status != …) after the jq pipeline had renamed .status to .st, so it compared against null, passed everything, and reported a green PR as nine failures. The committed version is verified in both directions — silent on a green PR with a skipped build, and still dirty on a genuinely pending run (per process.bom-format-detection-recipe's "verify your detector" rule).

Also noticed

decisions_validate.py now emits ::warning:: aggregate active-corpus is 5606 lines (budget 5600) — schedule a consolidation. Non-blocking, but the budget is crossed; worth its own consolidation pass.

Done-when

  • ci.monitor-armed-at-pr-open amended with skipped/failure/cancelled classification + verified filter
  • Catalog regenerated, decisions-validate: OK
  • adversarial review passed
Surfaced while landing #436/#583 on 2026-07-25. ## What happened My CI monitor enumerated per-context statuses and treated anything `!= "success"` as red. On two fully green PRs it announced "NOT all green" because `Build & push image (amd64)` reports **`skipped`**. Nothing was blocked — Gitea's combined `/status` endpoint already treats `skipped` as non-blocking and reported `overall=success`, and both PRs merged cleanly through the consent gate. But the monitor is the thing an agent reads to decide whether to investigate, so a false red costs a diagnosis cycle every time. ## Two corrections worth recording 1. **The image job is skipped on *every* PR, not because of docs-only.** It is `if:`-gated at the JOB level on `github.event_name != 'pull_request'` — images are built only on push-to-main and tags. I initially misattributed the skip to #416's docs-only mechanism, which is a plausible-sounding wrong diagnosis: docs-only deliberately gates individual STEPS precisely so required jobs still report `success` (`ci.docs-only-skip-steps`). `decisions.md` already states "`build` is `skipped` on every PR" from the branch-protection angle; what was missing is the monitor-authoring consequence. 2. **`skipped` / `failure` / `cancelled` are three different things.** `skipped` = deliberately not applicable, settled, not red. `failure` = a real red (check `ci.killed-job-triage` first). `cancelled` = no verdict (`ci.cancelled-is-not-a-verdict`). A monitor that collapses them misreports. ## Change Amended `ci.monitor-armed-at-pr-open` (`docs/decisions/workflow-process.md`) — the record that already tells you to arm a monitor — with the classification rules, a preference for gating on the combined `.state`, and a working enumeration filter. **The snippet needed fixing during authoring**, which is itself the lesson: my first draft used `select(.status != …)` after the jq pipeline had renamed `.status` to `.st`, so it compared against `null`, passed everything, and reported a green PR as nine failures. The committed version is verified in both directions — silent on a green PR with a skipped `build`, and still dirty on a genuinely pending run (per `process.bom-format-detection-recipe`'s "verify your detector" rule). ## Also noticed `decisions_validate.py` now emits `::warning:: aggregate active-corpus is 5606 lines (budget 5600) — schedule a consolidation`. Non-blocking, but the budget is crossed; worth its own consolidation pass. ## Done-when - [x] `ci.monitor-armed-at-pr-open` amended with skipped/failure/cancelled classification + verified filter - [x] Catalog regenerated, `decisions-validate: OK` - [x] adversarial review passed
timothy added the ci-cdpriority: low labels 2026-07-25 11:56:32 +02:00
Author
Owner

Closing record

Outcome: Shipped via PR #593 (merged, 2fee3f0b). ci.monitor-armed-at-pr-open now carries the monitor-classification rules: prefer gating on the combined /status .state, and if you enumerate per-context, treat skipped as settled-not-red. Prose + Signals: only — heading, key: and lifecycle fields unchanged, [decisions-edit] in the commit, so the decisions lifecycle job saw an edit rather than an unlogged removal.

Root cause: My CI monitor filtered per-context statuses on != "success" and so reported skipped as red on two fully green PRs. The underlying CI fact was already recorded (ci.docs-only-skip-steps states "build is skipped on every PR") — the gap was that no record connected it to writing a monitor, and ci.monitor-armed-at-pr-open told you to arm one without saying how to classify what comes back.

Decisions/conventions changed: ci.monitor-armed-at-pr-open amended (no supersession, no new key).

Reusable knowledge:

  1. skipped ≠ failure ≠ cancelled. Settled-not-applicable; a real red; no verdict at all. Gitea's combined .state already gets this right — a PR with a skipped build reports overall=success — which is why gating on the combined endpoint is safer than hand-rolling per-context logic.
  2. Build & push image (amd64) is skipped on EVERY PR — job-level if: github.event_name != 'pull_request'; images build only on push-to-main and tags. Do not attribute that skip to the docs-only mechanism: docs-only gates individual steps precisely so required jobs still report success in seconds. I made exactly that misattribution first, and it is the kind of wrong-but-coherent explanation that survives because it sounds right.
  3. A knowledge gap can be a connection gap, not a missing fact. The CI behaviour was documented; what was missing was the link from it to the task that trips over it. Worth checking, when something bites, whether the fact exists somewhere already and only the pointer is absent.
  4. Verify a detector in both directions before documenting it. My first draft of the jq filter used select(.status != …) after the pipeline had renamed .status to .st — it compared against null, passed everything, and reported a green PR as nine failures. Caught only by running it. The committed version is proven silent on a green PR carrying a skipped build and proven to still report a genuinely pending run.

Verification: Bounded independent fact-check (cheapest tier, capped scope — proportionate to a docs leaf, but run rather than skipped because the record asserts things about CI wiring). All six claims TRUE: the job-level if: at docker-build.yml:518; docs-only gating steps at :534/557/565/573; ci.docs-only-skip-steps active and containing the quoted phrase; no dangling key references among the four cross-referenced keys; the record header byte-identical to origin/main; and the documented jq filter emitting only pending/failure on a synthetic [success, skipped, pending, failure] payload. decisions-validate: OK, catalog regenerated, CI green on 2fee3f0b.

Deferred: decisions_validate.py now warns the aggregate active corpus is 5606 lines against a 5600 budget — non-blocking, but the budget is crossed and wants its own consolidation pass. Not filed as a separate issue yet; flagging here so the next session sees it.

Docs updated: docs/decisions/workflow-process.md (ci.monitor-armed-at-pr-open amended), docs/decisions/README.md (regenerated).

## Closing record **Outcome:** Shipped via PR #593 (merged, `2fee3f0b`). `ci.monitor-armed-at-pr-open` now carries the monitor-classification rules: prefer gating on the combined `/status` `.state`, and if you enumerate per-context, treat `skipped` as settled-not-red. Prose + `Signals:` only — heading, `key:` and lifecycle fields unchanged, `[decisions-edit]` in the commit, so the `decisions lifecycle` job saw an edit rather than an unlogged removal. **Root cause:** My CI monitor filtered per-context statuses on `!= "success"` and so reported `skipped` as red on two fully green PRs. The underlying CI fact was *already* recorded (`ci.docs-only-skip-steps` states "`build` is `skipped` on every PR") — the gap was that no record connected it to **writing a monitor**, and `ci.monitor-armed-at-pr-open` told you to arm one without saying how to classify what comes back. **Decisions/conventions changed:** `ci.monitor-armed-at-pr-open` amended (no supersession, no new key). **Reusable knowledge:** 1. **`skipped` ≠ `failure` ≠ `cancelled`.** Settled-not-applicable; a real red; no verdict at all. Gitea's combined `.state` already gets this right — a PR with a skipped `build` reports `overall=success` — which is why gating on the combined endpoint is safer than hand-rolling per-context logic. 2. **`Build & push image (amd64)` is skipped on EVERY PR** — job-level `if: github.event_name != 'pull_request'`; images build only on push-to-main and tags. **Do not** attribute that skip to the docs-only mechanism: docs-only gates individual *steps* precisely so required jobs still report `success` in seconds. I made exactly that misattribution first, and it is the kind of wrong-but-coherent explanation that survives because it sounds right. 3. **A knowledge gap can be a *connection* gap, not a missing fact.** The CI behaviour was documented; what was missing was the link from it to the task that trips over it. Worth checking, when something bites, whether the fact exists somewhere already and only the pointer is absent. 4. **Verify a detector in both directions before documenting it.** My first draft of the jq filter used `select(.status != …)` after the pipeline had renamed `.status` to `.st` — it compared against `null`, passed everything, and reported a green PR as nine failures. Caught only by running it. The committed version is proven silent on a green PR carrying a skipped `build` *and* proven to still report a genuinely pending run. **Verification:** Bounded independent fact-check (cheapest tier, capped scope — proportionate to a docs leaf, but run rather than skipped because the record asserts things about CI wiring). All six claims TRUE: the job-level `if:` at `docker-build.yml:518`; docs-only gating steps at :534/557/565/573; `ci.docs-only-skip-steps` active and containing the quoted phrase; **no dangling key references** among the four cross-referenced keys; the record header byte-identical to `origin/main`; and the documented jq filter emitting only `pending`/`failure` on a synthetic `[success, skipped, pending, failure]` payload. `decisions-validate: OK`, catalog regenerated, CI green on `2fee3f0b`. **Deferred:** `decisions_validate.py` now warns the aggregate active corpus is **5606 lines against a 5600 budget** — non-blocking, but the budget is crossed and wants its own consolidation pass. Not filed as a separate issue yet; flagging here so the next session sees it. **Docs updated:** `docs/decisions/workflow-process.md` (`ci.monitor-armed-at-pr-open` amended), `docs/decisions/README.md` (regenerated).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#592