release: cut and promote v26.11.0 to prod (first release on jazz) #515

Closed
opened 2026-07-20 23:14:37 +02:00 by timothy · 1 comment
Owner

Done

What was done: Cut v26.11.0 and took it all the way through the release ritual to running in prod — the first ErsatzTV release promoted to jazz after the host move (server-management#633). Held the tag mid-flight at Timothy's request until #498 (QSV/VA-API decode) landed, then re-baselined on the new main.

Tag v26.11.0 @ 601d855b — 83 commits since v26.10.0, headlined by #498 (QSV profiles decode via VA-API, QsvPreferNativeDecoder default ON) and #67 (unified channel logo / on-screen bug).

Pre-flight

  • main CI green on the tag target; vulnerable-package scan clean (15/15 projects)
  • Full local gate: 3945 passed, 0 failed
  • docs/decisions.md: zero superseded entries, nothing to consolidate
  • Carries a dual-provider migration (Add_FFmpegProfile_QsvPreferNativeDecoder, SQLite + MySQL)

Release gate — ZAP DAST on jazz against the immutable :26.11.0: FAIL-NEW 0 / PASS 118 across 215 imported API URLs. The single WARN is the SPA catch-all returning text/html on /, /app and unmatched paths — expected by design, ZAP assumes a pure-API surface. semgrep SAST: 175 rules × 4478 targets, 0 findings (verified non-vacuous — --quiet produces zero bytes either way).

Promotion — Komodo DeployStack jazz-media, all 6 stages green. Pre-deploy chain fired as designed: image-change trigger (5cc2556a → ca5e1fa1) → 286 M backup, integrity_check=ok, off-box PBS → migration smoke PASS on the prod-copy → only ersatztv recreated.

Live verification — prod runs registry manifest digest sha256:ca5e1fa1…, matched against :26.11.0 rather than inferred from tag equality (the #350 lesson). Container healthy, migration 20260720175125_… applied on the live DB, 43 channels in M3U and XMLTV, 4397 programmes, 0 [ERR]/[FTL].

Root cause (of the doc defects found): the release ritual was written for bumblebee and silently broke in three ways after the host move — none would have thrown an error.

  1. security-scan.sh on bumblebee → would have scanned the wrong host.
  2. jazz is a reused name — it was an earlier name for the .99 host, so pre-migration references meaning today's bumblebee read as "jazz", including one asserting jazz (192.168.1.99). A find-and-replace in either direction corrupts the other set, so the fix is a Hosts table + resolve by IP, not by name.
  3. DeployStack media-servers names a dead stack: a media-servers stack still exists on bumblebee (unhealthy, the stopped leftovers) so the wrong name doesn't error, and container labels hide it (compose project is still media-servers; the Komodo stack is jazz-media). Also auto_update: false now, so the documented Global-Auto-Update fallback no longer exists.

Near-miss worth recording: jazz-media's pre_deploy field reads as a no-op (ssh … true) and I nearly filed a lost-safety-hook regression. It's an authorized_keys forced command — the true is discarded and etv-komodo-predeploy.sh runs. Judge the hook by the deploy's Pre Deploy stage log, never the config field.

Files changed: docs/ci-cd.md (Hosts table + name-reuse warning, scan host, docker host, jazz-media + no-auto-update, version-table rows for v26.10.0/v26.11.0), CLAUDE.md (deploy host, config volume, Komodo stack), docs/m3u-xmltv.md (stale jazz → bumblebee).

PRs: #501, #507, #514 (all docs-only, merged green).

Deferred:

  • #498 is inert in prod — the only FFmpeg profile is VAAPI, so QsvPreferNativeDecoder (stored 1) does nothing until a profile is switched to QSV. That switch is the actual payoff and is Timothy's call; confirming the fix needs a few channel cold-starts watched afterwards.
  • Migration leftovers on bumblebee — stopped ersatztv / ersatztv-test / jellyfin plus the dead media-servers Komodo stack. Deliberately not reaped (not mine to remove); they are what made trap #3 silent. Belongs to server-management.

Follow-up issues: none filed in this repo — the remaining cleanup is server-management's (bumblebee leftovers).

Docs updated: docs/ci-cd.md, CLAUDE.md, docs/m3u-xmltv.md (PRs #501/#507/#514); vault ~/homelab-docs/Docker/ErsatzTV.md (auto-update now OFF, resolved the open "confirm the backup chain exists on jazz" caveat by execution, recorded both traps) — pushed as f507d42. Project memory: prod-moved-to-jazz-and-name-reuse, komodo-stack-is-jazz-media.

## Done **What was done**: Cut **v26.11.0** and took it all the way through the release ritual to running in prod — the first ErsatzTV release promoted to **jazz** after the host move (server-management#633). Held the tag mid-flight at Timothy's request until #498 (QSV/VA-API decode) landed, then re-baselined on the new `main`. Tag `v26.11.0` @ `601d855b` — **83 commits** since v26.10.0, headlined by #498 (QSV profiles decode via VA-API, `QsvPreferNativeDecoder` default **ON**) and #67 (unified channel logo / on-screen bug). **Pre-flight** - `main` CI green on the tag target; vulnerable-package scan clean (15/15 projects) - Full local gate: **3945 passed, 0 failed** - `docs/decisions.md`: zero superseded entries, nothing to consolidate - Carries a dual-provider migration (`Add_FFmpegProfile_QsvPreferNativeDecoder`, SQLite + MySQL) **Release gate** — ZAP DAST on jazz against the **immutable** `:26.11.0`: FAIL-NEW **0** / PASS 118 across 215 imported API URLs. The single WARN is the SPA catch-all returning `text/html` on `/`, `/app` and unmatched paths — expected by design, ZAP assumes a pure-API surface. semgrep SAST: **175 rules × 4478 targets, 0 findings** (verified non-vacuous — `--quiet` produces zero bytes either way). **Promotion** — Komodo `DeployStack jazz-media`, all 6 stages green. Pre-deploy chain fired as designed: image-change trigger (`5cc2556a → ca5e1fa1`) → 286 M backup, `integrity_check=ok`, off-box PBS → **migration smoke PASS** on the prod-copy → only `ersatztv` recreated. **Live verification** — prod runs registry **manifest digest** `sha256:ca5e1fa1…`, matched against `:26.11.0` rather than inferred from tag equality (the #350 lesson). Container `healthy`, migration `20260720175125_…` applied on the live DB, **43 channels** in M3U and XMLTV, **4397 programmes**, 0 `[ERR]`/`[FTL]`. **Root cause** (of the doc defects found): the release ritual was written for bumblebee and **silently** broke in three ways after the host move — none would have thrown an error. 1. `security-scan.sh on bumblebee` → would have scanned the wrong host. 2. **`jazz` is a reused name** — it was an earlier name for the .99 host, so pre-migration references meaning *today's bumblebee* read as "jazz", including one asserting `jazz (192.168.1.99)`. A find-and-replace in either direction corrupts the other set, so the fix is a Hosts table + **resolve by IP, not by name**. 3. `DeployStack media-servers` names a **dead** stack: a `media-servers` stack still exists on bumblebee (`unhealthy`, the stopped leftovers) so the wrong name doesn't error, and container labels hide it (compose *project* is still `media-servers`; the Komodo *stack* is `jazz-media`). Also `auto_update: false` now, so the documented Global-Auto-Update fallback no longer exists. **Near-miss worth recording**: `jazz-media`'s `pre_deploy` field reads as a no-op (`ssh … true`) and I nearly filed a lost-safety-hook regression. It's an `authorized_keys` **forced command** — the `true` is discarded and `etv-komodo-predeploy.sh` runs. Judge the hook by the deploy's `Pre Deploy` **stage log**, never the config field. **Files changed**: `docs/ci-cd.md` (Hosts table + name-reuse warning, scan host, docker host, `jazz-media` + no-auto-update, version-table rows for v26.10.0/v26.11.0), `CLAUDE.md` (deploy host, config volume, Komodo stack), `docs/m3u-xmltv.md` (stale `jazz` → bumblebee). **PRs**: #501, #507, #514 (all docs-only, merged green). **Deferred**: - **#498 is inert in prod** — the only FFmpeg profile is VAAPI, so `QsvPreferNativeDecoder` (stored `1`) does nothing until a profile is switched to QSV. That switch is the actual payoff and is Timothy's call; confirming the fix needs a few channel cold-starts watched afterwards. - **Migration leftovers on bumblebee** — stopped `ersatztv` / `ersatztv-test` / `jellyfin` plus the dead `media-servers` Komodo stack. Deliberately not reaped (not mine to remove); they are what made trap #3 silent. Belongs to server-management. **Follow-up issues**: none filed in this repo — the remaining cleanup is server-management's (bumblebee leftovers). **Docs updated**: `docs/ci-cd.md`, `CLAUDE.md`, `docs/m3u-xmltv.md` (PRs #501/#507/#514); vault `~/homelab-docs/Docker/ErsatzTV.md` (auto-update now OFF, resolved the open "confirm the backup chain exists on jazz" caveat by execution, recorded both traps) — pushed as `f507d42`. Project memory: `prod-moved-to-jazz-and-name-reuse`, `komodo-stack-is-jazz-media`.
timothy added the ad-hoc label 2026-07-20 23:14:37 +02:00
Author
Owner

Deferred items — both now closed out

1. #498 activated in prod. Profile 1 switched hardwareAcceleration: Vaapi → Qsv (keeping qsvPreferNativeDecoder: true), so the release's headline fix is no longer inert. Verified by the live ffmpeg command line — -hwaccel vaapi decode + -init_hw_device qsv=hw@va + -c:v h264_qsv encode, exactly the Jellyfin pattern — and 6/6 channel cold-starts clean against the issue's measured 50% failure rate. Full evidence in #498.

Rollback if ever needed: PUT /api/v1/ffmpeg/profiles/1 with hardwareAcceleration: "Vaapi". (The API normalizes padMode to HardwareIfPossible on the QSV path.)

2. Bumblebee leftovers filed as server-management#641 (priority: low) — the stopped ersatztv/ersatztv-test/jellyfin/dispatcharr containers plus the dead media-servers Komodo stack. Framed around why it matters rather than tidiness: while that stack exists, DeployStack media-servers silently succeeds against the wrong, dead target, and the container labels don't reveal it because the compose project on jazz is still media-servers.

## Deferred items — both now closed out **1. #498 activated in prod.** Profile 1 switched `hardwareAcceleration: Vaapi → Qsv` (keeping `qsvPreferNativeDecoder: true`), so the release's headline fix is no longer inert. Verified by the live ffmpeg command line — `-hwaccel vaapi` decode + `-init_hw_device qsv=hw@va` + `-c:v h264_qsv` encode, exactly the Jellyfin pattern — and **6/6 channel cold-starts clean** against the issue's measured 50% failure rate. Full evidence in #498. Rollback if ever needed: `PUT /api/v1/ffmpeg/profiles/1` with `hardwareAcceleration: "Vaapi"`. (The API normalizes `padMode` to `HardwareIfPossible` on the QSV path.) **2. Bumblebee leftovers** filed as **server-management#641** (`priority: low`) — the stopped `ersatztv`/`ersatztv-test`/`jellyfin`/`dispatcharr` containers plus the dead `media-servers` Komodo stack. Framed around why it matters rather than tidiness: while that stack exists, `DeployStack media-servers` silently succeeds against the wrong, dead target, and the container labels don't reveal it because the compose *project* on jazz is still `media-servers`.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#515