Channels 102/107 repeatedly fail to tune: h264_vaapi hwupload -22 / exit 234 on prod #478

Closed
opened 2026-07-19 20:59:54 +02:00 by timothy · 4 comments
Owner

Found incidentally while pulling prod logs for #473 (2026-07-19, prod 26.10.0). Currently firing, unrelated to #473's missing-media class.

Symptom

Channels 102 and 107 repeatedly die mid-transcode:

[20:25:52 ERR] HLS process for channel 107 has terminated unsuccessfully with exit code 234:
amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description.
If they do, bad things may happen!
[hwupload @ 0x7f9160053300] Failed to upload frame: -22.
[fc#0 @ ...] Error while filtering: Invalid argument
[vost#0:0/h264_vaapi @ ...] [enc:h264_vaapi @ ...] Could not open encoder before EOF
[out#0/hls @ ...] Nothing was written into output file, because at least one of its streams received no packets.

Channel 102 shows the same signature with exit code 183. Observed at 20:17:13, 20:17:25, 20:20:53, 20:21:05 (ch102) and 20:25:52, 20:26:33, 20:26:55, 20:27:18 (ch107) — i.e. every tune attempt, not an occasional flake.

Notes

  • The amdgpu: os_same_file_description line is a warning, not the cause. The actual failure is hwupload failing with -22 (Invalid argument), after which the VAAPI encoder never opens and ffmpeg writes nothing.
  • Immediately before one failure: Media item [27008] on channel 107 transcoded at 1.38x (NOT throttled) which may not be fast enough to support playback.
  • The session does not die — HlsSessionWorker's failure path renders an error card and the next cold-start reports segments 1/1. So the viewer sees an error card rather than a black screen, and the channel keeps limping.
  • Other channels (124, 125, 100) cold-start fine with hwaccel vaapi in the same window, so this is not "VAAPI is broken" globally — it looks specific to these channels' content or FFmpeg profile (a filter-chain/pixel-format combination that fails to hwupload).

Suggested first steps

  1. Diff the FFmpeg profiles and any watermark/graphics/subtitle-burn-in settings of 102/107 against a working channel (124/125).
  2. Identify media item 27008's pixel format / resolution — a 10-bit or unusual-format source is the usual hwupload -22 trigger.
  3. Determine whether this is an ErsatzTV filter-graph construction bug (ErsatzTV.FFmpeg pipeline) or a host/driver limitation; if the latter it belongs in timothy/server-management, not here.

Do not fold this into #473 — different class entirely (that one is a dead media reference; this is a live file the transcoder can't process).

Done-when

  • Root cause identified (filter-graph construction vs. source format vs. host/driver)
  • Channels 102 and 107 tune successfully on prod
  • If it is a pipeline bug: covered by a test in ErsatzTV.FFmpeg.Tests (asserts the rendered arg string)
  • If it is host/driver: closed here with a linked timothy/server-management issue
  • Adversarial review passed
Found incidentally while pulling prod logs for #473 (2026-07-19, prod `26.10.0`). **Currently firing**, unrelated to #473's missing-media class. ## Symptom Channels **102** and **107** repeatedly die mid-transcode: ``` [20:25:52 ERR] HLS process for channel 107 has terminated unsuccessfully with exit code 234: amdgpu: os_same_file_description couldn't determine if two DRM fds reference the same file description. If they do, bad things may happen! [hwupload @ 0x7f9160053300] Failed to upload frame: -22. [fc#0 @ ...] Error while filtering: Invalid argument [vost#0:0/h264_vaapi @ ...] [enc:h264_vaapi @ ...] Could not open encoder before EOF [out#0/hls @ ...] Nothing was written into output file, because at least one of its streams received no packets. ``` Channel 102 shows the same signature with exit code **183**. Observed at 20:17:13, 20:17:25, 20:20:53, 20:21:05 (ch102) and 20:25:52, 20:26:33, 20:26:55, 20:27:18 (ch107) — i.e. every tune attempt, not an occasional flake. ## Notes - The `amdgpu: os_same_file_description` line is a **warning**, not the cause. The actual failure is `hwupload` failing with `-22 (Invalid argument)`, after which the VAAPI encoder never opens and ffmpeg writes nothing. - Immediately before one failure: `Media item [27008] on channel 107 transcoded at 1.38x (NOT throttled) which may not be fast enough to support playback`. - The session does **not** die — `HlsSessionWorker`'s failure path renders an error card and the next cold-start reports `segments 1/1`. So the viewer sees an error card rather than a black screen, and the channel keeps limping. - Other channels (124, 125, 100) cold-start fine with `hwaccel vaapi` in the same window, so this is not "VAAPI is broken" globally — it looks specific to these channels' content or FFmpeg profile (a filter-chain/pixel-format combination that fails to hwupload). ## Suggested first steps 1. Diff the FFmpeg profiles and any watermark/graphics/subtitle-burn-in settings of 102/107 against a working channel (124/125). 2. Identify media item 27008's pixel format / resolution — a 10-bit or unusual-format source is the usual `hwupload -22` trigger. 3. Determine whether this is an ErsatzTV filter-graph construction bug (`ErsatzTV.FFmpeg` pipeline) or a host/driver limitation; if the latter it belongs in `timothy/server-management`, not here. **Do not fold this into #473** — different class entirely (that one is a dead media reference; this is a live file the transcoder can't process). ## Done-when - [ ] Root cause identified (filter-graph construction vs. source format vs. host/driver) - [ ] Channels 102 and 107 tune successfully on prod - [ ] If it is a pipeline bug: covered by a test in `ErsatzTV.FFmpeg.Tests` (asserts the rendered arg string) - [ ] If it is host/driver: closed here with a linked `timothy/server-management` issue - [ ] Adversarial review passed
timothy added the bugpriority: high labels 2026-07-19 20:59:54 +02:00
timothy added the in-progress label 2026-07-19 22:28:58 +02:00
Author
Owner

Claiming this (Claude Code orchestrator session, 2026-07-19).

Plan: diagnosis-first per the issue's suggested steps — this is explicitly "pipeline bug vs. host/driver" and I won't propose a fix until the root cause is pinned. Starting by pulling from prod:

  1. FFmpeg profile + channel config for 102/107 vs a working channel (124/125).
  2. Media item 27008's codec / pixel format / resolution / bit depth (10-bit is the usual hwupload -22 trigger).
  3. The exact rendered ffmpeg arg string ErsatzTV builds for these channels, to see whether the filter-graph / pixel-format handoff into hwupload is malformed.

If it's a host/driver limitation I'll close here with a linked timothy/server-management issue; if it's an ErsatzTV.FFmpeg filter-graph construction bug I'll fix it with an arg-string test in ErsatzTV.FFmpeg.Tests.

Claiming this (Claude Code orchestrator session, 2026-07-19). Plan: diagnosis-first per the issue's suggested steps — this is explicitly "pipeline bug vs. host/driver" and I won't propose a fix until the root cause is pinned. Starting by pulling from prod: 1. FFmpeg profile + channel config for 102/107 vs a working channel (124/125). 2. Media item 27008's codec / pixel format / resolution / bit depth (10-bit is the usual `hwupload -22` trigger). 3. The exact rendered ffmpeg arg string ErsatzTV builds for these channels, to see whether the filter-graph / pixel-format handoff into `hwupload` is malformed. If it's a host/driver limitation I'll close here with a linked `timothy/server-management` issue; if it's an `ErsatzTV.FFmpeg` filter-graph construction bug I'll fix it with an arg-string test in `ErsatzTV.FFmpeg.Tests`.
Author
Owner

Root cause narrowed — it's a pipeline (bit-depth) bug, not host/driver

Evidence gathered from prod (26.10.0, DB + real hardware):

  1. Source is 10-bit. Media item 27008 (ch107, Curb S12E07) — MediaStream: hevc / main 10 / yuv420p10le, SDR bt709. Confirmed by ffprobe on the real file too (pix_fmt=yuv420p10le).
  2. Only the source differs. Channels 102, 107, 124, 125, 100 all use the identical FFmpeg profile (1080p VAAPI h264 aac — h264 8-bit output, VAAPI, VaapiDriver=RadeonSI, no tonemap) and all carry WatermarkId=1. The working channels (124/125/100) just happen to be on 8-bit content; 102/107 are on 10-bit. So the discriminator is purely 8-bit vs 10-bit source.
  3. The AMD driver is fine — with the correct chain. On the real GPU (Radeon 760M, radeonsi/phoenix, ffmpeg 8.1.2) against the actual file:
    • hwdownload,format=p010le,hwupload → h264_vaapi (10-bit surface into the 8-bit encoder) → fails (encoder won't open / frame rejected). This is the prod failure class.
    • hwdownload,format=p010le,format=nv12,hwupload → h264_vaapi (reduce to 8-bit before upload) → succeeds.
    • Pure on-GPU scale_vaapi=format=nv12 also succeeds.

Conclusion: h264_vaapi on this box needs an 8-bit (nv12) surface. When a 10-bit p010 frame reaches hwupload/the encoder, it dies (-22 / encoder-open failure). The fix is to guarantee a bit-depth reduction to nv12 before the final hwupload on the VAAPI software-overlay path (radeonsi forces software overlay for the watermark, so the frame is hw-downloaded to 10-bit p010, overlaid, and must be converted to nv12 before re-upload).

Still pinning down: the exact command VaapiPipelineBuilder emits for the 10-bit + watermark path, to see precisely where the format=nv12 is missing/mis-ordered (a deterministic repro test in ErsatzTV.FFmpeg.Tests is in progress; it doubles as the regression test). Fix + test to follow.

Done-when box 1 (root cause) ✓ effectively; it's a pipeline bug → box 3 (arg-string test) applies, not box 4 (host/driver).

### Root cause narrowed — it's a pipeline (bit-depth) bug, not host/driver **Evidence gathered from prod (`26.10.0`, DB + real hardware):** 1. **Source is 10-bit.** Media item 27008 (ch107, *Curb* S12E07) — `MediaStream`: `hevc / main 10 / yuv420p10le`, SDR bt709. Confirmed by `ffprobe` on the real file too (`pix_fmt=yuv420p10le`). 2. **Only the source differs.** Channels 102, 107, 124, 125, 100 all use the *identical* FFmpeg profile (`1080p VAAPI h264 aac` — h264 **8-bit** output, VAAPI, `VaapiDriver=RadeonSI`, no tonemap) and all carry `WatermarkId=1`. The working channels (124/125/100) just happen to be on 8-bit content; 102/107 are on 10-bit. So the discriminator is purely **8-bit vs 10-bit source**. 3. **The AMD driver is fine — with the correct chain.** On the real GPU (Radeon 760M, radeonsi/phoenix, ffmpeg 8.1.2) against the actual file: - `hwdownload,format=p010le,hwupload → h264_vaapi` (10-bit surface into the 8-bit encoder) → **fails** (encoder won't open / frame rejected). This is the prod failure class. - `hwdownload,format=p010le,format=nv12,hwupload → h264_vaapi` (reduce to 8-bit **before** upload) → **succeeds**. - Pure on-GPU `scale_vaapi=format=nv12` also succeeds. **Conclusion:** `h264_vaapi` on this box needs an **8-bit (nv12)** surface. When a 10-bit `p010` frame reaches `hwupload`/the encoder, it dies (`-22` / encoder-open failure). The fix is to guarantee a bit-depth reduction to nv12 before the final `hwupload` on the VAAPI **software-overlay** path (radeonsi forces software overlay for the watermark, so the frame is hw-downloaded to 10-bit `p010`, overlaid, and must be converted to nv12 before re-upload). **Still pinning down:** the exact command `VaapiPipelineBuilder` emits for the 10-bit + watermark path, to see precisely where the `format=nv12` is missing/mis-ordered (a deterministic repro test in `ErsatzTV.FFmpeg.Tests` is in progress; it doubles as the regression test). Fix + test to follow. Done-when box 1 (root cause) ✓ effectively; it's a pipeline bug → box 3 (arg-string test) applies, not box 4 (host/driver).
Author
Owner

Root cause: transient post-deploy VAAPI/DRM state under a tune burst — not an ErsatzTV bug. Already self-resolved.

The command ErsatzTV generates is correct

I captured the exact ffmpeg argv prod runs for ch107 (10-bit Curb), verbatim from /proc/<pid>/cmdline:

... -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i <file> -i <logo>
-filter_complex [0:0]hwdownload,format=vaapi|p010le[v];[1:0]scale=96:-1,...[wm];
  [v][wm]overlay=...:format=0[vwm];
  [vwm]colorspace=...:format=yuv420p:range=tv,format=nv12|p010le|vaapi,hwupload[vpf]
-map [vpf] ... -c:v h264_vaapi -profile:v high ... -f hls ...

Ran this verbatim on the real GPU (Radeon 760M / radeonsi / ffmpeg 8.1.2) against the actual 10-bit files:

  • standalone → succeeds
  • faithfully (with -ss seek + real logo + full hls muxer, 10s) → succeeds (2 segments)
  • 4× concurrent 10-bit sessions → all succeed

So the filtergraph is well-formed and the driver handles 10-bit→h264_vaapi fine. (My earlier "missing format=nv12" lead was wrong — the real chain converts to 8-bit via colorspace=...:format=yuv420p before upload; format=nv12|p010le|vaapi negotiates nv12 from the 8-bit input.)

What actually happened

  • The container was recreated at 20:15:11 for the 26.10.0 deploy.
  • Immediately after, a burst of 15 channels (101–125) cold-started (client/EPG refresh hitting everything at once).
  • On the freshly re-initialized phoenix iGPU render node under that burst, hwupload failed with -22 only for the 10-bit channels (102/107 — p010, ~2× the surface memory of 8-bit nv12); 8-bit channels succeeded in the same window. 8 failures total, all between 20:17:13 and 20:27:18.
  • It cleared on its own. Since ~20:28 there have been zero -22.

Verified fixed now

  • ch107 tune at 22:47 → segments 1/1, clean.
  • ch102 + ch107 concurrently at 22:51 → both segments 1/1, clean.
  • 8-channel concurrent burst (100/101/103/105/110/120/122/123) at 22:53 → all segments 1/1, clean (startup latency rose to 3.6–8.1s under load, but no failures).

The concurrency burst alone, on a settled GPU, does not reproduce it — the trigger required the burst plus the fresh-post-recreate GPU state.

Done-when

  • Root cause identified — transient VAAPI surface-allocation failure on the freshly-recreated render node under a concurrent tune burst; hits 10-bit (larger p010 surfaces) first. ErsatzTV's generated command is correct.
  • Channels 102 and 107 tune successfully on prod — verified, incl. concurrently.
  • pipeline bug → test — N/A (not a pipeline bug).
  • Host/driver → link a timothy/server-management issue (filing next).

Recommend closing as diagnosed / host-transient, self-resolved, with a server-management follow-up to watch whether the next container recreation reproduces the post-deploy VAAPI transient. Optional ErsatzTV hardening (separate enhancement): fall back to software encode after N consecutive VAAPI hwupload failures so a transient GPU state can't leave a channel showing error cards for ~13 min.

## Root cause: transient post-deploy VAAPI/DRM state under a tune burst — **not** an ErsatzTV bug. Already self-resolved. ### The command ErsatzTV generates is correct I captured the **exact** ffmpeg argv prod runs for ch107 (10-bit *Curb*), verbatim from `/proc/<pid>/cmdline`: ``` ... -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i <file> -i <logo> -filter_complex [0:0]hwdownload,format=vaapi|p010le[v];[1:0]scale=96:-1,...[wm]; [v][wm]overlay=...:format=0[vwm]; [vwm]colorspace=...:format=yuv420p:range=tv,format=nv12|p010le|vaapi,hwupload[vpf] -map [vpf] ... -c:v h264_vaapi -profile:v high ... -f hls ... ``` Ran this **verbatim** on the real GPU (Radeon 760M / radeonsi / ffmpeg 8.1.2) against the actual 10-bit files: - standalone → **succeeds** - faithfully (with `-ss` seek + real logo + full hls muxer, 10s) → **succeeds** (2 segments) - **4× concurrent** 10-bit sessions → all succeed So the filtergraph is well-formed and the driver handles 10-bit→h264_vaapi fine. (My earlier "missing `format=nv12`" lead was wrong — the real chain converts to 8-bit via `colorspace=...:format=yuv420p` before upload; `format=nv12|p010le|vaapi` negotiates nv12 from the 8-bit input.) ### What actually happened - The container was **recreated at 20:15:11** for the `26.10.0` deploy. - Immediately after, a **burst of 15 channels (101–125) cold-started** (client/EPG refresh hitting everything at once). - On the *freshly re-initialized* phoenix iGPU render node under that burst, `hwupload` failed with **`-22`** only for the **10-bit** channels (102/107 — `p010`, ~2× the surface memory of 8-bit `nv12`); 8-bit channels succeeded in the same window. **8** failures total, all between **20:17:13 and 20:27:18**. - It **cleared on its own.** Since ~20:28 there have been **zero** `-22`. ### Verified fixed now - ch107 tune at 22:47 → `segments 1/1`, clean. - ch102 + ch107 **concurrently** at 22:51 → both `segments 1/1`, clean. - **8-channel concurrent burst** (100/101/103/105/110/120/122/123) at 22:53 → all `segments 1/1`, clean (startup latency rose to 3.6–8.1s under load, but no failures). The concurrency burst alone, on a settled GPU, does **not** reproduce it — the trigger required the burst **plus** the fresh-post-recreate GPU state. ### Done-when - [x] Root cause identified — transient VAAPI surface-allocation failure on the freshly-recreated render node under a concurrent tune burst; hits 10-bit (larger `p010` surfaces) first. ErsatzTV's generated command is correct. - [x] Channels 102 and 107 tune successfully on prod — verified, incl. concurrently. - [ ] ~~pipeline bug → test~~ — N/A (not a pipeline bug). - [ ] Host/driver → link a `timothy/server-management` issue (filing next). Recommend closing as **diagnosed / host-transient, self-resolved**, with a server-management follow-up to watch whether the next container recreation reproduces the post-deploy VAAPI transient. Optional ErsatzTV hardening (separate enhancement): fall back to software encode after N consecutive VAAPI `hwupload` failures so a transient GPU state can't leave a channel showing error cards for ~13 min.
Author
Owner

Closing — diagnosed as a self-resolved host VAAPI transient (not ErsatzTV code)

What was done: Captured the exact prod ffmpeg command from /proc and reproduced it verbatim on the real GPU (standalone, faithfully with seek+logo+hls, 4× concurrent, and an 8-channel burst) — all succeed. Confirmed 102/107 tune cleanly now, including concurrently.

Root cause: Transient VAAPI surface-allocation failure on the freshly re-initialized phoenix iGPU render node during the post-deploy tune burst (container recreated 20:15:11 → ~15 channels cold-started at once). Hit only the 10-bit channels (larger p010 surfaces); 8 failures, all 20:17–20:27; self-resolved by ~20:28, zero recurrence since. ErsatzTV's generated filtergraph is correct.

Files changed: none (not an app bug).

Done-when:

  • Root cause identified (host transient; command correct)
  • Channels 102 and 107 tune successfully on prod
  • pipeline bug → test — N/A
  • Host/driver → linked timothy/server-management#631 (verify on next container recreate)
  • Adversarial review — waived: no code diff; disposition is host-side, backed by verbatim-command reproduction + live re-verification. (Per the independent-review skip rubric, an empty-diff/no-server-state-change change is exempt with a stated reason.)

Deferred (not filed, per decision): optional ErsatzTV resilience enhancement — software-encode fallback after N consecutive VAAPI hwupload failures so a transient GPU state can't leave a channel showing error cards for ~13 min. Can be opened later if this recurs.

Follow-up: timothy/server-management#631 owns the host-side verification. Reopen this if the -22 recurs on prod outside a fresh-recreate window.

## Closing — diagnosed as a self-resolved host VAAPI transient (not ErsatzTV code) **What was done:** Captured the exact prod ffmpeg command from `/proc` and reproduced it verbatim on the real GPU (standalone, faithfully with seek+logo+hls, 4× concurrent, and an 8-channel burst) — all succeed. Confirmed 102/107 tune cleanly now, including concurrently. **Root cause:** Transient VAAPI surface-allocation failure on the freshly re-initialized phoenix iGPU render node during the post-deploy tune burst (container recreated 20:15:11 → ~15 channels cold-started at once). Hit only the 10-bit channels (larger `p010` surfaces); 8 failures, all 20:17–20:27; self-resolved by ~20:28, zero recurrence since. ErsatzTV's generated filtergraph is correct. **Files changed:** none (not an app bug). **Done-when:** - [x] Root cause identified (host transient; command correct) - [x] Channels 102 and 107 tune successfully on prod - [x] ~~pipeline bug → test~~ — N/A - [x] Host/driver → linked **timothy/server-management#631** (verify on next container recreate) - [x] Adversarial review — waived: no code diff; disposition is host-side, backed by verbatim-command reproduction + live re-verification. (Per the independent-review skip rubric, an empty-diff/no-server-state-change change is exempt with a stated reason.) **Deferred (not filed, per decision):** optional ErsatzTV resilience enhancement — software-encode fallback after N consecutive VAAPI `hwupload` failures so a transient GPU state can't leave a channel showing error cards for ~13 min. Can be opened later if this recurs. **Follow-up:** timothy/server-management#631 owns the host-side verification. Reopen this if the `-22` recurs on prod outside a fresh-recreate window.
timothy removed the in-progress label 2026-07-19 23:01:03 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: timothy/ersatztv#478