Channels 102/107 repeatedly fail to tune: h264_vaapi hwupload -22 / exit 234 on prod
#478
Closed
opened 2026-07-19 20:59:54 +02:00 by timothy
·
4 comments
No Branch/Tag Specified
main
renovate/dotnet-monorepo
renovate/meziantou.analyzer-3.x
renovate/lucene.net
renovate/sqlitepclraw.bundle_e_sqlite3-3.x
renovate/cliwrap-3.x
913-scripted-playout-401
release/v26.15.0-notes
issue-806-guard-populations
scratch/767b-poisoned
scratch/767b-control
release/v26.14.0-notes
release/v26.14.0
docs/510-skill-logo-bug-policy
fix/510-watermark-resolution-policy
fix/629-verdict-classifier-falseopens
fix/609-decisions-edit-token-scope
issue-135-clear-to-none
release/v26.12.0-notes
fix/409b-lastscan-api-parity
fix/401-updatechannel-mirror-422
fix/327-playlist-rename-validation
fix/410-scancancel-log-level
fix/409-447-librariesscreen-neverscanned
fix/338-zap-exit-code
fix/367-plex-budget-message
fix/310-debom-legacy-cs
ci/604-lane-rebalance
feat/388-design-mirror
feat/247-test-ownership
feat/247-primary-action
feat/357-player-owned-playback
feat/357-jellyfin-plugin-poc
fix/289-mcp-hardening
issue58-mcp
feat/244-channels-extract
ci/auto-bump-prod-compose
feat/multi-rerun-collections-api
feat/collections-api
feat/quick-wins
feat/185-docs-part2
feat/140-collections-screen
feat/146-channel-edit
feat/147-classic-ui-link
issue22-renovate-dashboard
feat/91-cutover
feat/63-composite-create
feat/65-library-browse
feat/85-epg
feat/86-schedule-editor
feat/109-dashboard-data
feat/99-session-tracking
fix/dockerfile-node-tag
feat/59-spa-foundation
docs/59-ui-redesign-brief
feat/102-json-guide
feat/111-schedule-durations
feat/104-artwork-upload
feat/103-media-sources-api
feat/playouts-read-api
feat/108-health-api
feat/105-picker-list-endpoints
issue-97-channel-state-api
issue42-jellyfin-musicvideos
issue46-rest-api-error-contract
dependabot/nuget/ErsatzTV.FFmpeg.Tests/multi-d307a2e06f
qsv-improvements
hdr-vulkan-cuda-test
v26.16.0
v26.15.0
v26.14.0
v26.13.0
v26.12.0
v26.11.0
v26.10.0
v26.9.0
v26.8.0
v26.7.0
blazor-final
v26.6.0
v26.5.0
v26.4.0
v26.3.1
v26.3.0
v26.2.0
v26.1.1
v26.1.0
v25.9.0
v25.8.0
v25.7.1
v25.7.0
v25.6.0
v25.5.0
v25.4.0
v25.3.1
v25.3.0
v25.2.0
v25.1.0
v0.8.8-beta
v0.8.7-beta
v0.8.6-beta
v0.8.5-beta
v0.8.4-beta
v0.8.3-beta
v0.8.2-beta
v0.8.1-beta
v0.8.0-beta
v0.7.9-beta
v0.7.8-beta
v0.7.7-beta
v0.7.6-beta
v0.7.5-beta
v0.7.4-beta
v0.7.3-beta
v0.7.2-beta
v0.7.1-beta
v0.7.0-beta
v0.6.9-beta
v0.6.8-beta
v0.6.7-beta
v0.6.6-beta
v0.6.5-beta
v0.6.4-beta
v0.6.3-beta
v0.6.2-beta
v0.6.1-beta
v0.6.0-beta
v0.5.8-beta
v0.5.7-beta
v0.5.6-beta
v0.5.5-beta
v0.5.4-beta
v0.5.3-beta
v0.5.2-beta
v0.5.1-beta
v0.5.0-beta
v0.4.5-alpha
v0.4.4-alpha
v0.4.3-alpha
v0.4.2-alpha
v0.4.1-alpha
v0.4.0-alpha
v0.3.8-alpha
v0.3.7-alpha
develop
v0.3.6-alpha
v0.3.5-alpha
v0.3.4-alpha
v0.3.3-alpha
v0.3.2-alpha
v0.3.1-alpha
v0.3.0-alpha
v0.2.5-alpha
v0.2.4-alpha
v0.2.3-alpha
v0.2.2-alpha
v0.2.1-alpha
v0.2.0-alpha
v0.1.5-alpha
v0.1.4-alpha
v0.1.3-alpha
v0.1.2-alpha
v0.1.1-alpha
v0.1.0-alpha
v0.0.62-alpha
v0.0.61-alpha
v0.0.60-alpha
v0.0.59-alpha
v0.0.58-alpha
v0.0.57-alpha
v0.0.56-alpha
v0.0.55-alpha
v0.0.54-alpha
v0.0.53-alpha
v0.0.52-alpha
v0.0.51-alpha
v0.0.50-alpha
v0.0.49-prealpha
v0.0.48-prealpha
v0.0.47-prealpha
v0.0.46-prealpha
v0.0.45-prealpha
v0.0.44-prealpha
v0.0.43-prealpha
v0.0.42-prealpha
v0.0.41-prealpha
v0.0.40-prealpha
v0.0.39-prealpha
v0.0.38-prealpha
v0.0.37-prealpha
v0.0.36-prealpha
v0.0.35-prealpha
v0.0.34-prealpha
v0.0.33-prealpha
v0.0.32-prealpha
v0.0.31-prealpha
v0.0.30-prealpha
v0.0.29-prealpha
v0.0.28-prealpha
v0.0.27-prealpha
v0.0.26-prealpha
v0.0.25-prealpha
v0.0.24-prealpha
v0.0.23-prealpha
v0.0.22-prealpha
v0.0.21-prealpha
v0.0.20-prealpha
v0.0.19-prealpha
v0.0.18-prealpha
v0.0.17-prealpha
v0.0.16-prealpha
v0.0.15-prealpha
v0.0.14-prealpha
v0.0.13-prealpha
v0.0.12-prealpha
v0.0.11-prealpha
v0.0.10-prealpha
v0.0.9-prealpha
v0.0.8-prealpha
v0.0.7-prealpha
v0.0.6-prealpha
v0.0.5-prealpha
v0.0.4-prealpha
v0.0.3-prealpha
v0.0.2-prealpha
v0.0.1-prealpha
Labels
Clear labels
ad-hoc
api
bug
ci-cd
content
dependencies
enhancement
frontend
in-progress
jellyfin
parked
priority: high
priority: low
priority: medium
review
security
One-off / ad-hoc work not tracked by a dedicated issue
REST API / HTTP endpoints
Something isn't working
Build, test, deploy pipeline
Channel content / schedules / playlists
Dependency updates (Renovate)
New feature or improvement
ChicoryTV React SPA frontend
Claimed by an active session — do not pick up
Jellyfin tuner / IPTV integration
Excluded from automatic queue pickup; work only when explicitly selected
Adversarial review finding
Security / vulnerability fix
Milestone
No items
No Milestone
Projects
Clear projects
No projects
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: timothy/ersatztv#478
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found incidentally while pulling prod logs for #473 (2026-07-19, prod
26.10.0). Currently firing, unrelated to #473's missing-media class.Symptom
Channels 102 and 107 repeatedly die mid-transcode:
Channel 102 shows the same signature with exit code 183. Observed at 20:17:13, 20:17:25, 20:20:53, 20:21:05 (ch102) and 20:25:52, 20:26:33, 20:26:55, 20:27:18 (ch107) — i.e. every tune attempt, not an occasional flake.
Notes
amdgpu: os_same_file_descriptionline is a warning, not the cause. The actual failure ishwuploadfailing with-22 (Invalid argument), after which the VAAPI encoder never opens and ffmpeg writes nothing.Media item [27008] on channel 107 transcoded at 1.38x (NOT throttled) which may not be fast enough to support playback.HlsSessionWorker's failure path renders an error card and the next cold-start reportssegments 1/1. So the viewer sees an error card rather than a black screen, and the channel keeps limping.hwaccel vaapiin the same window, so this is not "VAAPI is broken" globally — it looks specific to these channels' content or FFmpeg profile (a filter-chain/pixel-format combination that fails to hwupload).Suggested first steps
hwupload -22trigger.ErsatzTV.FFmpegpipeline) or a host/driver limitation; if the latter it belongs intimothy/server-management, not here.Do not fold this into #473 — different class entirely (that one is a dead media reference; this is a live file the transcoder can't process).
Done-when
ErsatzTV.FFmpeg.Tests(asserts the rendered arg string)timothy/server-managementissueClaiming this (Claude Code orchestrator session, 2026-07-19).
Plan: diagnosis-first per the issue's suggested steps — this is explicitly "pipeline bug vs. host/driver" and I won't propose a fix until the root cause is pinned. Starting by pulling from prod:
hwupload -22trigger).hwuploadis malformed.If it's a host/driver limitation I'll close here with a linked
timothy/server-managementissue; if it's anErsatzTV.FFmpegfilter-graph construction bug I'll fix it with an arg-string test inErsatzTV.FFmpeg.Tests.Root cause narrowed — it's a pipeline (bit-depth) bug, not host/driver
Evidence gathered from prod (
26.10.0, DB + real hardware):MediaStream:hevc / main 10 / yuv420p10le, SDR bt709. Confirmed byffprobeon the real file too (pix_fmt=yuv420p10le).1080p VAAPI h264 aac— h264 8-bit output, VAAPI,VaapiDriver=RadeonSI, no tonemap) and all carryWatermarkId=1. The working channels (124/125/100) just happen to be on 8-bit content; 102/107 are on 10-bit. So the discriminator is purely 8-bit vs 10-bit source.hwdownload,format=p010le,hwupload → h264_vaapi(10-bit surface into the 8-bit encoder) → fails (encoder won't open / frame rejected). This is the prod failure class.hwdownload,format=p010le,format=nv12,hwupload → h264_vaapi(reduce to 8-bit before upload) → succeeds.scale_vaapi=format=nv12also succeeds.Conclusion:
h264_vaapion this box needs an 8-bit (nv12) surface. When a 10-bitp010frame reacheshwupload/the encoder, it dies (-22/ encoder-open failure). The fix is to guarantee a bit-depth reduction to nv12 before the finalhwuploadon the VAAPI software-overlay path (radeonsi forces software overlay for the watermark, so the frame is hw-downloaded to 10-bitp010, overlaid, and must be converted to nv12 before re-upload).Still pinning down: the exact command
VaapiPipelineBuilderemits for the 10-bit + watermark path, to see precisely where theformat=nv12is missing/mis-ordered (a deterministic repro test inErsatzTV.FFmpeg.Testsis in progress; it doubles as the regression test). Fix + test to follow.Done-when box 1 (root cause) ✓ effectively; it's a pipeline bug → box 3 (arg-string test) applies, not box 4 (host/driver).
Root cause: transient post-deploy VAAPI/DRM state under a tune burst — not an ErsatzTV bug. Already self-resolved.
The command ErsatzTV generates is correct
I captured the exact ffmpeg argv prod runs for ch107 (10-bit Curb), verbatim from
/proc/<pid>/cmdline:Ran this verbatim on the real GPU (Radeon 760M / radeonsi / ffmpeg 8.1.2) against the actual 10-bit files:
-ssseek + real logo + full hls muxer, 10s) → succeeds (2 segments)So the filtergraph is well-formed and the driver handles 10-bit→h264_vaapi fine. (My earlier "missing
format=nv12" lead was wrong — the real chain converts to 8-bit viacolorspace=...:format=yuv420pbefore upload;format=nv12|p010le|vaapinegotiates nv12 from the 8-bit input.)What actually happened
26.10.0deploy.hwuploadfailed with-22only for the 10-bit channels (102/107 —p010, ~2× the surface memory of 8-bitnv12); 8-bit channels succeeded in the same window. 8 failures total, all between 20:17:13 and 20:27:18.-22.Verified fixed now
segments 1/1, clean.segments 1/1, clean.segments 1/1, clean (startup latency rose to 3.6–8.1s under load, but no failures).The concurrency burst alone, on a settled GPU, does not reproduce it — the trigger required the burst plus the fresh-post-recreate GPU state.
Done-when
p010surfaces) first. ErsatzTV's generated command is correct.pipeline bug → test— N/A (not a pipeline bug).timothy/server-managementissue (filing next).Recommend closing as diagnosed / host-transient, self-resolved, with a server-management follow-up to watch whether the next container recreation reproduces the post-deploy VAAPI transient. Optional ErsatzTV hardening (separate enhancement): fall back to software encode after N consecutive VAAPI
hwuploadfailures so a transient GPU state can't leave a channel showing error cards for ~13 min.Closing — diagnosed as a self-resolved host VAAPI transient (not ErsatzTV code)
What was done: Captured the exact prod ffmpeg command from
/procand reproduced it verbatim on the real GPU (standalone, faithfully with seek+logo+hls, 4× concurrent, and an 8-channel burst) — all succeed. Confirmed 102/107 tune cleanly now, including concurrently.Root cause: Transient VAAPI surface-allocation failure on the freshly re-initialized phoenix iGPU render node during the post-deploy tune burst (container recreated 20:15:11 → ~15 channels cold-started at once). Hit only the 10-bit channels (larger
p010surfaces); 8 failures, all 20:17–20:27; self-resolved by ~20:28, zero recurrence since. ErsatzTV's generated filtergraph is correct.Files changed: none (not an app bug).
Done-when:
pipeline bug → test— N/ADeferred (not filed, per decision): optional ErsatzTV resilience enhancement — software-encode fallback after N consecutive VAAPI
hwuploadfailures so a transient GPU state can't leave a channel showing error cards for ~13 min. Can be opened later if this recurs.Follow-up: timothy/server-management#631 owns the host-side verification. Reopen this if the
-22recurs on prod outside a fresh-recreate window.