Vulkan Video Decoding: Usage Guide and FAQ

[philipl]

Vulkan Video Decoding: Usage Guide and FAQ

On the 28th of May, we reached the significant milestone of finally merging all the required functionality into ffmpeg, libplacebo, and mpv to do end-to-end Vulkan video decoding and presentation. Although the functionality is now all there, we still have a complex landscape in terms of what is supported on what hardware and with what drivers. This document attempts to lay out all the requirements and limitations, so you have a chance of successfully using the feature.

Why should I even care about Vulkan video decoding?

It's a fair question. Right now, the actual functionality you gain access to is not terribly different from what you would get with existing video decoding APIs that ffmpeg and mpv already support. However, Vulkan video decoding has the potential to be a credible cross-vendor, cross-platform API that is well supported, and can work efficiently with Vulkan based filtering and post-processing. In the short term, there will be rough edges, and only a narrow set of supported codecs (only H.264 and H.265 are standardised today, with AV1 still a work in progress), but in the future, it will hopefully be the most performant and capable way to do hardware decoding and processing.

Software Requirements

We have finally reached the point where all the required components have made official releases. Ensure you are running with the following releases or newer:

• ffmpeg: 6.1
• libplacebo: v6.292.0
• mpv: 0.37.0

These requirements mean you will likely have to compile everything for yourself (especially on Linux), but Shinchiro's windows builds appear to be functional.

Hardware and Driver Requirements

• Intel on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the ANV driver
  • An MR is open for support in the hasvk driver
• AMD on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the RADV driver
  • The AMDVLK driver is not known to support video decoding
• Nvidia on Linux or Windows
  • Drivers >= 535.xx
  • Any GPU supported by those drivers
• AMD on Windows
  • Drivers >= 23.9.3
  • RDNA 1/2/3 GPUs
• Intel on Windows
  • Unclear and untested
  • There is evidence that drivers from at least 101.4311 expose the video decoding extensions
  • No idea what hardware is supported
• MacOS (for the record)
  • Although Vulkan on MacOS is supported via MoltenVk, this does not include the decoding extensions, and probably never will. Just keep using videotoolbox.

Usage

The basic command line arguments:

mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan

Depending on your system configuration, you may additionally need to force the right gpu-context:

• Windows: --gpu-context=winvk
• Linux X11: --gpu-context=x11vk
• Linux Wayland: --gpu-context=waylandvk

Capabilities

• Global
  • The Vulkan Video Decoding specs only cover H.264, H.265, and AV1. Older codecs will require you to use your traditional hardware decoding API (VAAPI, nvdec, dx11va, etc) for now, and possibly forever - depending on whether effort is put into exposing older codecs.
• Intel
  • You must set ANV_VIDEO_DECODE=1 in your environment to expose video decoding
  • Both H.264 and H.265 are supported
  • VK_EXT_descriptor_buffer support, required for Vulkan filters, has been merged but no release includes it yet
• AMD
  • You must set RADV_PERFTEST=video_decode in your environment to expose video decoding
  • Both H.264 and H.265 are supported
  • Vulkan filters are supported
• Nvidia
  • Both H.264 and H.265 are supported
  • AV1 is supported with > 550.54.14 drivers
  • Vulkan filters are supported

[Jules-A]

AMD on Windows appears to be 5000/6000 series according to https://www.amd.com/en/support/kb/release-notes/rn-rad-win-22-11-2-vlk-video-code-decode (possibly 7000 works?). It doesn't work on Polaris: 0.339][e][ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension!

[llyyr]

After bc28dce, you need to patch your Mesa with this PR to get AV1 decoding on radv.

[aufkrawall]

Reliably crashes driver for me with this sample: https://0x0.st/HbHJ.mkv
Other files seem to work for whatever reason (hwdec beeing used confirmed).
Latest git-master of mpv, libplacebo, ffmpeg and radv on 6700 xt.

[philipl]

Sounds like you probably need to file a mesa issue for that - and even if it was an mpv issue, please open a separate one. This FAQ isn't the place to report it.

[cyanreg]

@aufkrawall AMD currently needs https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227 to avoid crashes, which also fixes quite a lot of bugs

[alakittty]

 (+) Video --vid=1 (*) (hevc 3828x1592 23.976fps)
 (+) Audio --aid=1 --alang=eng (*) (eac3 6ch 48000Hz)
     Subs  --sid=1 --slang=eng (subrip)
     Subs  --sid=2 --slang=eng 'SDH' (subrip)
     Subs  --sid=3 --slang=fre (subrip)
[vo/gpu-next/libplacebo] vk->CreateDevice(vk->physd, &dinfo, PL_VK_ALLOC, &vk->dev): VK_ERROR_OUT_OF_HOST_MEMORY (../../../../../src_packages/libplacebo/src/vulkan/context.c:1308)
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video

what does this error mean, my device has amd and i used the following config
mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
mpv was working fine with gpu-next and gpu-api=vulkan till yesterdays' shinchiro release. todays release gave me this error
maybe @shinchiro needs to bump libplacebo and ffmpeg version as per required?

[Jules-A]

AMD on Windows
Special 22.11.2 drivers with video decoding support
Hardware supported by those drivers (documented as 5000/6000 series. Apparently not newer 7000 series)

I don't know if that specific driver is required, they may have been included in 23.2.1 where they added other extensions, the driver notes aren't clear. 7000 series wasn't launched when the 22-11-2 Vulkan video drivers were released so that driver doesn't officially support them. If they did include the video extensions in newer drivers then they may support 7000 series but someone with those GPUs will need to test.

[cyanreg]

Yeah, I've reported to nvidia months ago that their descriptor buffer implementation is broken, but they haven't fixed it yet.
If you want filters working, ask nvidia to fix it.

[philipl]

I found this video will white screen when using ffmpeg vulkan decoding, SW decoding or nvpro vk_video_decoder have no problem, I don't know if this is also NVIDIA's problem (also already reported to NVIDIA)
test.mp4

Github claims you attached an audio-only file...

[llyyr]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

[philipl]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

It's an HEVC file. My experience on ANV is that any HEVC playback pegs the GPU at 100%

[philipl]

Also:

[ffmpeg/video] hevc: First slice in a frame missing.

This is a highly questionable sample.

[lavilao]

Will this work with hasvk in the future? I tried using it through wine (I dont want to have to compile everything) and got this error [ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension! even though the output of vulkaninfo is:

vulkaninfo | grep video                                                                     ✔  10s  
	VK_KHR_video_decode_h264                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 7
	VK_KHR_video_queue                          : extension revision 8
			videoCodecOperations:
			videoCodecOperations: count = 1

[CharlesMengCA]

NVIDIA 1650 Super, Tested on windows with: mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
5 videos played OK.

This video has problem: https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo/
Switched back to hwdec=nvdec, the problem is gone.

[24fpsDaVinci]

for fedora, you need to edit mesa.spec and rebuild with codecs enabled -D video-codecs=h264dec,h264enc,h265dec,h265enc,vc1dec, codecs is disabled by default.

rpmfusion-infra/fedy#110

https://themaister.net/blog/2023/01/05/vulkan-video-shenanigans-ffmpeg-radv-integration-experiments/

https://forum.endeavouros.com/t/compiling-mesa-22-2-with-codec-support-the-easy-way/30390

[llyyr]

This branch has been adapted for the new KHR extension, but it still doesn't work on NVIDIA.

It also doesn't work on radv's implementation of the protocol because there were changes that it hasn't updated for yet. Just wait for it to be merged

[Glog78]

#11739 (comment) << same happens with 550.40.53 (aka the current vulkan beta driver)

maybe the resulting coredump info helps a little -> https://paste.rs/4S1iz

[kkost]

FYI- VK_EXT_descriptor_buffer support has been merged into mesa (git/mainline) for Intel/anv, and vulkan filters appear to be working now.

[atohmdiy]

I tried to use vulkan video on my Manjaro linux system. Manjaro is stable repo and everything should be working. Everything is set as in the FAQ / Guide.
My system is a 7800x3d cpu and 6900 XT GPU.
Problem is i get these errors in the console for h264 and h265 playback :

[ffmpeg/video] hevc: Device does not support decoding hevc!
[ffmpeg/video] h264: Device does not support decoding h264!

I still get an image of course, but i assume that with this error decoding is done by the cpu.

[JoonasPitk]

I tried to use vulkan video on my Manjaro linux system. Manjaro is stable repo and everything should be working. Everything is set as in the FAQ / Guide. My system is a 7800x3d cpu and 6900 XT GPU. Problem is i get these errors in the console for h264 and h265 playback :

[ffmpeg/video] hevc: Device does not support decoding hevc! [ffmpeg/video] h264: Device does not support decoding h264!

I still get an image of course, but i assume that with this error decoding is done by the cpu.

Manjaro doesn't build their Mesa with patented codecs, hence why you're missing H264 and HEVC codecs. You can check which codecs are supported for your hardware with vainfo.

[cyanreg]

AV1 decoding now works with the following branch on both RADV and Nvidia: https://github.com/cyanreg/FFmpeg/tree/av1dec
Testing appreciated.

[rubyFeedback]

For me personally, the whole vulkan situation has been quite painful - not sure if this is nvidia related (I have a cheap, crappy nvidia card) or not. High quality information is in general highly appreciated, because right now, even after reading the initial intro on top of this thread, I still don't really know how to solve the problems I have. I can no longer play anything right now via mpv, e. g. getting errors such as

[vo/gpu/drm] Failed to set up VT switcher. Terminal switching will be unavailable.
[vo/gpu/drm] Failed to create DRM atomic context, no DRM Atomic support
[vo/gpu-next/drm] Can't handle VT release - signal already used
[vo/gpu-next/drm] Failed to set up VT switcher. Terminal switching will be unavailable.
[vo/gpu-next/drm] Failed to create DRM atomic context, no DRM Atomic support

This is all quite painful because I know it all works in theory (default manjaro proprietary
nvidia on the same computer and everything works fine).

[temis01]

AV1 decoding now works with the following branch on both RADV and Nvidia: https://github.com/cyanreg/FFmpeg/tree/av1dec Testing appreciated.

Hi !

Could you tell me how to build mpv with your ffmpeg repo ?
I am using shinchiro's clang procedure and successfully build mpv with the official git.
Is it even possible with another branch of ffmpeg?
Thanks

[temis01]

I tried changing the repo url ... but I missed the "GIT_TAG av1dec" + "GIT_REMOTE_NAME origin" then it failed at ffmpeg prefix.
Thanks

[temis01]

@cyanreg ffmpeg repo does not work for me

[cyanreg]

Try https://github.com/cyanreg/FFmpeg/tree/av1dec2
I'll have more time to test in a few hours

[temis01]

getting same old

av1dec2.txt

[cyanreg]

AV1 should work on NVIDIA now with git master of ffmpeg.

[temis01]

@cyanreg using your master ffmpeg repo + shinchiro git :
had to comment out --enable-libdvdnav + --enable-libdvdread in ffmprg.cmake cause configure said options are unknown !
Then, getting a lot of cr*p at build step :

ffmpeg-build-err.log

[temis01]

The patch worked for me.
mpv 0.38
Windows 11 22631.3593
RTX 3050 8Go + drv 552.44

#13375

[wyup]

vulkan renderer working good (except green screen with hwdec=vulkan) on i5-13500 iGPU (UHD 770), Win11 x64, shinchiro build v3, latest intel driver 101.5518, Vulkan api version 1.3.280, conformance 1.3.6.3.
Same timing performance to d3d11 on with gpu-next, gpu-api=vulkan, gpu-context=winvk,tone-mapping=bt.2446a, profile=gpu-hq, deband. It doesn't crash as before. I hope hwdec=vulkan works aswell soon.

mpv N-115189-g0664cbd73 (May 13) libplacebo version: v7.349.0 (v6.338.0-130-gbf1855c-dirty)

Good job!