README.rpi.md

MDK: Multimedia Development Kit

Use generic linux sdk package for raspberry pi3/4 with modern RaspberryPi OS

Changelog

API

Features

• Simple and powerful API set
• Cross platform: Windows(x86, arm), UWP, Linux, macOS, Android, iOS, tvOS, visionOS, Raspberry Pi
• Hardware accelerated decoders
• 0-copy GPU rendering for all platforms and all renderers(Vulkan is WIP.)
• Dynamic OpenGL
• OpenGL, D3D11, D3D12, Vulkan and Metal rendering w/ or w/o user provided context
• Integrated with any platform native ui apps, gui toolkits or other apps via OpenGL, D3D11/12, Vulkan and Metal (WinUI3, OBS, Flutter, Qt, SDL, GLFW, SFML, .NET Avalonia etc.) easily
• HDR display, HDR to SDR and SDR to HDR tone mapping. You can use HDR display in Qt6(6.6+ for macOS, 6.x for windows), OBS Studio and more.
• Dolby Vision rendering, including Profile 5
• Seamless/Gapless media and bitrate switch for any media
• Optimized Continuous seeking. As fast as mpv, but much lower cpu, memory and gpu load. Suitable for timeline preview
• Subtitle rendering, including ass, plain text, bitmap, closed caption
• Smart FFmpeg runtime, dynamic load, binary compatible with 4.0~7.x
• Professional codecs: GPU accelerated HAP codec rendering, Blackmagic RAW, R3D

About SDK for Legacy Raspberry Pi

SDK is cross built by clang 17 with

• cmake toolchain file https://github.com/wang-bin/cmake-tools/blob/master/raspberry-pi.clang.cmake
• sysroot: https://sourceforge.net/projects/avbuild/files/raspberry-pi/raspberry-pi-sysroot.tar.xz/download
• ffmpeg: https://sourceforge.net/projects/avbuild/files/raspberry-pi/ffmpeg-master-raspberry-pi-clang-lite.tar.xz/download
• libc++ 17

SDK can be used by any C or C++11 compiler, e.g. g++, clang

pi 4: ALSA_DEVICE=sysdefault

clang >= 9 result dso can not be loaded on rpi1, maybe there is a bug in lld

Runtime Requirements

• gapless playback hardware decoders may requires 256M GPU memory

Supported Graphics APIs:

• OpenGL
• OpenGL ES2/3

Supported Decoders:

• mmal: builtin mmal decoder implementation
• MMAL: FFmpeg mmal decoder implementation
• V4L2: or "FFmpeg:hwaccel=v4l2m2m"

Examples

In legacy driver environment, hardware decoder (MMAL, mmal) supports zero copy rendering in GLES2 and has the best performance

• legacy driver gles2: ./window -c:v MMAL test.mp4

In mesa vc4/6 driver environment(fake/full kms), hardware decoder is available, but zero copy rendering is not

• vc4/6 egl+es2: LD_PRELOAD=libX11.so.6 ./window -c:v MMAL test.mp4 # if not link against libX11(except weak) or libpulse
• vc4/6 glx: GL_ES=0 LD_PRELOAD=libGL.so.1 ./window -c:v MMAL test.mp4 # if not link against libGL or libOpenGL
• vc4/6 wayland: ./window -c:v MMAL test.mp4 # assume weston is running in x11 or CLI mode via weston-launch
• vc4/6 gbm: ./window -surface gbm -c:v MMAL test.mp4 # assume weston is running # in CLI mode

Tested on rpi1 and rpi3.

Note: the latest sdk links against libGL.so.1, so no need to set LD_PRELOAD.

Use in CMake Projects

	include(mdk-sdk-dir/lib/cmake/FindMDK.cmake)
	target_link_libraries(your_target PRIVATE mdk)

Qt qmake project

include($$MDK_SDK_DIR/mdk.pri)

Live streams (RTSP, RTMP etc.) low latency

    player.setProperty("avformat.fflags", "+nobuffer");
    player.setProperty("avformat.fpsprobesize", "0");

Source code:

• some examples using mdk sdk
• OBS Studio plugin
• QtMultimedia plugin
• MFT decoder module
• dav1d decoder module

Copyright (c) 2016-2024 WangBin(the author of QtAV) Free for opensource softwares, non-commercial softwares, flutter, QtAV donors and contributors.