[META] Codec improvement and additional support including AV1/H.265, Intel/AMD GPU encoding support about selkies-gstreamer

[ehfd]

AV1 encoders will come soon to NVIDIA and Intel. AV1 should be possible to be used as a decoding codec for major web browsers. This should be available to GStreamer soon and confirmed it is in the works.

[ehfd]

#42 implements vaapih264enc. However, for Intel GPUs this is only limited to LIBVA_DRIVER_NAME=i965, which means that the iHD driver is unable to be used. Moreover, performance with ximagesrc is very low with Xorg on Intel GPUs somehow.

vah264enc is going to come in GStreamer 1.21 or 1.22. This is a more modern implementation, and vaapih264enc will get frozen. Investigate this when it comes.

AMD's AMF API and Intel's Media Foundation API may also deliver better performance. Therefore, this issue is not being closed and more contribution is needed.

Note: https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/

Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/gstwebrtc_app.py
Anyone with GStreamer experience, or willing to learn can solve this.

Supporting GPU statistics is also needed. GPUtils only supports NVIDIA.

[ehfd]

Relevant:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1489
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1494

[ehfd]

More discussions are going on related to the VAAPI plugin in #42.

[maxpain]

Does selkies-gstreamer support AV1 codec for now?

[ehfd]

bce15ba

Thanks to austriangam3r:

got the amd hardware acceleration working with the newest gstreamer version, the segfaults happen when some property of the data channel is accessed in newer gstreamer versions
Change this in gstwebrtc_app.py and the segfaults disappear

[ehfd]

Does selkies-gstreamer support AV1 codec for now?

@maxpain Depends on the web browser. I'm trying to make updates.

[maxpain]

@ehfd Chrome supports WebRTC AV1 encoder starting from v90 beta

[ehfd]

We need this. https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1489

AV1 support tends to be meaningless without support from NVIDIA.

[ehfd]

vah264enc is added in commit 736292c as well as the upgrade to GStreamer 1.22.

The msdk plugins for Intel GPUs and amf plugins for AMD GPUs may provide additional improvements.

https://gstreamer.freedesktop.org/documentation/msdk/msdkh264enc.html
https://gstreamer.freedesktop.org/documentation/msdk/msdkh265enc.html
https://gstreamer.freedesktop.org/documentation/msdk/msdkav1enc.html
https://gstreamer.freedesktop.org/documentation/msdk/msdkvp9enc.html

https://gstreamer.freedesktop.org/documentation/amfcodec/amfh264enc.html
https://gstreamer.freedesktop.org/documentation/amfcodec/amfh265enc.html

[ehfd]

The above addition is NOT tested, thus everyone is encouraged to feedback.

[ehfd]

subprojects/gst-plugins-bad/sys/amfcodec/meson.build:53:4: ERROR: Problem encountered: amf plugin supports only Windows

AMD users on Linux are stuck with va until the Vulkan backend is implemented to GStreamer.

[ehfd]

• Support of qsv and openh264 (at least from GStreamer) is open for contribution (GStreamer build pipeline is already ready, just the addition of the GST-Python pipelines).
  qsv provides an Intel-specific plugin for video encoding, and is an equivalent better reimplementation of msdk.
  I myself will be procrastinating the above for now because the equivalent functional counterpart va (PLEASE TEST IF YOU HAVE AN INTEL OR AMD GPU) is available, and some waiting time is important for making sure the new release is without bugs.

• openh264 has a relaxed license compared to x264 and would be appealing to corporations who do not want GPL programs in the pipeline.

• Moreover, it is required to investigate the usage of video/x-raw(memory:VAMemory) in vah264enc just like how video/x-raw(memory:GLMemory) or video/x-raw(memory:CUDAMemory) is used in NVIDIA GPUs.

• Support new NVENC elements over the legacy nvh264enc encoder: https://discourse.gstreamer.org/t/nvcodec-nvenc-nvidia-deprecates-support-for-old-videocodec-sdk-h-264-hevc-encoder-presets-with-driver-r550-in-q124/182/5

• Since H.265 is not supported by a bulk of web browsers in WebRTC (note: things are changing, look below), nvav1enc is also open for prioritized contribution.

This is an example of how you are supposed to contribute (just the src/selkies_gstreamer/gstwebrtc_app.py):
736292c#diff-0f90616a6ae1fe3eafe009fbdcc1de77f509f538eb735d2262f277a078f10e3d

I will implement these after 6-12 months if there are no contributions.

Anything other than these will be done after GStreamer 1.24 is out.

[ehfd]

https://webrtc.googlesource.com/src/+/972f28335a7f989ee1ffccb7c64fda1f29e09747

H.265 coming to Chromium?

WebKit/WebKit#15494

And an apparent solution to the packetization issue in Apple.

[ehfd]

Note to self regarding x264enc - option added to the pipeline:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3034

[ehfd]

Encoder settings: https://github.com/LizardByte/Sunshine/blob/29a1b1539fb2e0e82d02e521954b163c6d9d55d1/src/video.cpp#L568-L994