raspivid support about mediadevices

[alexey-kravtsov]

@lumenier1 Hi! In my project I'm using hardware encoding on Raspberry Pi 3 with GStreamer - just try x264enc.
For gstreamer-send example you can change webrtc.DefaultPayloadTypeVP8 to webrtc.DefaultPayloadTypeH264 and webrtc.VP8 to webrtc.H264 in main.go. However there are ~3 sec latency in transmitted video, I've fixed it in pion/example-webrtc-applications#45

[djmaze]

@alexey-kravtsov Does x264enc really support hardware acceleration on the Raspberry Pi? It works for me but leads to a CPU load of about 1.5 to 2 (in my test setup).

I switched to using omxh264enc in my test app. That seems to have decreased the load down to less than 0.5.

[alexey-kravtsov]

@djmaze actually I'm not 100% sure about x264enc. I've tried omxh264enc - it really loads CPU far less, but I have ~10s startup delay with it. Could you please share your pipeline with omxh264enc or test app?

[djmaze]

There is almost no startup delay for me.

The complete video pipeline in the application on the Pi looks like this:

v4l2src ! video/x-raw, width=640, height=480, framerate=15/1 ! queue ! video/x-raw,format=I420 ! omxh264enc control-rate=1 target-bitrate=600000 ! h264parse config-interval=3 ! video/x-h264,stream-format=byte-stream | appsink name=appsink

In order to see the difference, here is the pipeline when running on x64 machines:

v4l2src ! video/x-raw, width=640, height=480, framerate=15/1 ! queue ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency key-int-max=20 ! video/x-h264,stream-format=byte-stream ! appsink name=appsink

EDIT: Maybe the SPS insertion every 3 seconds (using the h264parse element) is the crucial missing part in your pipeline? I read about that somewhere, so I added it, which made my pipeline work at all.

[alexey-kravtsov]

Thank you! I've missed h264parse in my pipeline, so I don't know why it was working at all before :) I've slightly modified your example for my project (added videoconvert before encoder), and now it works fine for me
v4l2src ! video/x-raw, width=640, height=480 ! videoconvert ! video/x-raw,format=I420 ! omxh264enc control-rate=1 target-bitrate=600000 ! h264parse config-interval=3 ! video/x-h264,stream-format=byte-stream ! appsink name=appsink

[djmaze]

Great to hear it works for you. Was quite some trial & error and searching around the nets, because no one explains this kind of stuff to you..

Btw, I also used the videoconvert when running on x64 but it at least seemed not necessary when running on a Pi 3 for me.

[lherman-cs]

@lumenier1 now you can use mediadevices to do hardware encoding on a raspberry pi 3, https://github.com/pion/mediadevices/tree/master/examples/webrtc.

Here's the snippet:

package main

import (
	"fmt"

	"github.com/pion/mediadevices"
	"github.com/pion/mediadevices/examples/internal/signal"
	"github.com/pion/mediadevices/pkg/frame"
	"github.com/pion/mediadevices/pkg/prop"
	"github.com/pion/webrtc/v3"

	"github.com/pion/mediadevices/pkg/codec/mmal"
	_ "github.com/pion/mediadevices/pkg/driver/camera"     // This is required to register camera adapter
)

func main() {
	config := webrtc.Configuration{
		ICEServers: []webrtc.ICEServer{
			{
				URLs: []string{"stun:stun.l.google.com:19302"},
			},
		},
	}

	// Wait for the offer to be pasted
	offer := webrtc.SessionDescription{}
	signal.Decode(signal.MustReadStdin(), &offer)

         // mmal package uses Raspberry Pi's hardware encoder.
         // Reference: https://github.com/raspberrypi/userland/tree/master/interface/mmal 
	mmalParams, err := mmal.NewParams()
	if err != nil {
		panic(err)
	}
	mmalParams.BitRate = 500_000 // 500kbps

	codecSelector := mediadevices.NewCodecSelector(
		mediadevices.WithVideoEncoders(&mmalParams),
	)

	mediaEngine := webrtc.MediaEngine{}
	codecSelector.Populate(&mediaEngine)
	api := webrtc.NewAPI(webrtc.WithMediaEngine(&mediaEngine))
	peerConnection, err := api.NewPeerConnection(config)
	if err != nil {
		panic(err)
	}

	// Set the handler for ICE connection state
	// This will notify you when the peer has connected/disconnected
	peerConnection.OnICEConnectionStateChange(func(connectionState webrtc.ICEConnectionState) {
		fmt.Printf("Connection State has changed %s \n", connectionState.String())
	})

	s, err := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
		Video: func(c *mediadevices.MediaTrackConstraints) {
			c.FrameFormat = prop.FrameFormat(frame.FormatI420)
			c.Width = prop.Int(640)
			c.Height = prop.Int(480)
		},
		Codec: codecSelector,
	})
	if err != nil {
		panic(err)
	}

	for _, track := range s.GetTracks() {
		track.OnEnded(func(err error) {
			fmt.Printf("Track (ID: %s) ended with error: %v\n",
				track.ID(), err)
		})

		_, err = peerConnection.AddTransceiverFromTrack(track,
			webrtc.RtpTransceiverInit{
				Direction: webrtc.RTPTransceiverDirectionSendonly,
			},
		)
		if err != nil {
			panic(err)
		}
	}

	// Set the remote SessionDescription
	err = peerConnection.SetRemoteDescription(offer)
	if err != nil {
		panic(err)
	}

	// Create an answer
	answer, err := peerConnection.CreateAnswer(nil)
	if err != nil {
		panic(err)
	}

	// Sets the LocalDescription, and starts our UDP listeners
	err = peerConnection.SetLocalDescription(answer)
	if err != nil {
		panic(err)
	}

	// Output the answer in base64 so we can paste it in browser
	fmt.Println(signal.Encode(answer))
	select {}
}

[G2G2G2G]

@lherman-cs why is it using google's servers?