UVC Support about android-usb-gadget

[tejado]

Commit 5aaf88d fixes a few things in the UVC function. Now android-usb-gadget is able to create a UVC gadget which works with "uvc-gadget" (https://github.com/wlhe/uvc-gadget). Tested with Pixel 5 and self-build Kernel.

uvc-gadget

Compiling uvc-gadget for arm64

sudo apt install android-tools-adb android-tools-fastboot git
curl -O http://dl.google.com/android/repository/android-ndk-r12b-linux-x86_64.zip
unzip android-ndk-r12b-linux-x86_64.zip
./android-ndk-r12b/build/tools/make_standalone_toolchain.py --arch arm64 --install-dir ~/arm

git clone https://github.com/wlhe/uvc-gadget 
cd uvc-gadget 
~/arm/bin/clang -pie uvc-gadget.c

Transfer the "a.out" bin to your target device.

Using uvc-gadget

I was not able to get uvc-gadget to work with any of the camera video device files. It seems that v4l2 on Android has a different specification than on generic Linux.
But to test just the UVC functionaility, you can use the dummy image mode:

chmod +x a.out
./a.out -i /storage/emulated/0/test.mjpeg -u /dev/video5 -f 1 -r 1

With this, the Android device gets detected as a UVC Webcam under Windows 10 and plays the mjpeg video in a loop.
Example mjpeg file: https://filesamples.com/samples/video/mjpeg/sample_1280x720.mjpeg

USB Tool

I attached a signed interims version of android-usb-gadget with the fixed UVC function.
Don't add UVC as a function to the existing default gadget - create a new UVC gadget. The "adb" function in the default gadget does not work stable together with other functions and you also can't disable it, as Android will immediately enable it again.

Debug and release version: android-usb-gadget.zip

[luchfilip]

After successfully enabling the UVC function using USB Gadget App, I am not able to write to /dev/video3 with the main error being:

java.io.IOException: write failed: EINVAL (Invalid argument)
at libcore.io.IoBridge.write(IoBridge.java:654)
at java.io.FileOutputStream.write(FileOutputStream.java:401)
at java.io.FileOutputStream.write(FileOutputStream.java:379)

Steps I have taken:

1. Enable wifi debugging so we can use the USB for UVC only

In terminal On the Rooted Pixel 4A with custom ROM/Kernel (thanks @tejado 🙏 )

su
setprop service.adb.tcp.port 5555
stop adbd
start adbd

On my computer computer:

adb connect 192.168.1.XXX
adb -s 192.168.1.XXX:5555 shell

2. Configure UVC function in Android USB Gadget

Select +, UVC, enable it
Connect the Android device to PC

3. Set the permissions for newly created /dev/video3 stream so we can write to it from an Android app without the need for root access:

chmod 666 /dev/video3
chcon u:object_r:zero_device:s0 /dev/video3

Thanks again @tejado for the chcon command, saved me from having to deal with root access on the Android app.

4. On another Android app, grab the camera frames as following

I get 24 frames per second using imageAnalyzer from CameraX:

imageAnalyzer = ImageAnalysis.Builder()
.setTargetResolution(Size(640, 480))
.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)

convert the 3 YUV planes to ByteArray

imageAnalyzer.setAnalyzer(cameraExecutor) { image ->
// getting image in ImageProxy format
val imageByteArray = image.toByteArray()
...
}

Then based on UVC Specifications I build the header of the frame:

// UVC Video Payload Header has a payload size of 26 bytes, and the UVC MJPEG Payload Header has a payload size of 12 bytes.
val header = ByteBuffer.allocate(26)
val frameSize = image.width * image.height * ImageFormat.getBitsPerPixel(image.format) / 8
// End of Header: This field indicates the end of the UVC Video Payload Header and the start of the video data for the frame. It is a single byte that is set to the value 0x01.
 val EOH = 0x01
// Error Code: This field indicates the error code for the frame. It is a single byte that is set to the value 0x00.
val ERR = 0x00
// Still Image: This field indicates whether the video frame is a still image or a video frame.
// It is a single byte that is set to the value 0x00 if the frame is a video frame, or to the value 0x01 if the frame is a still image.
val STI = 0x00
// Reserved: This field is reserved for future use. It is a single byte that is set to the value 0x00.
val REST = 0x00
// Source ID: This field specifies the source of the video frame, such as a camera or a video file. It is a single byte that is assigned a unique value for each source.
val SRC = 0x00
// // Presentation Time Stamp: This field specifies the time at which the video frame was captured or generated.
// It is a 32-bit value that represents the number of 100-nanosecond intervals since a reference time, such as the start of the video stream or the beginning of a recording.
val PTS = (System.currentTimeMillis() - referenceTime) * 10000
// // End of Frame: This field indicates the end of the video data for the frame and the start of the next frame. It is a single byte that is set to the value 0x01.
val endOfFrame = 0x01
val FID = (frameId).toByte()

Add all of the above to the header

header.putInt(frameSize)
header.putShort(image.width.toShort())
header.putShort(image.height.toShort())
header.put(image.format.toByte())
header.put(((EOH shl 7) or (ERR shl 6) or (STI shl 5) or (REST shl 4) or SRC).toByte())
header.putLong(PTS)
header.put(endOfFrame.toByte())
header.put(FID)

Open the FileOutputStream and try to write the header and the image:

val uvcFileOutputStream = FileOutputStream("/dev/video3", true)
uvcFileOutputStream.write(header.toByteArray() + data)
uvcFileOutputStream.close()

Getting java.io.IOException: write failed: EINVAL (Invalid argument)

I tried to break down the imageByteArray into smaller chunks and write to the FileOutputStream by leveraging the endOfFrame or sameFrame bit in the header. Tried 512bytes, 1024bytes, 256kb, 512kb (image size output is smaller than 1024kb)

To test the UVC stream file and ensure it is writeable, I used the FFmpeg Android Library
Based on their docs, the ffmpeg -f video4linux2 -list_formats all -i /dev/video3 command should list the possible formats I can write to the stream.

Running their sample app with the following command:

String[] command = {"-f", "video4linux2", "-list_formats", "all", "-i", "/dev/video3"};
FFmpeg.getInstance(this).execute(command)

FFmpeg suggests /dev/video3 is not a video capture device:

Compared to /dev/video2 which it recognizes as a video device but fails to open it, probably due to it being used by the system.

Next steps/ideas:

1. make sure the /dev/video3 is writeable with the expected video stream format. Maybe we need to configure the file
2. The UVC Video Payload Header payload size, which is the size of the metadata information carried by the header, is 26 bytes according to the UVC specification. This includes the fields listed above, such as EOH, PTS, FID, which provide information about the video frame. The UVC metadata packet also includes a 16-byte header that specifies the packet type, size, and number. The total size of the UVC metadata packet, including both the header and the payload, is therefore 42 bytes. So I might be missing the metadata packet header.