fpgadeveloper / rpi-camera-fmc

Example designs and documentation for the RPi Camera FMC

Reference design for RPi Camera FMC

Description

This project demonstrates the Opsero RPi Camera FMC used to connect 4x Raspberry Pi cameras (or compatible cameras) to one of the target FPGA/MPSoC boards listed below. The designs contain:

• 4x MIPI CSI capture pipelines
• VVAS MultiScaler kernel for hardware accelerated image processing
• Video Mixer IP based display pipeline to DisplayPort output
• Video Codec Unit (VCU)

The high level block diagram below illustrates the design:

The video pipe sub-blocks are illustrated below:

Important links:

• The RPi Camera FMC datasheet
• The user guide for these reference designs
• To report an issue
• For technical support: Contact Opsero

Requirements

This project is designed for version 2022.1 of the Xilinx tools (Vivado/Vitis/PetaLinux). If you are using an older version of the Xilinx tools, then refer to the release tags to find the version of this repository that matches your version of the tools.

In order to test this design on hardware, you will need the following:

• Vivado 2022.1
• Vitis 2022.1
• PetaLinux Tools 2022.1
• Linux PC for build
• One or more Raspberry Pi Camera Module 2 and/or Digilent Pcam 5C cameras
• 1x RPi Camera FMC
• 1x DisplayPort monitor that supports 1080p video
• One of the supported target boards listed below

Target designs

Target board	Target design	FMC slot used	Cameras	VCU
ZCU104	zcu104	LPC	4	YES
ZCU102	zcu102_hpc0	HPC0	4	NO
ZCU102	zcu102_hpc1	HPC1	2 (note 1)	NO
ZCU106	zcu106_hpc0	HPC0	4	YES
PYNQ-ZU	pynqzu	LPC	2 (note 2)	NO
Genesys-ZU	genesyszu	LPC	2 (note 2)	YES
UltraZed EV carrier	uzev	HPC	4	YES

Notes:

1. The HPC1 connector of the ZCU102 board can only support 2 cameras due to it's pin assignment. This design supports the CAM0 and CAM1 slots as labelled on the RPi Camera FMC.
2. The pynqzu and genesyszu target designs have video pipelines for only 2 cameras: CAM1 and CAM2 as labelled on the RPi Camera FMC. This is due to the resource limitations of the devices on these boards.

Build instructions

This repo contains submodules. To clone this repo, run:

git clone --recursive https://github.com/fpgadeveloper/rpi-camera-fmc.git

Source Vivado and PetaLinux tools:

source <path-to-petalinux>/2022.1/settings.sh
source <path-to-vivado>/2022.1/settings64.sh

Build all (Vivado project, accelerator kernel and PetaLinux):

cd rpi-camera-fmc/PetaLinux
make petalinux TARGET=uzev

Contribute

We strongly encourage community contribution to these projects. Please make a pull request if you would like to share your work:

• if you've spotted and fixed any issues
• if you've added designs for other target platforms
• if you've added software support for other cameras

Thank you to everyone who supports us!

The TODO list

• Develop PYNQ support
• Get a Genesys-ZU board and test the design on hardware.
• Software support for more cameras (this will be an ongoing task due to the number of cameras available).

About us

Opsero Inc. is a team of FPGA developers delivering FPGA products and design services to start-ups and tech companies. Follow our blog, FPGA Developer, for news, tutorials and updates on the awesome projects we work on.

rpi-camera-fmc's Issues

Video-Mixer Modetest & Kernel Call trace Issue

Hi, I created and tested the design by following your doc for the ZCU102 board using Vivado/Petalinux 2022.1. There was no problem testing bare-metal design.
In the petalinux side, there was no problem working with MIPI capture pipelines. However, there was a problem working with the Video Mixer Pipeline. While doing the modetest, I did not found any planes information. I have attached some part of it below.

Planes:
id	crtc	fb	CRTC x,y	x,y	gamma size	possible crtcs
39	0	0	0,0		0,0	0       	0x00000001
  formats: VYUY UYVY YUYV YVYU YU16 YV16 YU24 YV24 NV16 NV61 GREY Y10  BG24 RG24 XB24 XR24 XB30 XR30 YU12 YV12 NV12 NV21 XV15 XV20 X403
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
	38 tpg:
		flags: range
		values: 0 1
		value: 0
40	41	46	0,0		0,0	0       	0x00000001
  formats: AB24 AR24 RA24 BA24 BG24 RG24 RA15 BA15 RA12 BA12 RG16 BG16
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 1
	32 alpha:
		flags: range
		values: 0 255
		value: 255
	33 g_alpha_en:
		flags: range
		values: 0 1
		value: 1

I did not find any information of mixer crtc connection with DP Encoder in your system-user.dtsi. My pl.dtsi did not have that information either. So, I updated it from system-user.dtsi file

 &v_mix{

	crtc_mixer_portv_mix: port@0 {
		reg = <0>;
		mixer_crtc:endpoint{
			reg=<0>;
			remote-endpoint=<&dp_encoder>;
		};
	};
};

&zynqmp_dpsub{

	dp_port: port@0 {
		reg = <0>;
		dp_encoder: endpoint {
			remote-endpoint = <&mixer_crtc>;
		};
	};
	
};

While doing the modetest, I got some more information about the vmixer planes.

Planes:
id	crtc	fb	CRTC x,y	x,y	gamma size	possible crtcs
34	0	0	0,0		0,0	0       	0x00000001
  formats: BG24
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
35	0	0	0,0		0,0	0       	0x00000001
  formats: BG24
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
36	0	0	0,0		0,0	0       	0x00000001
  formats: BG24
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
37	0	0	0,0		0,0	0       	0x00000001
  formats: BG24
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
38	0	0	0,0		0,0	0       	0x00000001
  formats: BG24
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 1
47	0	0	0,0		0,0	0       	0x00000002
  formats: VYUY UYVY YUYV YVYU YU16 YV16 YU24 YV24 NV16 NV61 GREY Y10  BG24 RG24 XB24 XR24 XB30 XR30 YU12 YV12 NV12 NV21 XV15 XV20 X403
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 0
	46 tpg:
		flags: range
		values: 0 1
		value: 0
48	0	0	0,0		0,0	0       	0x00000002
  formats: AB24 AR24 RA24 BA24 BG24 RG24 RA15 BA15 RA12 BA12 RG16 BG16
  props:
	9 type:
		flags: immutable enum
		enums: Overlay=0 Primary=1 Cursor=2
		value: 1
	40 alpha:
		flags: range
		values: 0 255
		value: 255
	41 g_alpha_en:
		flags: range
		values: 0 1
		value: 1

Then, I entered the following command
modetest -D a02b0000.v_mix -s 51:1920x1080@BG24

I got following issues.
`zcu102-rpi-fmc-2022-1:~$ modetest -D a02b0000.v_mix -s 51:1920x1080@BG24
trying to open device 'i915'...done
setting mode 1920x1080-60.00Hz on connectors 51, crtc 39

[ 1728.636240] [drm:drm_crtc_commit_wait] ERROR flip_done timed out
[ 1728.642437] [drm:drm_atomic_helper_wait_for_dependencies] ERROR [CRTC:39:crtc-0] commit wait timed out
[ 1738.876267] [drm:drm_crtc_commit_wait] ERROR flip_done timed out
[ 1738.882458] [drm:drm_atomic_helper_wait_for_dependencies] ERROR [CONNECTOR:51:DP-1] commit wait timed out
[ 1749.116294] [drm:drm_crtc_commit_wait] ERROR flip_done timed out
[ 1749.122494] [drm:drm_atomic_helper_wait_for_dependencies] ERROR [PLANE:38:plane-4] commit wait timed out
[ 1749.232295] ------------[ cut here ]------------
[ 1749.236907] [CRTC:39:crtc-0] vblank wait timed out
[ 1749.241728] WARNING: CPU: 1 PID: 681 at drivers/gpu/drm/drm_atomic_helper.c:1514 drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1749.253810] Modules linked in: mali(O) xilinx_gamma xilinx_csi2rxss xilinx_vpss_scaler xilinx_demosaic xlnx_mixer uio_pdrv_genirq dmaproxy(O)
[ 1749.266512] CPU: 1 PID: 681 Comm: modetest Tainted: G W O 5.15.19-xilinx-v2022.1 #1
[ 1749.275286] Hardware name: ZynqMP ZCU102 Rev1.0 (DT)
[ 1749.280235] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1749.287187] pc : drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1749.293707] lr : drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1749.300226] sp : ffff80000edfb9f0
[ 1749.303525] x29: ffff80000edfb9f0 x28: 0000000000000005 x27: 0000000000000000
[ 1749.310652] x26: 0000000000000001 x25: 0000000000000038 x24: ffff000805055800
[ 1749.317779] x23: 0000000000000001 x22: 0000000000000000 x21: ffff000802227f00
[ 1749.324906] x20: ffff0008052e7100 x19: 0000000000000000 x18: ffffffffffffffff
[ 1749.332033] x17: ffff800009e44ab8 x16: 0000000000000001 x15: ffff80008edfb717
[ 1749.339160] x14: 0000000000000000 x13: ffff8000095860a8 x12: 0000000000000933
[ 1749.346287] x11: 0000000000000311 x10: ffff8000095860a8 x9 : ffff8000095860a8
[ 1749.353414] x8 : 00000000fffff7ff x7 : ffff8000095b20a8 x6 : 0000000000000001
[ 1749.360541] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 1749.367668] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000800106300
[ 1749.374796] Call trace:
[ 1749.377226] drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1749.383398] drm_atomic_helper_commit_tail+0x80/0xa0
[ 1749.388354] commit_tail+0x128/0x17c
[ 1749.391922] drm_atomic_helper_commit+0x148/0x174
[ 1749.396618] drm_atomic_commit+0x4c/0x60
[ 1749.400534] drm_atomic_helper_set_config+0xa4/0x100
[ 1749.405490] drm_mode_setcrtc+0x19c/0x670
[ 1749.409492] drm_ioctl_kernel+0xc4/0x11c
[ 1749.413407] drm_ioctl+0x214/0x44c
[ 1749.416802] __arm64_sys_ioctl+0xb8/0xe0
[ 1749.420717] invoke_syscall+0x54/0x124
[ 1749.424458] el0_svc_common.constprop.0+0xd4/0xfc
[ 1749.429154] do_el0_svc+0x48/0xb0
[ 1749.432462] el0_svc+0x28/0x80
[ 1749.435509] el0t_64_sync_handler+0xa4/0x130
[ 1749.439771] el0t_64_sync+0x1a0/0x1a4
[ 1749.443426] ---[ end trace 42f231a4eafbf293 ]---
[ 1759.612326] [drm:drm_crtc_commit_wait] ERROR flip_done timed out
[ 1759.618523] [drm:drm_atomic_helper_wait_for_dependencies] ERROR [CRTC:39:crtc-0] commit wait timed out
[ 1769.852354] [drm:drm_crtc_commit_wait] ERROR flip_done timed out
[ 1769.858543] [drm:drm_atomic_helper_wait_for_dependencies] ERROR [PLANE:38:plane-4] commit wait timed out
[ 1769.968353] ------------[ cut here ]------------
[ 1769.972964] [CRTC:39:crtc-0] vblank wait timed out
[ 1769.977787] WARNING: CPU: 1 PID: 634 at drivers/gpu/drm/drm_atomic_helper.c:1514 drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1769.989868] Modules linked in: mali(O) xilinx_gamma xilinx_csi2rxss xilinx_vpss_scaler xilinx_demosaic xlnx_mixer uio_pdrv_genirq dmaproxy(O)
[ 1770.002570] CPU: 1 PID: 634 Comm: kworker/1:0 Tainted: G W O 5.15.19-xilinx-v2022.1 #1
[ 1770.011604] Hardware name: ZynqMP ZCU102 Rev1.0 (DT)
[ 1770.016554] Workqueue: events drm_mode_rmfb_work_fn
[ 1770.021423] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1770.028376] pc : drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1770.034895] lr : drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1770.041414] sp : ffff80000ad53b60
[ 1770.044713] x29: ffff80000ad53b60 x28: 0000000000000006 x27: 0000000000000000
[ 1770.051840] x26: 0000000000000001 x25: 0000000000000038 x24: ffff000805055800
[ 1770.058967] x23: 0000000000000001 x22: 0000000000000000 x21: ffff000802227180
[ 1770.066094] x20: ffff0008052e7100 x19: 0000000000000000 x18: ffffffffffffffff
[ 1770.073221] x17: 3a38333a454e414c x16: 505b202a524f5252 x15: ffff80008ad53887
[ 1770.080348] x14: 0000000000000000 x13: ffff8000095860a8 x12: 00000000000009b1
[ 1770.087475] x11: 000000000000033b x10: ffff8000095860a8 x9 : ffff8000095860a8
[ 1770.094602] x8 : 00000000fffff7ff x7 : ffff8000095b20a8 x6 : 0000000000000001
[ 1770.101729] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 1770.108856] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000806638280
[ 1770.115984] Call trace:
[ 1770.118415] drm_atomic_helper_wait_for_vblanks.part.0+0x278/0x2a0
[ 1770.124586] drm_atomic_helper_commit_tail+0x80/0xa0
[ 1770.129542] commit_tail+0x128/0x17c
[ 1770.133110] drm_atomic_helper_commit+0x148/0x174
[ 1770.137807] drm_atomic_commit+0x4c/0x60
[ 1770.141722] drm_framebuffer_remove+0x444/0x4d4
[ 1770.146244] drm_mode_rmfb_work_fn+0x74/0x9c
[ 1770.150507] process_one_work+0x1d8/0x390
[ 1770.154508] worker_thread+0x298/0x4e0
[ 1770.158250] kthread+0x120/0x130
[ 1770.161471] ret_from_fork+0x10/0x20
[ 1770.165039] ---[ end trace 42f231a4eafbf294 ]---`

I am just stuck here.
I also followed this link
https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841850/Video+Mixer