haohaonju / centerfusion

NN-based radar-camera post sensor fusion implemented by TensorRT

Camera & Radar feature-level sensor fusion for object detection

watchme.mp4

Installation

For regular python package installation, type in pip install -r requirements.txt

It is not possible for CenterFusion to run without the support of DCNv2 plugin(A Deformable convolutional network algorighm), you should install it with python and compile it as TensorRT plugin.

• For python package installation, see here

• For compiling the DCNv2-TensorRT plugin, see here

You should then be able to build our source code by

cd PATH/TO/THIS/PROJECT
mkdir build && cd build
cmake .. -DTRT_LIB_PATH=${TRT_LIB_PATH}
make 

where ${TRT_LIB_PATH} refers to the library path where you built your DCN-TRT plugin, only in this way can your TRT onnx parser recognize the dcn node in ONNX graph.

Generating samples

Please download and preprocess nuscenes dataset according to this, assuming your directory structure is like this :

${CF_ROOT}
`-- data
  `-- nuscenes
      |-- annotations_6sweeps
      |-- maps
      |-- samples
      |   |-- CAM_BACK
      |   |   | -- xxx.jpg
      |   |   ` -- ...
      |   |-- CAM_BACK_LEFT
      |   |-- CAM_BACK_RIGHT
      |   |-- CAM_FRONT
      |   |-- CAM_FRONT_LEFT
      |   |-- CAM_FRONT_RIGHT
      |   |-- RADAR_BACK_LEFT
      |   |   | -- xxx.pcd
      |   |   ` -- ...
      |   |-- RADAR_BACK_RIGHT
      |   |-- RADAR_FRON
      |   |-- RADAR_FRONT_LEFT
      |   `-- RADAR_FRONT_RIGHT
      |-- sweeps
      |-- v1.0-mini
      |-- v1.0-test
      `-- v1.0-trainval

move the annotation file data/val_top1000.json to data/nuscenes/annotations_6sweeps . run the following commands :

cd tools/CenterFusion
sh experiments/create_data.sh

Then you should have the generated datas in data/predata :

- images, contains 1000 frame input images for trt engines, each has its shape (3, 448, 800)
- calibs, contains 1000 frame camera intrinsics, each has its shape (3,4)
- pc_3ds, contains 1000 frame radar points, each has its shape  (5,1000), each row stands for [loc_x, loc_y, loc_z, velo_x, velo_y]
- data_num.bin, wich shape (1000,), records valid point nums for each radar frame 

These datas wille be used to feed the trt engines.

Run trt engine with samples

After the installation and input data generation, you can simply go to ${CF_ROOT}, type in sh run.sh to run the project. Note that for the first time you run the project, it will take some time to generate trt engine from onnx files.
Then you should have the generated results in directory ${CF_ROOT}/results

Visualization the results with ros

To show the results like the video do, you should previously install ros package according you ubuntu version. Then you should compile with you python package

sudo apt-get install python-catkin-tools python3-dev python3-catkin-pkg-modules python3-numpy python3-yaml ros-melodic-cv-bridge
# Create catkin workspace
cd ${CF_ROOT}/tools/visualization/catkin_workspace
catkin init
# Instruct catkin to set cmake variables, feel free to change path according to your python version
catkin config -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6m -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/ libpython3.6m.so
# Instruct catkin to install built packages into install place. It is $CATKIN_WORKSPACE/install folder
catkin config --install

# Find version of cv_bridge in your repository
apt-cache show ros-melodic-cv-bridge | grep Version
# Version: 1.13.0-0bionic.20220127.152918
# Checkout right version in git repo. In our case it is 1.13.0
cd src/vision_opencv/
git checkout 1.13.0
cd ../../
# Build
catkin build cv_bridge
# Extend environment with new package
source install/setup.bash --extend  #or source install/setup.zsh

Open another two terminals, one type in rescore, another type in rviz, in this terminal, type in python fusion_det_cpp.py You can then add topics nusc_image/Image, nusc_ego_car/Marker, nusc_3dbox/MarkerArray , then you'll be able to see the detection results.

Exporting to onnx

Our example use the pretrain model centerfusion_e60 (you can export you own model according to this process), you can turn to here to see the detailed metrics of this default model. To export the default model, download this model weight and put it to tools/CenterFusion/models .

Before exporting to onnx, you should previousely install the CenterFusion python dependencies

cd tools/CenterFusion
pip install -r requirements.txt

For DCNv2 python package installation, pls turn to here.

You can then exporting as onnx models by

cd tools/CenterFusion
sh experiments/export.sh

Then you'll see two onnx files in tools/CenterFusion/models.

Computation speed

The computation latency(by millisecond) is computed for each module, you can see the below table

	Preprocess	CameraInfer	FrustumAssoc	FeatMerge	FusInfer	PostProcess	Total
engine_fp16	0.09	7.44	7.64	0.05	1.00	0.62	16.84
engine_fp32	0.16	12.03	8.81	0.04	3.05	0.75	24.84

Computation Graph

The main modules of procession can be seen here

The most innovative parts are pc_dep generation and frustum association, we'll illustrate the main ideas.

PC_DEP Generation

This is a step where we encode the raw radar points as a structured pseudo-image data format .

Given each radar points, we firstly generate its coordinated pillar(with size 0.2, 0.2, 1.5 for length,width,height) in camera-viewed 3d space, we then project its 8 corners to image pixel coordinates, calculating its 2d top-left & bottom-right corners, which defines a 2d box, we insert (loc_z, velo_x,velo_z) of this radar point to each pixel inside this 2d box , when two boxs are intersected, we insert the nearer point feature to the intersected area. The 3d pillars and generated pc_dep can be seen here :

Frustum Association

An object detected using the image features (left), generating the ROI frustum based on object’s 3D bounding box (middle), and the BEV of the ROI frustum showing radar detections inside the frustum (right). δ is used to increase the frustum size in the testing phase. d is the ground truth depth in the training phase and the estimated object depth in the testing phase.

Acknowledgements

This project refers to some codes from CenterFusion but some codes have been slightly modified.

Contact

Haohao by [email protected]

centerfusion's Issues

您好，刚接触深度学习和多传感器融合方向，有些问题想向您请教

尽管你给出了比较详尽的注释，但是我实在是太菜了导致对您的代码文件不是很理解，如果可以的话可以加VX咨询一下您吗，我的VX是lengyue0115 万分感谢

why pip install -r requirements.txt, many dependency related with ros pip package could not installed

which type of GPU to get the Computation speed in the README?

复现示例正常，本地数据集可视化结果错误

博主您好，感谢您的分享，运行了您的示例代码，效果是正常的，但是当使用本地数据训练的pth->onnx，效果就很不正常，框非常大，debug了一下，发现第二阶段的fusion-onnx是没有问题的，问题有可能出现在图片预处理阶段或者是centernet-onnx文件，看了create_data.py中`def prefetch_test(opt):
"""
imgs = data['images'][1.0].numpy() : 1 x 2 x 3 x H_in x W_in, two input image, uint8 dtype
img = data['image'].numpy().squeeze() : torch.Size([1, 900, 1600, 3]), one input image , fp32 dtype
meta = data['meta'] :
{1.0: {'calib': tensor([[[1.2528e+03, 0.0000e+00, 8.2659e+02, 0.0000e+00],
[0.0000e+00, 1.2528e+03, 4.6998e+02, 0.0000e+00],
[0.0000e+00, 0.0000e+00, 1.0000e+00, 0.0000e+00]]]),
'c': tensor([[800., 450.]]),
's': tensor([1600.], dtype=torch.float64),
'height': tensor([900]),
'width': tensor([1600]),
'out_height': tensor([112]),
'out_width': tensor([200]),
'inp_height': tensor([448]),
'inp_width': tensor([800]),
'trans_input': tensor([[[ 0.5000, -0.0000, 0.0000], [ 0.0000, 0.5000, -1.0000]]], dtype=torch.float64),
'trans_output': tensor([[[ 0.1250, -0.0000, 0.0000], [ 0.0000, 0.1250, -0.2500]]], dtype=torch.float64)}}

            pc_2d = data['pc_2d']    : 1 x 3 x 1000, point clout dense tensor
            pc_N = data['pc_N']    : 53, valid point num
            pc_dep = data['pc_dep'] : torch.Size([1, 3, 112, 200]),    radar points    in    sparse image shape
            pc_3d = data['pc_3d'] : 1 x 18 x 1000, point cloud dense tensor , why 18 channels ? 
            """

`
我们的图片尺寸是[1280720],这块我们后续预处理为[1280768]，想请教下, 看看您这边有没有什么排查错误的思路。

Problem installing packages from requirements.txt

Hello @HaohaoNJU. Thank you for providing this repo. I wanted to ask you that while running requirements.txt file, there are many packages that are not getting installed as it says (This for an example)

ERROR: Could not find a version that satisfies the requirement image-geometry==1.16.0 (from -r requirements.txt (line 54)) (from versions: none)
ERROR: No matching distribution found for image-geometry==1.16.0 (from -r requirements.txt (line 54))

In fact there are almost every package that shows this kind of error. Can you guide me on how I can build this environment. My current virtual env is running on python 3.8.

why split model to img_model and fusion_model?

According to https://github.com/HaohaoNJU/CenterFusion/blob/main/tools/CenterFusion/src/convert_onnx.py#L141-L142

询问代码耗时过大原因

作者您好：
我在nvidia Xavier上跑，各模块耗时如下，
[06/12/2023-15:49:10] [I] Average PreProcess Time: 0.39536 ms
[06/12/2023-15:49:10] [I] Average DetectionInfer Time: 98.6102 ms
[06/12/2023-15:49:10] [I] Average FrustumAssoc Time: 41.1462 ms
[06/12/2023-15:49:10] [I] Average merge Time: 0.177472 ms
[06/12/2023-15:49:10] [I] Average FusInfer Time: 9.79856 ms
[06/12/2023-15:49:10] [I] Average PostProcess Time: 1.84218 ms
平均单帧用时大概在150-170ms，但是您的computation speed如下，
engine_fp16 | 0.09ms | 7.44ms | 7.64ms | 0.05ms | 1.00ms | 0.62ms | 16.84ms。
我想请问一下您是在什么设备上跑的，有什么好的思路或者方法来排查耗时吗？