mjoshi07 / multi-sensor-fusion-scene-segmentation

Final Destination

Multi-Sensor-Fusion-Scene-Segmentation

ENPM673 Spring 2022: Perception for Autonomous Robots

Project 4

RGB segmentation and lidar fusion

Optical flow segmentation and lidar fusion

Instructions -

FusionNet

Semantic segmentation using RGB, LiDAR and Optical flow sensor data Trained on vkitti dataset.

Data Preparation

Download vkitti_1.3.1_depthgt.tar, vkitti_1.3.1_flowgt.tar, vkitti_1.3.1_rgb.tar, vkitti_1.3.1_scenegt.tar from this link. Extract the tarballs into one folder. Use the path of this folder for <path_to_dataset> argument throughout the following instruction.

Training

To train, run the following commands:

• python main.py --dataset_path <path_to_dataset> [--lidar] [--optical_flow]

The --lidar and --optical_flow flags are optional. Specifying the flags will train the network with corresponding LiDAR and oflow data. Without the flags, the network will be trained only on RGB data.

Testing

To test, run the following commands:

• python recons.py --dataset_path <path_to_dataset> --checkpoint_path <path_checkpoint_file> [--lidar] [--optical_flow]

Specify one or both of --lidar and --optical_flow flags if you had specified it during training. A pretrained checkpoint file can be downloaded from here. Note that if you're using our checkpoint file, do not specify the --lidar and --optical_flow flags.

Results

The below image shows the results of FusionNet. The pictures on the left column are the actual segmentation ground truth and the pictures on the right column are the segmentation masks obtained from our net.

BiSeNetv2:

For training and testing RGB and Optical flow data follow the instrcutions on the ReadMe of BiSeNet Download initial training weights from *here

Lidar Fusion

To fuse segmentation results from optical flow or rgb with lidar points, modify the input data here

Multi-Sensor-Fusion-Scene-Segmentation/lidar_fusion/Code/lidar_fusion.py

Line 87 in 36d1966

data_path = "../Data/KITTI_data"

Convert lidar .bin files to .pcd using the function

Multi-Sensor-Fusion-Scene-Segmentation/lidar_fusion/Code/Utils.py

Line 11 in 36d1966

def convert_bin_to_pcd(binary_file, pcd_filepath):

Run the following command after setting all the data

python3 lidar_fusion.py

The output after execution will look something like this

Resources

• Raft: optical flow: learnopencv
• Raft: optical flow: paperswithcode
• Flownet 2.0: optical flow
• ppr: SegFlow: Segmentation + optical flow
• git: SegFlow: Segmentation + optical flow
• KITTI Scene Flow dataset

Authors -

Naitri Rajyaguru (117361919)

Mayank Joshi (117555264)

Nitesh Jha (117525366)

Aneesh Dandime (118418506)

Tanuj Thakkar (117817539)

Credits:

• BiSeNetv2