shathe / 3d-mininet

Official Implementation in Pytorch and Tensorflow of 3D-MiniNet: Learning a 2D Representation from Point Clouds for Fast and Efficient 3D LIDAR Semantic Segmentation

3D-MiniNet: Learning a 2D Representation from Point Clouds for Fast and Efficient 3D LIDAR Semantic Segmentation

Introduction

This repository contains the implementation of 3D-MiniNet, a fast and efficient method for semantic segmentation of LIDAR point clouds.

The following figure shows the basic building block of our 3D-MiniNet:

3D-MiniNet overview. It takes P groups of N points each and computes semantic segmentation of the M points of the point cloud where PxN=M.

It consists of two main modules: our proposed learning module (on the left) which learns a 2D tensor which is fed to the second module, an efficient FCNN backbone (on the right) which computes the 2D semantic segmentation. Each 3D point of the point cloud is given a semantic label based on the 2D segmentation.

Code (Pytorch and Tensorflow implementation)

Our PyTorch code is based on Milioto et al. code and the Tensorflow code is based on Biasutti et al. code. For copyright license, please check both code base licenses.

We took their code base and integrate our approach. Therefore, please, consider also citing or checking their work.

Citation

If you find 3D-MiniNet useful, please consider citing:

@article{alonso2020MiniNet3D,
  title={3D-MiniNet: Learning a 2D Representation from Point Clouds for Fast and Efficient 3D LIDAR Semantic Segmentation},
  author={Alonso, I{\~n}igo and Riazuelo, Luis and Montesano, Luis and Murillo, Ana C},
booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2020},
  organization={IEEE}
}

3d-mininet's Issues

Code Release

@Shathe thanks for informing about the wonderful work , can you let us know when is the source code for training , testing /inference avaialble ??

Convert pre-trained model to .onnx format

I tried to convert the pre-trained 3D-MiniNet to .onnx format by using the script provided in a similar issue of lidar-bonnetal here.

Specifically, I changed the part that is related to the dummy input of the model in the following way:
dinput = torch.randn(1, 11, 64, 2048, device='cuda', requires_grad=True)
input_points = torch.randn(1, 11, 16, 8192, device='cuda', requires_grad=True)
dummy_input = [dinput, input_points]
and the rest of the script remains the same.

However, I get the following error related to ONNX

Failed to export an ONNX attribute, since it's not constant, please try to make things (e.g., kernel size) static if possible

I tried to apply the fix that was suggested here, but it didn't work.

Is there any other way of converting it to ONNX format?
I am using exactly the same versions of packages as in the requirements.txt file

val iou

hello ，i infer the val dataset using Pretrained Model，and the average iou of 3D-MiniNet-tiny-KNN is 63.4 ,however test dataset only 46.9 in paper . it is same with you ?

Single point cloud Inference without dataloader

Nice Work! Can you provide a simple inference example where we can provide the input (point cloud) to the model without using dataloader?
As I see here proj_output = self.model([proj_in, proj_chan_group_points], proj_mask) we need three args for inferenc. it would be better to get them without dataloader (for example, processing live data from lidar sensor or from a bag)

Thanks

Pretrained Model on Train Set Only

Hi @Shathe, thanks for open-sourcing your work!

Regarding the public checkpoints of 3D-MiniNet, are there any that are trained only on the train set, rather than train+val? Thanks a lot for your help!

Weights for head module are not the correct dimensions

Hello! I'm working on a school project that requires realtime 3D semantic segmentation and this code has helped immensly!

When trying to load a Segmentator module from the pytorch-code directory, I come across the following error message:

Successfully loaded model backbone weights
Successfully loaded model decoder weights
Couldn't load head, using random weights. Error:  Error(s) in loading state_dict for Sequential:
        size mismatch for 0.weight: copying a param with shape torch.Size([20, 48, 3, 3]) from checkpoint, the shape in current model is torch.Size([34, 48, 3, 3]).
        size mismatch for 0.bias: copying a param with shape torch.Size([20]) from checkpoint, the shape in current model is torch.Size([34]).

Which given that I want to perform segmentation, is not ideal because this means it does not work.
I'm sure I'm using the correct model names and everything, and what's even weirder is that the decoder weights finish with a 34, not a 20, so the actual pytorch code seems to be fine. What isn't correct are the weights for the head that are stored in the 3D-MiniNet-x directories. I tried all three model sizes and they all have this same issue.

system dependencies

Hello.

When I ran command lines for system dependencies, I got an error like

sudo apt-get install -yqq build-essential ninja-build \

python3-dev python3-pip apt-utils curl git cmake unzip autoconf autogen
libtool mlocate zlib1g-dev python3-numpy python3-wheel wget
software-properties-common openjdk-8-jdk libpng-dev
libxft-dev ffmpeg python3-pyqt5.qtopengl

E: Unable to locate package autogen
E: Unable to locate package openjdk-8-jdk
E: Package 'ffmpeg' has no installation candidate
E: Unable to locate package python3-pyqt5.qtopengl
E: Couldn't find any package by glob 'python3-pyqt5.qtopengl'
E: Couldn't find any package by regex 'python3-pyqt5.qtopengl'

Do you know how to solve it?

Thanks in advance.

Any updates on code release?

Hello, I'm looking forward to trying your excellent method.
Is there any updates on code release?

some of the prediction label's total num doesn't match points num

when i use visualize to watch the infer result it simply raise valueError and tells me that the num of the two doesn't match,some are equal but many are still not equal.how to solve this.

About the smooth the resulting class weights

Hi, dear sir. Look like the weight of each class of your code is not same to rangenet++.

3D-MiniNet/pytorch_code/lidar-bonnetal/train/tasks/semantic/modules/trainer.py

Lines 78 to 81 in 21a2783

	self.loss_w = 1 / (content + epsilon_w) # get weights
	power_value = 0.25
	self.loss_w = np.power(self.loss_w, power_value) * np.power(10, 1 - power_value)
	for x_cl, w in enumerate(self.loss_w): # ignore the ones necessary to ignore

Have your results show that the smooth weight is much better? Since there is not have a ablation study.
Looking forward to your reply