mcg-nju / link

[CVPR 2023] LinK: Linear Kernel for LiDAR-based 3D Perception

LinK: Linear Kernel for LiDAR-based 3D Perception

Official PyTorch implementation of LinK, from the following paper:

LinK: Linear Kernel for LiDAR-based 3D Perception. CVPR 2023.
Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, Limin Wang
Multimedia Computing Group, Nanjing University
[arxiv][Conference version]

---

LinK is a large kernel backbone for 3D perception tasks, consisting of a linear kernel generator and a pre-aggregation strategy. The two designs scale up the perception range into 21x21x21 with linear complexity.

---

Model Zoo

Segmentation on SemanticKITTI(val)

name	kernel config	mIoU	model
LinK	cos_x:(2x3)^3	67.72	model
LinK	cos:(3x7)^3	67.50	model
LinK(encoder-only)	cos_x:(2x3)^3	67.33	model
LinK(encoder-only)	cos:(3x5)^3	67.07	model

Detection on nuScenes

• Validation

name	kernel config	NDS	mAP	model
LinK	cos:(3x7)^3	69.5	63.6	model

• Test

name	kernel config	NDS	mAP	model
LinK	cos:(3x7)^3	71.0	66.3	model
LinK(TTA)	cos:(3x7)^3	73.4	69.8	model

Installation

Clone this repo to your workspace.

git clone https://github.com/MCG-NJU/LinK.git
cd LinK

Semantic Segmentation

please check segmentation/INSTALL.md and segmentation/GET_STARTED.md.

Detection

see detection/INSTALL.md and detection/GET_STARTED.md.

Citation

If you find our work helpful, please consider citing:

@InProceedings{lu2023link,
    author    = {Lu, Tao and Ding, Xiang and Liu, Haisong and Wu, Gangshan and Wang, Limin},
    title     = {LinK: Linear Kernel for LiDAR-Based 3D Perception},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {1105-1115}
}

@article{lu2022app,
  title={APP-Net: Auxiliary-point-based Push and Pull Operations for Efficient Point Cloud Classification},
  author={Lu, Tao and Liu, Chunxu and Chen, Youxin and Wu, Gangshan and Wang, Limin},
  journal={arXiv preprint arXiv:2205.00847},
  year={2022}
}

Contact

• Tao Lu: [email protected]
• Xiang Ding: [email protected]
• Haisong Liu: [email protected]

Acknowledgement

Our code is based on CenterPoint, SPVNAS, spconv, torchsparse. And we thank a lot for the kind help from Ruixiang Zhang, Xu Yan and Yukang Chen.

link's Issues

Problems in the Evaluation and Submission of segmentation

Thanks for this excellent piece of work! I encountered "Warning: could not find environment variable "-x", mpirun was unable to find the specified executable file, and therefore did not launch the job. Warning: could not find environment variable "-x", mpirun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 0; it may have occurred for other processes as well", but I have already run the chmod +x evaluate.sh command, and I would like to know how to create a hyperlink to the semantickitti dataset if it is placed in a different location, and is hyperlink necessary?

Problems encountered when installing torchsparse

As mentioned in the paper, training with 3090 is feasible. Installing CUDA10.2 in 3090 is difficult. So I'm using CUDA11.3, Python3.7.10, pytorch1.10.1. The previous steps performed according to installation, and detectioin was added to PYTHONPATH, but when executing the last step, "pip install./torchsparse-u/", the error is as follows

Processing ./torchsparse-u
Preparing metadata (setup.py) ... done
Building wheels for collected packages: torchsparse
Building wheel for torchsparse (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [112 lines of output]
...
subprocess.CalledProcessError: Command '['which', 'c++']' returned non-zero exit status 1.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torchsparse
Running setup.py clean for torchsparse
Failed to build torchsparse
ERROR: Could not build wheels for torchsparse, which is required to install pyproject.toml-based projects

What might be the cause of this problem? Could you please give some reference configuration advice on using CUDA11.3 installation project environment on 3090?Besides,is the requirement.txt exactly the same as the requirement.txt in CenterPoint?

how to visualize eval result

when I run the follow command:

python -m torch.distributed.launch --nproc_per_node=4 --master_port 23456 ./tools/train.py \
  configs/nusc/voxelnet/nusc_centerpoint_voxelnet_0075voxel_fix_bn_z_elkv3.py \
  --work_dir work_dirs/0075cbgs_cos3x7_group2_15+5 \
  --resume_from work_dirs/0075cbgs_cos3x7_group2/epoch_15.pth

I get a prediction.pkl in work_dir work_dirs/0075cbgs_cos3x7_group2_15+5 directory, I want visualize the result, but I find the ./detection/tools/visual.py can not work well, so where can I find the scrip to visualize predict result.

some questions about training

Hi,

Thank you for the great work!

I tried to use pcdet to run Link, I modified SpMiddleResNetFHDELKv3 model. It works, but I had some questions during the debug process.

1. Funtion small_to_large_v2: (line 84 in ts_elk.py). F.spdevoxelize requires only 3 input elements. So can I change the new_feat = F.spdevoxelize(f, idx_query, weights, kernel_size) to new_feat = F.spdevoxelize(f, idx_query, weights)?
2. I had nan in training process. When I debug it, I found that here divides by 0 （Funtion small_to_large_v2: new_feat = new_feat[:,:-1] / (new_feat[:,-1:]) ). Can I add a small value here (e.g. 1e-8) ?

When I finished training, I found that Link was a little better (1 mAP) than CenterPoint(3d spconv). Is this normal? Is it possible that the changes I made above are causing poor performance?

问一个问题

只刷点云，刷了nuscenes数据集，效果还可以，这样投稿能中吗？我看你还专门刷了kitti，有谁告诉你的，请指点一下，我论文刚写完，准备投稿

about modified Torchsparse

Hi,could you please tell the differences between torchsparse1.14.0 and the modified Torchsparse ? cause there is a new version Torchsparse that is easier to install , when i want to use your torchsparse I need to change my gcc version to lower version.

File "/datac/lcl/PCRLv2-finetune/brats19/brats19/models/torchsparse/nn/modules/activation.py", line 3, in <module> from torchsparse import SparseTensor ImportError: cannot import name 'SparseTensor' from 'torchsparse' (/datac/lcl/PCRLv2-finetune/brats19/brats19/models/torchsparse/__init__.py)