[Project Page] [Paper] [Video] [Slides] [Poster]
Created by Itai Lang1,2, Dror Aiger2, Forrester Cole2, Shai Avidan1, and Michael Rubinstein2.
1Tel Aviv University 2Google Research
Scene flow estimation is a long-standing problem in computer vision, where the goal is to find the 3D motion of a scene from its consecutive observations. Recently, there have been efforts to compute the scene flow from 3D point clouds. A common approach is to train a regression model that consumes source and target point clouds and outputs the per-point translation vector. An alternative is to learn point matches between the point clouds concurrently with regressing a refinement of the initial correspondence flow. In both cases, the learning task is very challenging since the flow regression is done in the free 3D space, and a typical solution is to resort to a large annotated synthetic dataset.
We introduce SCOOP, a new method for scene flow estimation that can be learned on a small amount of data without employing ground-truth flow supervision. In contrast to previous work, we train a pure correspondence model focused on learning point feature representation and initialize the flow as the difference between a source point and its softly corresponding target point. Then, in the run-time phase, we directly optimize a flow refinement component with a self-supervised objective, which leads to a coherent and accurate flow field between the point clouds. Experiments on widespread datasets demonstrate the performance gains achieved by our method compared to existing leading techniques while using a fraction of the training data.
If you find our work useful in your research, please consider citing:
@InProceedings{lang2023scoop,
author = {Lang, Itai and Aiger, Dror and Cole, Forrester and Avidan, Shai and Rubinstein, Michael},
title = {{SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow}},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {5281--5290},
year = {2023}
}
The code has been tested with Python 3.6.13, PyTorch 1.6.0, CUDA 10.1, and cuDNN 7.6.5 on Ubuntu 16.04.
Clone this repository:
git clone https://github.com/itailang/SCOOP.git
cd SCOOP/Create a conda environment:
# create and activate a conda environment
conda create -n scoop python=3.6.13 --yes
conda activate scoopInstall required packages:
sh install_environment.shCompile the Chamfer Distance op, implemented by Groueix et al. The op is located under auxiliary/ChamferDistancePytorch/chamfer3D folder. The following compilation script uses a CUDA 10.1 path. If needed, modify script to point to your CUDA path. Then, use:
sh compile_chamfer_distance_op.shThe compilation results should be created under auxiliary/ChamferDistancePytorch/chamfer3D/build folder.
Create folders for the data:
mkdir ./data/
mkdir ./data/FlowNet3D/
mkdir ./data/HPLFlowNet/We use the point cloud data version prepared Liu et al. from their work FlowNet3D. Please follow their code to acquire the data.
- Put the preprocessed FlyingThings3D dataset at
./data/FlowNet3D/data_processed_maxcut_35_20k_2k_8192/. This data is denoted as FT3Do. - Put the preprocessed KITTI dataset at
./data/flownet3d/kitti_rm_ground/. This dataset is denoted as KITTIo and its subsets are denoted as KITTIv and KITTIt.
We also use the point cloud data version prepared Gu et al. from their work HPLFlowNet. Please follow their code to acquire the data.
- Put the preprocessed FlyingThings3D dataset at
./data/HPLFlowNet/FlyingThings3D_subset_processed_35m/. This dataset is denoted as FT3Ds. - Put the preprocessed KITTI dataset at
./data/flownet3d/KITTI_processed_occ_final/. This dataset is denoted as KITTIs.
Note that you may put the data elsewhere and create a symbolic link to the actual location. For example:
ln -s /path/to/the/actual/dataset/location ./data/FlowNet3D/data_processed_maxcut_35_20k_2k_8192 Switch to the scripts folder:
cd ./scriptsTo train a model on 1,800 examples from the train set of FT3Do, run the following command:
sh train_on_ft3d_o.shEvaluate this model on KITTIo with 2,048 point per point cloud using the following command:
sh evaluate_on_kitti_o.shThe results will be saved to the file ./experiments/ft3d_o_1800_examples/log_evaluation_kitti_o.txt.
Evaluate this model on KITTIo with all the points in the point clouds using the following command:
sh evaluate_on_kitti_o_all_point.shThe results will be saved to the file ./experiments/ft3d_o_1800_examples/log_evaluation_kitti_o_all_points.txt.
To train a model on the 100 examples of KITTIv, run the following command:
sh train_on_kitti_v.shEvaluate this model on KITTIt with 2,048 point per point cloud using the following command:
sh evaluate_on_kitti_t.shThe results will be saved to the file ./experiments/kitti_v_100_examples/log_evaluation_kitti_t.txt.
Evaluate this model on KITTIt with all the points in the point clouds using the following command:
sh evaluate_on_kitti_t_all_points.shThe results will be saved to the file ./experiments/kitti_v_100_examples/log_evaluation_kitti_t_all_points.txt.
To train a model on 1,800 examples from the train set FT3Ds, run the following command:
sh train_on_ft3d_s.shEvaluate this model on KITTIs with 8,192 point per point cloud using the following command:
sh evaluate_on_kitti_s.shThe results will be saved to the file ./experiments/ft3d_s_1800_examples/log_evaluation_kitti_s.txt.
Evaluate this model on the test set of FT3Ds with 8,192 point per point cloud using the following command:
sh evaluate_on_ft3d_s.shThe results will be saved to the file ./experiments/ft3d_s_1800_examples/log_evaluation_ft3d_s.txt.
First, save results for visualization by adding the flag --save_pc_res 1 when running the evaluation script. For example, the script for evaluating on KITTIt.
The results will be saved to the folder ./experiments/kitti_v_100_examples/pc_res/.
Then, select the scene index that you would like to visualize and run the visualization script. For example, visualizing scene index #1 from KITTIt:
python visualize_scoop.py --res_dir ./../experiments/kitti_v_100_examples/pc_res --res_idx 1 The visualizations will be saved to the folder ./experiments/kitti_v_100_examples/pc_res/vis/.
First, download our pretrained models with the following command:
bash download_pretrained_models.shThe models (about 2MB) will be saved under pretrained_models folder in the following structure:
pretrained_models/
├── ft3d_o_1800_examples/model_e100.tar
├── ft3d_s_1800_examples/model_e060.tar
├── kitti_v_100_examples/model_e400.tar
Then, use the evaluation commands mentioned in section Training and Evaluation,
after changing the experiments folder in the evaluation scripts to the pretrained_models folder.
This project is licensed under the terms of the MIT license (see the LICENSE file for more details).
Our code builds upon the code provided by Puy et al., Groueix et al., Liu et al., and Gu et al. We thank the authors for sharing their code.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
![[Poster]](./doc/poster.png){kind=link}