peterzhousz / expansion

Upgrading Optical Flow to 3D Scene Flow through Optical Expansion, CVPR 2020 (Oral).

Optical Expansion

[project website]

If you find this work useful, please consider citing our paper:

@inproceedings{yang2020upgrading,
  title={Upgrading Optical Flow to 3D Scene Flow through Optical Expansion},
  author={Yang, Gengshan and Ramanan, Deva},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={1334--1343},
  year={2020}
}

Requirement

• python3
• opencv
• pytorch 1.2.0 (may also be compatible with other versions)
• tensorboard (only for training)

Useful tools (optional)

• gdown for downloading from google drive
• cvkit for visualizing .pfm files

Precomputed results

We provide precomputed optical flow, optical expansion, and monocular depth (from monodepth2) for kitti validation set. This should produce monocular scene flow results very close to Tab.1.

gdown https://drive.google.com/uc?id=1gThKsHOfsoGiLvgydVrDlqsvku0X_wjN -O ./precomputed.zip
unzip precomputed.zip

See demo.ipynb for more details.

Inference

Pretrained models

Download pre-trained models to ./weights (assuming gdown is installed, ~50MB each),

mkdir weights
mkdir weights/exp-driving
mkdir weights/exp-kitti-train
mkdir weights/exp-kitti-trainval
gdown https://drive.google.com/uc?id=1KMEqXlisLgK4n9alWRbgIWch7TTye56u -O ./weights/exp-driving/exp-driving.pth
gdown https://drive.google.com/uc?id=1ZjPc7P743R3b_5MbBbU_VpUMULYo-SWk -O ./weights/exp-kitti-train/exp-kitti-train.pth
gdown https://drive.google.com/uc?id=11Cf3NxbzGq2rdwdI2_HuQDlwIWNWMu7u -O ./weights/exp-kitti-trainval/exp-kitti-trainval.pth

modelname	training-set	flow-basemodel	flow-error (Fl-err/EPE)	expansion-error (1e4)	motion-in-depth-error (1e4)
exp-driving	Driving	flow-things	25.5%/8.874px	234.8	172.4
exp-kitti-train	Driving->KITTI-train	flow-kitti-train	6.0%/1.644px	107.3	73.6
exp-kitti-trainval	Driving->KITTI-trainval	flow-kitti-trainval	3.9%/1.144px	87.5	52.2

Try on a video sequence (>=2 frames)

Top: reference images; Bottom: mition-in-depth estimations (with kitti-finetuned model)

Run for KITTI sequence,

modelname=exp-kitti-trainval
CUDA_VISIBLE_DEVICES=0 python submission.py --dataset seq  --datapath ./input/kitti_2011_09_30_drive_0028_sync_11xx   --outdir ./weights/$modelname/ --loadmodel ./weights/$modelname/$modelname.pth  --testres 1 --fac 2 --maxdisp 512

Run for Blackbird sequence,

CUDA_VISIBLE_DEVICES=0 python submission.py --dataset seq  --datapath ./input/blackbird   --outdir ./weights/$modelname/ --loadmodel ./weights/$modelname/$modelname.pth  --testres 1 --fac 2 --maxdisp 512

Run for HD1K,

CUDA_VISIBLE_DEVICES=1 python submission.py --dataset seq  --datapath ./input/HD1K_000000_001x/   --outdir ./weights/$modelname/ --loadmodel ./weights/$modelname/$modelname.pth  --testres 1 --fac 2 --maxdisp 512

Run for Sintel,

CUDA_VISIBLE_DEVICES=1 python submission.py --dataset seq  --datapath ./input/Sintel/   --outdir ./weights/$modelname/ --loadmodel ./weights/$modelname/$modelname.pth  --testres 1 --fac 2 --maxdisp 512

Results will be saved to ./weights/$modelname/seq/ in .pfm format.

Assuming cvkit is already installed, to visualize motion-in-depth log(d2/d1),

sv weights/$modelname/seq/mid*

To visualize optical expansion,

sv weights/$modelname/seq/exp*

To visualize optical flow,

sv weights/$modelname/seq/flo*

To visualize occlusion estimation (current occlusion prediction modules gives wrong results, will release the correct ones soon),

sv weights/$modelname/seq/occ*

Evaluate on KITTI

Download KITTI-sceneflow dataset, and run

bash run_eval.sh

and results will be saved to ./weights/$modelname/2015val. It also computes error on our KITTI val set (ID 0,5,10,...195). To run this script, you'll need to point $datapath to kitti-sceneflow-path/.

To estimte optical flow and motion-in-depth on KITTI sceneflow test set, run

bash run_test.sh

and results will be saved to ./weights/$modelname/2015test.

Compute scene flow (2nd frame depth) from expansion

Simply do d2 = d1/tau, where d1 is the first frame disparity and tau is the motion-in-depth.

See demo.ipynb for a detailed walk-through of producing monocular scene flow results on KITTI.

Training

Note on flow backbone

This repo currently does not support training of optical flow. However, we provide pre-trained VCN models below. If you would like to pre-train VCN on other datasets, please use VCN repo instead. If you plan to swith to other flow backbones, modify models/VCN_exp.py L489-L513 accordingly.

Datasets

• Download the clean-pass, optical flow, disparity, and disparity change of Driving from SceneFlow
• KITTI scene flow dataset

Pre-trained optical flow models

Download pre-trained VCN models,

modelname	training-set	KITTI flow-error (Fl-err/EPE)
flow-things	C(hairs)->T(hings)	25.5%/8.874px
flow-kitti-train	C->T->K(ITTI)train	6.0%/1.644px
flow-kitti-trainval	C->T->Ktrainval	3.9%/1.144px

Train on Synthetic datasets

The following command freezes the pre-trained optical flow model and trains optical expansion on Driving for 40k iterations (taking ~9h on a TitanXp GPU).

CUDA_VISIBLE_DEVICES=0 python main.py --logname exp-d-1 --database data-path --savemodel save-path --loadflow path-to-pretrained-weights/weights/flow-things.pth.tar

Fine-tune on domain specific datasets (KITTI)

The following command freezes the pre-trained optical flow model and fine-tunes optical expansion on KITTI train set for 20k iterations.

CUDA_VISIBLE_DEVICES=0 python main.py --logname exp-kt-1 --database data-path --savemodel save-path --loadmodel path-to-pretrained-weights/weights/exp-driving.pth --loadflow path-to-pretrained-weights/weights/flow-kitti-train.pth.tar --niter 20000 --stage expansion2015train

The following command freezes the pre-trained optical flow model and fine-tunes optical expansion on KITTI train+val set for 20k iterations.

CUDA_VISIBLE_DEVICES=0 python main.py --logname exp-kt-1 --database data-path --savemodel save-path --loadmodel path-to-pretrained-weights/weights/exp-driving.pth --loadflow path-to-pretrained-weights/weights/flow-kitti-trainval.pth.tar --niter 20000 --stage expansion2015v