zhou13 / nerd

NeRD: Neural 3D Reflection Symmetry Detector

NeRD: Neural 3D Reflection Symmetry Detector

This repository contains the official PyTorch implementation of the paper: Yichao Zhou, Shichen Liu, Yi Ma. "NeRD: Neural 3D Reflection Symmetry Detector". CVPR 2021.

Introduction

We present NeRD, a Neural 3D Reflection Symmetry Detector, which combines the strength of learning-based recognition and geometry-based reconstruction to accurately recover the normal direction of objects' mirror planes. NeRD uses coarse-to-fine strategy to enumerate the symmetry planes and then find the best ones by building 3D cost volumes to examine the intra-image pixel correspondence from the symmetry.

Qualitative Measures

ResNet Regression	NeRD	ResNet Regression	NeRD

Errors of the mirror plane are marked in red.

Code Structure

Below is a quick overview of the function of key files.

########################### Data ###########################
data/
    shapenet-r2n2/              # default folder for the shapenet dataset
    new_pix3d/                  # default folder for the pix3d dataset
logs/                           # default folder for storing the output during training
########################### Code ###########################
config/                         # neural network hyper-parameters and configurations
    shapenet.yaml               # example config for shapenet
    pix3d.yaml                  # example config for pix3d
misc/                           # misc scripts that are not important
    find-radius.py              # script for generating figure grids
sym/                            # sym module so you can "import sym" in other scripts
    models/                     # neural network architectures
        symmetry_net.py         # wrapper for loss
        mvsnet.py               # 3D hourglass
    config.py                   # global variables for configuration
    datasets.py                 # reading the training data
    trainer.py                  # general trainer
train.py                        # script for training the neural network
eval.py                         # script for evaluating a dataset from a checkpoint
plot-angle.py                   # script for ploting angle error curves
plot-depth.py                   # script for ploting depth error curves

Reproducing NeRD

Installation

For the ease of reproducibility, you are suggested to install miniconda before following executing the following commands.

git clone https://github.com/zhou13/nerd
cd nerd
conda create -y -n nerd
source activate nerd
conda install -y pyyaml docopt matplotlib scikit-image opencv tqdm
# Replace cudatoolkit=10.2 with your CUDA version: https://pytorch.org/get-started/
conda install -y pytorch cudatoolkit=10.2 -c pytorch
mkdir data logs results

Downloading the Processed Datasets

Make sure curl is installed on your system and execute

cd data
../misc/gdrive-download.sh 1zdHsSb-xHhY8imIy3uWt_XfBW_YQX4Vh shapenet-r2n2.zip
../misc/gdrive-download.sh 1tzR5WDrbYTFkVu58p8qad6CXw6Bw2yNt new_pix3d.zip
unzip *.zip
rm *.zip
cd ..

If gdrive-download.sh does not work for you, you can download the pre-processed datasets manually for ShapeNet and Pix3D, and proceed accordingly.

Training (Optional)

Execute the following commands to train the neural networks from scratch with four GPUs (specified by -d 0,1,2,3):

python ./train.py -d 0,1,2,3 --identifier baseline config/shapenet.yaml
python ./train.py -d 0,1,2,3 --identifier baseline config/pix3d.yaml

The checkpoints and logs will be written to logs/ accordingly.

Pre-Trained Models

cd logs/
../misc/gdrive-download.sh 1VVd11U_8wPtrLKjufvcdbJ59h0AJQwCW 200610-234002-8ee0ad2-shapenet-latest.zip  # ShapeNet/Symmetry
../misc/gdrive-download.sh 1HcDCWRbafN4GKC4DzeG17Fo7iXimYI0v 200513-030330-c8e671c-shapenet-finetune.zip  # ShapeNet/Depth
../misc/gdrive-download.sh 1Nug2c3u1THbPNmTaiGlR8R508nDHP43Q 201113-224159-ec0e932-pix3d-001008000.zip  # Pix3d/Symmetry
unzip *.zip
rm *.zip
cd ..

Alternatively, you can download our reference pre-trained models for ShapeNet (Symmetry), ShapeNet (Depth), and Pix3D.

Evaluation

To evaluate the models with coarse-to-fine inference for symmetry plane prediction and depth map estimation, execute

python eval.py -d 0 --output results/nerd.npz logs/<your-checkpoint>/config.yaml logs/<your-checkpoint>/checkpoint_latest.pth.tar

The error statistics are printed on the screen and the error metrics are stored in results/nerd.npz. To calculate the error metrics and plot the error-percentage curves, execute

python plot-angle.py
python plot-depth.py

Acknowledgement

This work is supported by the research grant from Sony, the ONR grant N00014-20-1-2002, and the joint Simons Foundation-NSF DMS grant 2031899. We also thank Li Yi from Google Research for his comments.

Citing NeRD

If you find NeRD useful in your research, please consider citing:

@inproceedings{zhou2021nerd,
    author = {Zhou, Yichao and Liu, Shichen and Ma, Yi},
    title = {{NeRD}: Neural 3D Reflection Symmetry Detector},
    year = {2021},
    booktitle = {CVPR},
}

nerd's Issues

shapenet-r2n2.zip doesn't exist in the GDrive

shapenet-r2n2.zip doesn't exist in the GDrive? Can you share it again?

Error link to data source

The newly committed link for Train data throws a 404 error: https://drive.google.com/drive/folders/1zdHsSb-xHhY8imIy3uWt_XfBW_YQX4Vh

I guess it has not been made public yet.

How did you prepare the ground true depth in dataset?

Thank you very much for sharing your excellent work！
My question is how did you get depth files （*.exr） from point clouds in the Shapenet dataset, can I prepare them when I face other point cloud datasets? Could you please share the method or code? Thank you ！

How to use the trained model?

How to use the trained model to generate a symmetrical plane?

Figures for visualizaton (Fig 7 in the paper/Fig 14 in supp)

Hi, Yichao

I was wondering if you can kindly tell me how to generate those nice figures for visualization (e.g. Fig 7 in the paper/Fig 14 in supp). Tried several things using matplotlib, unfortunately still did not figure out. Appreciate it if you can shae the source code to draw them.

BTW, have a nice summer break!

Cheers,
Yancong

model output?

Thank you for a great job.
I would like to ask you, what is the output result of your model? Is w normal to the plane of symmetry or is it a plane of symmetry? I thought it was the normal line of the plane of symmetry, but your resulting visualization shows the plane of symmetry, which is a conflict for me. I'd like to know your answer. Looking forward to your reply!

no pix3d config file

There is no pix3d.yaml existed in the config

Question on generating the ground truth `w0`

HI, Yichao,

I am having some trouble understading this line:
https://github.com/zhou13/symmetrynet/blob/830bd9430c1750d31faffba60e18b075374c1b42/sym/datasets.py#L95

If I understand correctly, w0 is the the ground truth w. As you mentioned the shapenet objects are symmetrically aligned w.r.t Y-Z (x=0) plane, I suppose this line should be w0_ = LA.inv(RT).T @ np.array([1, 0, 0, 0]). Or am I completely wrong?

BTW, many congrats on your promotion! Impressive work!
Looking forward to your reply.

Yancong

visual code

Can you provide visual code?

About code

hi,Hello, can you ask what version of Pytorch is needed for this code to run?
thanks！