phoenixnn / amodal3det

Amodal Detection of 3D Objects: Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images

By Zhuo Deng, Longin Jan Latecki (Temple University). This paper was published in CVPR 2017.

License

Code is released under the GNU GENERAL PUBLIC LICENSE (refer to the LICENSE file for details).

Cite The Paper

If you use this project for your research, please consider citing:

@inproceedings{zhuo17amodal3det,
    author = {Zhuo Deng and Longin Jan Latecki},
    booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
    title = {Amodal Detection of 3D Objects: Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images},
    year = {2017}
}

Contents

1. System requirements
2. Basic Installation
3. Usage

System requirements

The code is tested on the following system:

1. OS: Ubuntu 14.04
2. Hardware: Nvidia Titan X (GPU usage: ~9GB)
3. Software: Caffe, CUDA-7.5, cuDNN v4, Matlab 2015a, Anaconda2

Basic Installation

1. clone the Amodal3Det repository:

   git clone https://github.com/phoenixnn/Amodal3Det.git
   

2. build Caffe:

   # assume you clone the repo into the local your_root_dir
   cd your_root_dir
   make all -j8 && make pycaffe

3. install cuDNN:

   sudo cp cudnn_folder/include/cudnn.h /usr/local/cuda-7.5/include/
   sudo cp cudnn_folder/lib64/*.so* /usr/local/cuda-7.5/lib64/

Usage

1. Download NYUV2 dataset with 3D annotations and unzip:

   wget 'https://cis.temple.edu/~latecki/TestData/DengCVPR2017/NYUV2_3D_dataset.zip' -P your_root_dir/dataset/NYUV2/

2. Download precomputed 2D segment proposals based on MCG3D and unzip:

   wget 'https://cis.temple.edu/~latecki/TestData/DengCVPR2017/Segs.zip' -P your_root_dir/matlab/NYUV2/

3. Download pretrained models and unzip:

   wget 'https://cis.temple.edu/~latecki/TestData/DengCVPR2017/pretrained.zip' -P your_root_dir/rgbd_3det/

   VGG16 comes from the Caffe Model Zoo, but is provided here for your convenience.

4. Run "your_root_dir/matlab/NYUV2/pipeline.m" in Matlab to extact required data.

5. Set up training/test data:

   run "setup_training_data.py" and "setup_testing_data.py" under your_root_dir/rgbd_3det/data respectively

6. Train model:

   cd your_root_dir
   ./trainNet.sh

7. Test model: run "test_cnn.py"

amodal3det's Issues

Data visualization

hi phoenixnn,

can you share the code or the tool used to visualize the detected bounding boxes?

  Where is the model file?

Hi:
I can't find  model file。

can matlab2014a do?

can matlab2014a do?

missing pre-trained weight

I found that in your training script, the training process require "rgbd_det_init_3d_allvgg.caffemodel"
but there is no such file in the 'model' directory downloaded from your link.
Am I suppose to use other pre-trained weight from model zoo instead ?

Thanks for your reply in advance

How to get Rtilt

I'm preparing a new dataset to run your code, but I don't know how to get a Rtilt for each image.
I have the 3D box and the camera's parameters.

Creating SGDSolver fails while training

Hi,

I get a problem while training the network using the command ./trainNet.sh

The log of the process ends:

Output will be saved to `output`
initializing network ...
normalizing bbox 3d regression targets (bbox_3d_targets) ...

and then the program exits, without reporting any errors.

I find the problem code may be the line 36 in /rgbd_3det/lib/cnn/train.py:
self.solver = caffe.SGDSolver(solver_prototxt)

but have no idea why and what happens.

On the other hand, the testing runs successfully.

Why you use gt2Dsel only in extract_GTinfo.m?

I think it should use gt2Dsel and gt2Dvis to obtain 2D ground truth,right?

what is the difference between gt2Dvis and gt2Dsel in annotations?

I found sometimes it is empty in gt2Dsel,but the corresponding it is not empty in gt3D and gt2Dvis

You can convert matlab code to python code?

Matlab - Segs file - Candidates.superpixel and candidates.label?

@phoenixnn - In the Segs file that you provided, the images have information in the form of candidates.superpixel and candidates.label. What exactly is that? Can you please provide the code to get that.

I wanted to use your code on another dataset but I cannot do that since I do not have candidates.superpixel and candidates.label information.

Thanks.

no module named cython_bbox

I meet the issue"No Module named cython_bbox",when I train model with the commend "./trainNet.sh".
I have install cython.I use python2.7 ,caffe cpu_only,ubuntu16.04.
Could anyone help tell me the reason?

where should i run trainNet.sh?

I meet an error when i run trainNet.sh under rgbd_3det/ . ImportError: No module named cython_bbox．how should I do ?

1080ti can't train

HI, i ues 1080ti(11G) and k40(12G) and allows meet the problem that "Check failed: error == cudaSuccess (2 vs. 0) out of memory",my cuda and cudnn is install correctly..what is the wrong?thanks!

About pipeline

Hi phoenixnn,
In get_dmap_f.m, it seems like dmap_f is not well defined. What is the dmap_f represent of ?

why do you minus 40 and 44 in your camera intrinsic matrix?

Hi Deng,

In the pipeline file: get_proposals.m,
you get the camera matrix by K = GetIntrinsicMat('nyu_color_standard_crop').

However, you subtract cx and cy by 40 and 44 due to the crop ( 427 x 561)), however, compared with original size of 480x640, I am not sure how to get these two number.

Could you please help me on that?

Shuaifeng

where can I load rgbd_det_init_3d_allvgg.caffemodel?

rgbd_det_init_3d_allvgg.caffemodel doesn't exist in models . where can I load rgbd_det_init_3d_allvgg.caffemodel?

Quesion about the make step!!

the running the following command:
make all -j8 && make pycaffe
i got the error:
make: *** No rule to make target 'all'. Stop.

How can i fix it?
Thanks man~

Proposal has NaN

After I followed steps in README.md to generate data and trained with those data,
the loss of 3d regression kept returning 'nan'.
And I finally found out that, somehow, the proposals generated by 'get_proposals.m' will causing some center of boxes become 'nan', which consist approximate 1% of proposals.
After removing those proposals, I could successfully train and test with those data,
anybody encounter the same problem?

Model file

Loss stays constant during training

I tried to train the network. However, as I noticed the loss stays constant throughout training.
loss_bbox_3d = around 0.6
loss_cls = around 1.4

Is anyone else facing this issue?