yifjiang / relative-depth-using-pytorch

Depth from a single RGB image using crowd-sourcing relative-depth datasets. The pytorch implementation of the NIPS paper: Single-Image Depth Perception in the Wild.

relative-depth-using-pytorch

The pytorch implementation of the NIPS paper:

Single-Image Depth Perception in the Wild, Neural Information Processing Systems (NIPS).

Setup

1.Install pyTorch as described in http://pytorch.org.

2. Clone this repo.

    git clone https://github.com/yifjiang/relative-depth-using-pytorch.git
   

3. Download and extract the DIW dataset from the project site. Download and extract DIW_test.tar.gz and DIW_train_val.tar.gz into 2 folders. Run the following command to download and extract DIW_Annotations.tar.gz. Then modify the filepath to images in DIW_test.csv, DIW_train.csv and DIW_val.csv to be the absolute file path where you extracted DIW_test.tar.gz and DIW_train_val.tar.gz.

    cd relative_depth
    mkdir data
    cd data
    wget https://vl-lab.eecs.umich.edu/data/nips2016/DIW_Annotations_splitted.tar.gz
    tar -xzf DIW_Annotations_splitted.tar.gz
    rm DIW_Annotations_splitted.tar.gz
   

Training and evaluating the networks

Testing on pre-trained models

Please first run the following commands to download the test data from our processed NYU dataset:

cd relative_depth
wget https://vl-lab.eecs.umich.edu/data/nips2016/data.tar.gz
tar -xzf data.tar.gz
rm data.tar.gz
cd data
python convert_csv_2_h5.py -i 750_train_from_795_NYU_MITpaper_train_imgs_800_points_resize_240_320.csv
python convert_csv_2_h5.py -i 45_validate_from_795_NYU_MITpaper_train_imgs_800_points_resize_240_320.csv

Then change directory into /relative_depth/src/experiment.

1. To evaluate the pre-trained model Ours(model trained on the NYU labeled training subset) on the NYU dataset, run the following command:

    python test_model_on_NYU.py -num_iter 1000 -prev_model_file ../results/Best_model_period1.pt -mode test -crop 8
   

2. To test on a single image, we provide a handy script test_on_one_image.py:

    python test_on_one_image.py -prev_model_file ../results/Best_model_period1.pt -input_image ../../data/4.png -output_image ../../data/4-out.png
   

   and also a cpu version of it (this script is confirmed to be able to run with pyTorch ver 0.1.12 and torchvision ver 0.1.18):

    python test_on_one_image_cpu.py -prev_model_file ../results/Best_model_period1.pt -input_image ../../data/4.png -output_image ../../data/4-out.png
   

Training

Please first change directory into /relative_depth/src/experiment.

To train the model Ours(model trained on the NYU labeled training subset), please run the following command:

python main.py -lr 0.001 -bs 4 -it 100000 -t_depth_file 750_train_from_795_NYU_MITpaper_train_imgs_800_points_resize_240_320.csv -v_depth_file 45_validate_from_795_NYU_MITpaper_train_imgs_800_points_resize_240_320.csv -rundir ./results

relative-depth-using-pytorch's Issues

DataLoader.py TypeError!

_this_sample_hdf5 = self.relative_depth_handle['hdf5_handle']['/data'][_hdf5_offset:_hdf5_offset+5,0:n_point]#todo:check this

TypeError:file object has no attribute 'getitem'

UpsamplingNearest2d' object has no attribute 'mode'

run the test_on_one_image.py

Processing sample ../../data/4.png
/usr/local/lib/python3.5/dist-packages/torch/nn/modules/upsampling.py:133: UserWarning: nn.UpsamplingNearest2d is deprecated. Use nn.Upsample instead.
warnings.warn("nn.UpsamplingNearest2d is deprecated. Use nn.Upsample instead.")
Traceback (most recent call last):
File "test_on_one_image.py", line 73, in
main()
File "test_on_one_image.py", line 43, in main
batch_output = (model(_processed_input)).float()
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/pi/Downloads/relative-depth-using-pytorch-master/src/experiment/models/hourglass.py", line 118, in forward
return self.seq(x)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/pi/Downloads/relative-depth-using-pytorch-master/src/experiment/models/hourglass.py", line 101, in forward
return self.list0+self.list1
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/pi/Downloads/relative-depth-using-pytorch-master/src/experiment/models/hourglass.py", line 77, in forward
return self.list0+self.list1
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/pi/Downloads/relative-depth-using-pytorch-master/src/experiment/models/hourglass.py", line 52, in forward
return self.list0+self.list1
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/pi/Downloads/relative-depth-using-pytorch-master/src/experiment/models/hourglass.py", line 27, in forward
return self.list0+self.list1
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/upsampling.py", line 134, in forward
return super(UpsamplingNearest2d, self).forward(input)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/upsampling.py", line 79, in forward
return F.upsample(input, self.size, self.scale_factor, self.mode)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 398, in getattr
type(self).name, name))
AttributeError: 'UpsamplingNearest2d' object has no attribute 'mode'

What version of Pytorch is used for this network?

AttributeError: 'UpsamplingNearest2d' object has no attribute 'mode'
How to solve this mistake？

How long to train a good model?

Thank you for your excellent code!
How long to get a good model based on DIW from scratch? I found the loss decrease very slowly...

ImportError when load the pretrained module

there's an importerror when I run:
python test_model_on_NYU.py -num_iter 1000 -prev_model_file ../results/Best_model_period1.pt -mode test -crop 8
ImportError: No module named models.hourglass

Error in running

Hello,
It seems that the content of links have been changed. In models, there is no .pt files. There are only .t7 files.

I cannot run the code!

Thanks

batch_input_cpu::error

when I test a image, i get error as following:
File "test_on_one_image.py", line 43, in main
_batch_input_cpu[0,:,:,:] = (img)
RuntimeError: The expanded size of the tensor (320) must match the existing size (240) at non-singleton dimension 2
Others, when i solve the size error, it show the model error?
Could you tell me how to solve it ?
And I want to konw the version of the packages you using.
Thank you!

pretrained model ?

Is there a pretrained model provided with this implementation ?

The convert_csv_2_h5.py?

Could you upload the file?I can't find it anywhere.
thank you so much!!

Error about tensor size when run testing on pre-trained models

Here is the error

Traceback (most recent call last):
  File "test_model_on_NYU.py", line 248, in <module>
    _batch_input_cpu[0] = t(crop(img))
RuntimeError: The expanded size of the tensor (320) must match the existing size (240) at non-singleton dimension 2

Not able to access the https://vl-lab.eecs.umich.edu/data/nips2016/data.tar.gz

I am getting 404 error in https://vl-lab.eecs.umich.edu/data/nips2016/data.tar.gz site. Could you please give me the updated URL?

Test Errors

Great job reproducing the system.

I was wondering how close you got to the numbers reported in the original paper?

'BatchNorm2d' object has no attribute 'track_running_stats'

Hi!! I'm using your network inside my little project (i'm using Pytorch 0.4.0 on GPU), but every time i try to test the pretrained (Ours model) by launching the command:
"python test_model_on_NYU.py -num_iter 1000 -prev_model_file ../results/Best_model_period1.pt -mode test -crop 8"
(NYU dataset correctly downloaded, command launched in the "src/experiments" directory)
this error occurs:

Traceback (most recent call last):
File "test_model_on_NYU.py", line 261, in
batch_output = model(Variable(_batch_input_cpu).cuda())
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/orobix/projects/fbarone/software/relarive-depth/relative-depth-using-pytorch/src/experiment/models/hourglass.py", line 118, in forward
return self.seq(x)
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 56, in forward
if self.training and self.track_running_stats:
File "/home/orobix/miniconda3/envs/fbarone/lib/python3.7/site-packages/torch/nn/modules/module.py", line 518, in getattr
type(self).name, name))
AttributeError: 'BatchNorm2d' object has no attribute 'track_running_stats'

I think it is version related, but i'm not sure.
Best regards!

Typo in loss calculation

Hi Yifan, there seems to be a typo in this line. It should be z_B instead of a_B.

Just wondering - how did the entire thing work with this typo in place? Or is this piece of code never used in actual training?