Inference script about sfmlearner-pytorch

[ClementPinard]

hello,

import torch
import models
from scipy.misc import imread, imsave
import numpy as np

disp_net = models.DispNetS().cuda()
a = torch.load('/path/to/pretrained')
disp_net.load_state_dict(a['state_dict'])

img = imread('/path/to/img').astype(np.float32)
img = np.transpose(img, (2, 0, 1))
tensor_img = torch.from_numpy(img).unsqueeze(0)
tensor_img = (tensor_img/255 - 0.5)/0.2
var_img = torch.autograd.Variable(tensor_img, volatile=True)

output = model(var_img)[0]
result = output.expand(3, output.size(1), output.size(2))
imsave('/path/to/result', 25.5*result.numpy())  # max disp is 10

An opencv version could be practical also (especially for colormaps), I may add a more interactive script in the future, stay tuned !

[mathmanu]

Thank you. I put that into a script that iterates over all images in a folder. These are the files:

SfmLearner-Pytorch-infer.zip

However, I get an error when I execute it. Could you please help.

Traceback (most recent call last):
File "infer.py", line 65, in
main()
File "infer.py", line 61, in main
result = output.expand(3, output.size(1), output.size(2))
File "/user/manu/files/apps/anaconda2/envs/pytorch/lib/python3.5/site-packages/torch/autograd/variable.py", line 722, in expand
return Expand.apply(self, sizes)
File "/user/manu/files/apps/anaconda2/envs/pytorch/lib/python3.5/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward
result = i.expand(*new_size)
RuntimeError: invalid argument 1: the number of sizes provided must be greater or equal to the number of dimensions in the tensor at /opt/conda/conda-bld/pytorch_1503968623488/work/torch/lib/THC/generic/THCTensor.c:309

[ClementPinard]

Thanks! You can make a pull request to be credited for that, and I will happily merge it (when it works)

Otherwise I'll push it next week.

On your problem I think you forgot the batch size. The correct expand is then :

result = output.expand(output.size(0),3,output.size(2), output.size(3))

I'm on phone in vacations so I did not look at your zip file, but essentially, the error code is telling you that your tensor (or variable) has more dimensions than the number of sizes you are trying to expand it to which makes me believe that your output is Nx1xHxW and not 1xHxW

[mathmanu]

Thanks. Attached the corrected script here:

SfmLearner-Pytorch-infer-v2.zip

Please go ahead and push this.

Would be great if you could add the functionality for generating colorful visualization for depth.

[mathmanu]

I get a very narrow output range from the net. Wonder whether this is correct output or not.

input: 0000000000.png processed. range: (4.8397942, 4.9460983) done.
input: 0000000001.png processed. range: (4.8330798, 4.946095) done.
input: 0000000002.png processed. range: (4.8287992, 4.9460936) done.
input: 0000000003.png processed. range: (4.8377218, 4.9460936) done.
input: 0000000004.png processed. range: (4.8373523, 4.9460988) done.
input: 0000000005.png processed. range: (4.8375549, 4.9460964) done.
input: 0000000006.png processed. range: (4.8378181, 4.9461002) done.
input: 0000000007.png processed. range: (4.8391299, 4.9460955) done.
input: 0000000008.png processed. range: (4.8340302, 4.9460993) done.
input: 0000000009.png processed. range: (4.8391366, 4.9460983) done.

[ClementPinard]

Hard to say without context, what are your image ? How well did the network perform during training ?

[mathmanu]

I amusing the images in the following folder for inference:
kitti_rawdata/data/2011_09_28/2011_09_28_drive_0002_sync/image_02/data/

During training, initial losses were:
train_loss validation_loss
1.5794556995472275 0.42218122299354854

After 100 epochs, these were the training a validation losses:
1.3643422244543053 0.3900231946222853

I can see that the loss has not come down significantly.

I can see that if I set mask-loss-weight to 0 (or a very small value), then the loss comes down quickly. Otherwise the loss remains high. The smooth_loss comes down sharply, but other losses do not reduce significantly.

For this training I used a batch size of 48 for faster training, but lower batch size also has the same behavior.

I am not using a pre-trained model for the training and starting from scratch. Is this okay?

[ClementPinard]

As photometric loss is weighted by mask, it is expected that it wll be lower if discount its loss to be as close as possible to 1. (masks will all converge to 0, which is a degenerated solution to photometric loss we aim to avoid)

I also experimented convergence problems, which is strange, because I had verified it before. I'm working on it and I will keep you updated

[ClementPinard]

Hi I identified the problem, it was the intrinsics that was not reinitialized each time. I'm running a new training now and it seems to work. Can you try it ? Thanks for double checking this error with me !

Inference script will come during the week ;)

[ClementPinard]

Done