lhao0301 / pytorch-liteflownet3

pytorch-liteflownet3

This is a personal reimplementation of LiteFlowNet3 [1] using PyTorch, which is inspired by the pytorch-liteflownet implementation of LiteFlowNet by sniklaus. Should you be making use of this work, please cite the paper accordingly. Also, make sure to adhere to the licensing terms of the authors.

For the original Caffe version of this work, please see: https://github.com/twhui/LiteFlowNet3

setup

The correlation layer is borrowed from NVIDIA-flownet2-pytorch

cd correlation_package
python setup.py install

usage

Download network-sintel.pytorch from Google-Drive . To run it on your demo pair of images, use the following command. Only sintel-model is supported now. It's tested with pytorch 1.3.0 and cuda-9.0, later pytorch/cuda version should also work.

python run.py

I am afraid that I cannot guarantee that this reimplementation is correct. However, it produced results pretty much identical to the implementation of the original authors in the examples that I tried. There are some numerical deviations that stem from differences in the DownsampleLayer of Caffe and the torch.nn.functional.interpolate function of PyTorch. Please feel free to contribute to this repository by submitting issues and pull requests.

comparison

license

As stated in the licensing terms of the authors of the paper, their material is provided for research purposes only. Please make sure to further consult their licensing terms.

references

[1]  @inproceedings{hui2020liteflownet3,
  title={LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation},
  author={Hui, Tak-Wai and Loy, Chen Change},
  booktitle={European Conference on Computer Vision},
  pages={169--184},
  year={2020},
  organization={Springer}
}

Acknowledgments

Many code of this repo are borrowed from pytorch-liteflownet. And the correlation layer is borrowed from NVIDIA-Flownet2-pytorch.

pytorch-liteflownet3's Issues

How to warp the image with optical flow (.flo)?

Thank you for your reimplementation of LiteFlowNet3 using PyTorch, it is very useful to me. However, I don’t know how to warp the image using the output optical flow file (.flo).
I use the next flow_warp function, but it does not work.

def flow_warp(x, flow, padding_mode='zeros'):
    """Warp an image or feature map with optical flow
    Args:
        x (Tensor): size (n, c, h, w)
        flow (Tensor): size (n, 2, h, w), values range from -1 to 1 (relevant to image width or height)
        padding_mode (str): 'zeros' or 'border'

    Returns:
        Tensor: warped image or feature map
    """
    assert x.size()[-2:] == flow.size()[-2:]
    n, _, h, w = x.size()
    x_ = torch.arange(w).view(1, -1).expand(h, -1)
    y_ = torch.arange(h).view(-1, 1).expand(-1, w)
    grid = torch.stack([x_, y_], dim=0).float().cuda()
    grid = grid.unsqueeze(0).expand(n, -1, -1, -1)
    grid[:, 0, :, :] = 2 * grid[:, 0, :, :] / (w - 1) - 1
    grid[:, 1, :, :] = 2 * grid[:, 1, :, :] / (h - 1) - 1
    grid += flow
    grid = grid.permute(0, 2, 3, 1)
    return torch.nn.functional.grid_sample(x, grid)

fatal error: cuda_runtime_api.h: No such file or directory

anaconda3/lib/python3.6/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:30: fatal error: cuda_runtime_api.h: No such file or directory
compilation terminated.

Is that possible to provide the pretrained model on Chairs or Chairs-Things3D ?

Thank you very much for the re-implementation.
I wonder if it is possible to provide also the pre-trained model which trained on Chairs only or on Chairs then Things3D ?
Thank you again.