Running a FlowNet-pytorch on a single image pair about flownet2-pytorch

[HuangJunJie2017]

i run a FlowNet-pytorch on a single image pair and obtain the correct result.

running a single image pair

add this python file to the project
#run_a_pair.py

import torch
import numpy as np
import argparse

from Networks.FlowNet2 import FlowNet2#the path is depended on where you create this module
from frame_utils import read_gen#the path is depended on where you create this module 

if __name__ == '__main__':
    #obtain the necessary args for construct the flownet framework
    parser = argparse.ArgumentParser()
    parser.add_argument('--fp16', action='store_true', help='Run model in pseudo-fp16 mode (fp16 storage fp32 math).')
    parser.add_argument("--rgb_max", type=float, default=255.)
    args = parser.parse_args()

    #initial a Net
    net = FlowNet2(args).cuda()
    #load the state_dict
    dict = torch.load("/home/hjj/PycharmProjects/flownet2_pytorch/FlowNet2_checkpoint.pth.tar")
    net.load_state_dict(dict["state_dict"])
    
    #load the image pair, you can find this operation in dataset.py
    pim1 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img0.ppm")
    pim2 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img1.ppm")
    images = [pim1, pim2]
    images = np.array(images).transpose(3, 0, 1, 2)
    im = torch.from_numpy(images.astype(np.float32)).unsqueeze(0).cuda()

    #process the image pair to obtian the flow 
    result = net(im).squeeze()

    
    #save flow, I reference the code in scripts/run-flownet.py in flownet2-caffe project 
    def writeFlow(name, flow):
        f = open(name, 'wb')
        f.write('PIEH'.encode('utf-8'))
        np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)
        flow = flow.astype(np.float32)
        flow.tofile(f)
        f.flush()
        f.close()

    data = result.data.cpu().numpy().transpose(1, 2, 0)
    writeFlow("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img.flo",data)

image pairs from

https://github.com/lmb-freiburg/flownet2/tree/master/data/FlyingChairs_examples

result

[haow94]

Hi @HuangJunJie2017 ,

thank you for providing your run_a_pair code.

After running it for one pair, I wanted to adapt your function, such that it initializes the network once, but can process multiple image pairs. When I load 2 image pairs, initialize the network net, and process image pair 1 and image pair 2 with the same network net, I get an error for the second image pair, i.e. for the second time I call result = net(im2).squeeze().

Thank you very much in advance!

[black0017]

hey that would be nice!
thanks for sharing the code .
is there any code to load kitti-2015 dataset ?

[black0017]

this git uses this one : https://github.com/lmb-freiburg/flownet2-docker

you can run it on an image pair if you follow the instuctions up until : 2.1 Optical flow for two single images

you probably have to download and convert the caffe models

[hajarSokeh]

Thanks black0017, I have already worked with Flownet0.2-Caffe version. But currently, I only need a Pytorch-version of Flownet which works on a pair of images. Do you have any suggestion?

[fitsumreda]

you may take a look at this.
it is an adaptation of this repo for the purpose you mentioned.
https://github.com/vt-vl-lab/pytorch_flownet2

[fitsumreda]

we'll provide one as well.
In the meanwhile, feel free to make a pull request.

[hajarSokeh]

Many thanks fitsumreda :)

[tlatlbtle]

mark, good issue.

[smallmango]

i get the Segmentation fault (core dumped), do you know how to solve it ?thank you !

[HuangJunJie2017]

@smallmango i don't understand what problem did you encounter. What does ''segmentation fault'' mean?

[smallmango]

@HuangJunJie2017 "segmentation fault" is the error. I solved the problem by reducing pytorch version to 0.3. Thank you for your kindness!

[smallmango]

@HuangJunJie2017 hi, the input image of your demo is '.ppm', can i use '.jpg' intead?

[HuangJunJie2017]

@smallmango you can refer to the definition of "read_gen" function. Pictures in the form of png, jpeg, ppm, jpg, bin or raw are ok.

[smallmango]

@HuangJunJie2017 Thank you! I get it.

[qiuyuda]

@hajarSokeh Hi，I'm trying to test a pair images with your code, could you tell me where to get the "Flownet2" module ? I can't find it in my "networks" file.
Is it in a document like "Flownet2.py" ?

I downloaded the code from this project: https://github.com/NVIDIA/flownet2-pytorch.git

---

I found a Flownet2 class in models.py and it seems work.

[laaRaa]

Hi,
I am running flownet on a single pair of images and for that I am using run_a_pair.py (python file posted earlier in this issue). When the input size is not a power of two, or at least one of its dimensions, then I have the error below. Someone had the same error?
Thanks!

Traceback (most recent call last):
File "run_a_pair.py", line 36, in
result = net(im).squeeze()
File "/home/raad/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/raad/flownet2-pytorch040_python3_2018_9_21/models.py", line 118, in forward
flownetc_flow2 = self.flownetc(x)[0]
File "/home/raad/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/raad/flownet2-pytorch040_python3_2018_9_21/networks/FlowNetC.py", line 111, in forward
concat4 = torch.cat((out_conv4,out_deconv4,flow5_up),1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 40 and 39 in dimension 2 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:111

[HuangJunJie2017]

@laaRaa make sure that the size of the input image must be a multiple of 2^6=64. You can find more details in my comment in this issues.
#60 (comment)

[laaRaa]

@HuangJunJie2017 thanks!

[seenitall]

@HuangJunJie2017 I'm getting the following:

  File "run_a_pair.py", line 26, in <module>
    dict = torch.load(model_file, pickle_module=pickle)
  File "/root/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 358, in load
    return _load(f, map_location, pickle_module)
  File "/root/miniconda3/lib/python3.7/site-packages/torch/serialization.py", line 532, in _load
    magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, '<'.

Not sure but after researching I read it may be related to being a model saved with python 2.7?

I tried changing the encoding of the pickle module, by:

from functools import partial
import pickle
model_file = 'FlowNet2_checkpoint.pth.tar'
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
dict = torch.load(model_file, pickle_module=pickle)

.. but it doesn't help. Thank you

[HuangJunJie2017]

@seenitall 'infer_pair.py' is not the file I proposed in this issue, as the code is quite unfamiliar to me. Sorry, I have no idea what problem you encounter.

[seenitall]

@seenitall 'infer_pair.py' is not the file I proposed in this issue, as the code is quite unfamiliar to me. Sorry, I have no idea what problem you encounter.

It's your run_a_file.py script when trying to load the pre-tained weights 'FlowNet2_checkpoint.pth.tar'.

[HuangJunJie2017]

@seenitall
it seems that the 'torch.load' function can not load the pre-trained model.
There are two possible reasons: something wrong with the function itself or something wrong with the checkpoint file.
some suggest:
1.try not to specify the mode of 'torch.load' and use 'dict = torch.load(model_file)' instead
2.test the 'torch.load' by loading other checkpoint files, if it work well ,may be your checkpoint file is broken.

[sklf]

@HuangJunJie2017 Hello! Thanks for your code. I want to know how to modify your code to use the FlowNet2S. I tried "from Networks.FlowNet2S import FlowNet2S". But it didn't work. So can you please tell me how to modify your code? Thank you very much!

[HuangJunJie2017]

@sklf please detail the problem you encounter, expecially the system's feedback.

[sklf]

@HuangJunJie2017 My code is as follow:

import torch
import numpy as np
import argparse
import mmcv
from models import FlowNet2S
from utils.frame_utils import read_gen 

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--rgb_max", type=float, default=255.)
    args = parser.parse_args()
    args.skip_training = True

    
    net = FlowNet2S(args).cuda()
    # load the state_dict
    dict = torch.load("/share1/home/chunyang/files/flownet2-pytorch/work/FlowNet2-S_checkpoint.pth.tar")
    net.load_state_dict(dict["state_dict"])

    # load the image pair, you can find this operation in dataset.py
    pim1 = read_gen("/share1/home/chunyang/files/flowtrack/test/frame_000088.png")
    pim2 = read_gen("/share1/home/chunyang/files/flowtrack/test/frame_000089.png")
    images = [pim1, pim2]
    images = np.array(images).transpose(3, 0, 1, 2)
    print(images.shape)
    im = torch.from_numpy(images.astype(np.float32)).unsqueeze(0).cuda()

    # process the image pair to obtian the flow

    result = net(im).squeeze()

   
    def writeFlow(name, flow):
        f = open(name, 'wb')
        f.write('PIEH'.encode('utf-8'))
        np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)
        flow = flow.astype(np.float32)
        print("flow's shape",flow.shape)
        flow.tofile(f)
        f.flush()
        f.close()
    data = result.data.cpu().numpy().transpose(1, 2, 0)
    writeFlow("./img0.flo", data)
    mmcv.flowshow("./img0.flo",wait_time = 10000)

And the error is
Traceback (most recent call last): File "run_a_pair.py", line 33, in <module> result = net(im).squeeze() AttributeError: 'tuple' object has no attribute 'squeeze'

[sklf]

@HuangJunJie2017 My code is as follow:

import torch
import numpy as np
import argparse
import mmcv
from models import FlowNet2S
from utils.frame_utils import read_gen 

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("--rgb_max", type=float, default=255.)
    args = parser.parse_args()
    args.skip_training = True

    
    net = FlowNet2S(args).cuda()
    # load the state_dict
    dict = torch.load("/share1/home/chunyang/files/flownet2-pytorch/work/FlowNet2-S_checkpoint.pth.tar")
    net.load_state_dict(dict["state_dict"])

    # load the image pair, you can find this operation in dataset.py
    pim1 = read_gen("/share1/home/chunyang/files/flowtrack/test/frame_000088.png")
    pim2 = read_gen("/share1/home/chunyang/files/flowtrack/test/frame_000089.png")
    images = [pim1, pim2]
    images = np.array(images).transpose(3, 0, 1, 2)
    print(images.shape)
    im = torch.from_numpy(images.astype(np.float32)).unsqueeze(0).cuda()

    # process the image pair to obtian the flow

    result = net(im).squeeze()

   
    def writeFlow(name, flow):
        f = open(name, 'wb')
        f.write('PIEH'.encode('utf-8'))
        np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)
        flow = flow.astype(np.float32)
        print("flow's shape",flow.shape)
        flow.tofile(f)
        f.flush()
        f.close()
    data = result.data.cpu().numpy().transpose(1, 2, 0)
    writeFlow("./img0.flo", data)
    mmcv.flowshow("./img0.flo",wait_time = 10000)

And the error is
Traceback (most recent call last): File "run_a_pair.py", line 33, in <module> result = net(im).squeeze() AttributeError: 'tuple' object has no attribute 'squeeze'

I find the reason. The code of FlowNet2S has such content:

if self.training:
            return flow2,flow3,flow4,flow5,flow6
        else:
            return self.upsample1(flow2*self.div_flow)

And it seems like the default value of self.training is 'True'. I commented out the code and it works now. However， I still can't find how to set the value of `self.training'. So I will be grateful if you can tell me how to set it. ^_^

[HuangJunJie2017]

@sklf self.training has two value, true for training process and false for application process. my code is just for applications.
some suggest:

1. try to summarize the experience of error analysis, as the the feedback error information is clear enough for you to diagnose the problem.

2. refer to the papers of this project to learn why the framework needs the different output for training process

[AshviniKSharma]

i run a FlowNet-pytorch on a single image pair and obtain the correct result.

running a single image pair

add this python file to the project
#run_a_pair.py

import torch
import numpy as np
import argparse

from Networks.FlowNet2 import FlowNet2#the path is depended on where you create this module
from frame_utils import read_gen#the path is depended on where you create this module 

if __name__ == '__main__':
    #obtain the necessary args for construct the flownet framework
    parser = argparse.ArgumentParser()
    parser.add_argument('--fp16', action='store_true', help='Run model in pseudo-fp16 mode (fp16 storage fp32 math).')
    parser.add_argument("--rgb_max", type=float, default=255.)
    args = parser.parse_args()

    #initial a Net
    net = FlowNet2(args).cuda()
    #load the state_dict
    dict = torch.load("/home/hjj/PycharmProjects/flownet2_pytorch/FlowNet2_checkpoint.pth.tar")
    net.load_state_dict(dict["state_dict"])
    
    #load the image pair, you can find this operation in dataset.py
    pim1 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img0.ppm")
    pim2 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img1.ppm")
    images = [pim1, pim2]
    images = np.array(images).transpose(3, 0, 1, 2)
    im = torch.from_numpy(images.astype(np.float32)).unsqueeze(0).cuda()

    #process the image pair to obtian the flow 
    result = net(im).squeeze()

    
    #save flow, I reference the code in scripts/run-flownet.py in flownet2-caffe project 
    def writeFlow(name, flow):
        f = open(name, 'wb')
        f.write('PIEH'.encode('utf-8'))
        np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)
        flow = flow.astype(np.float32)
        flow.tofile(f)
        f.flush()
        f.close()

    data = result.data.cpu().numpy().transpose(1, 2, 0)
    writeFlow("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img.flo",data)

image pairs from

https://github.com/lmb-freiburg/flownet2/tree/master/data/FlyingChairs_examples

result

Hey, thanks for your work. But I am facing some issues regarding this code.
this function is taking around 5 seconds per pair of images. What is the reason behind it and how can this be made faster?

[zzzzyq-18]

@ HuangJunJie2017谢谢！我得到它。

@HuangJunJie2017 Thank you! I get it.

Excuse me read_gen define where ah, how you can use the JPG format of the test

[zzzzyq-18]

我在单个图像对上运行FlowNet-pytorch并获得正确的结果。

运行单个图像对

将此python文件添加到项目
＃run_a_pair.py

import torch
import numpy as np
import argparse

from Networks.FlowNet2 import FlowNet2#the path is depended on where you create this module
from frame_utils import read_gen#the path is depended on where you create this module 

if __name__ == '__main__':
    #obtain the necessary args for construct the flownet framework
    parser = argparse.ArgumentParser()
    parser.add_argument('--fp16', action='store_true', help='Run model in pseudo-fp16 mode (fp16 storage fp32 math).')
    parser.add_argument("--rgb_max", type=float, default=255.)
    args = parser.parse_args()

    #initial a Net
    net = FlowNet2(args).cuda()
    #load the state_dict
    dict = torch.load("/home/hjj/PycharmProjects/flownet2_pytorch/FlowNet2_checkpoint.pth.tar")
    net.load_state_dict(dict["state_dict"])
    
    #load the image pair, you can find this operation in dataset.py
    pim1 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img0.ppm")
    pim2 = read_gen("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img1.ppm")
    images = [pim1, pim2]
    images = np.array(images).transpose(3, 0, 1, 2)
    im = torch.from_numpy(images.astype(np.float32)).unsqueeze(0).cuda()

    #process the image pair to obtian the flow 
    result = net(im).squeeze()

    
    #save flow, I reference the code in scripts/run-flownet.py in flownet2-caffe project 
    def writeFlow(name, flow):
        f = open(name, 'wb')
        f.write('PIEH'.encode('utf-8'))
        np.array([flow.shape[1], flow.shape[0]], dtype=np.int32).tofile(f)
        flow = flow.astype(np.float32)
        flow.tofile(f)
        f.flush()
        f.close()

    data = result.data.cpu().numpy().transpose(1, 2, 0)
    writeFlow("/home/hjj/flownet2-master/data/FlyingChairs_examples/0000007-img.flo",data)

来自的图像对

https://github.com/lmb-freiburg/flownet2/tree/master/data/FlyingChairs_examples

结果

您好！我想请问一下您是怎么实现可视化的