lmb-freiburg / flownet2-docker

Dockerfile and runscripts for FlowNet 2.0 (estimation of optical flow)

FlowNet 2.0 Docker Image

This repository contains a Dockerfile and scripts to build and run neural networks for optical flow estimation in Docker containers. We also provide some example data to test the networks.

If you use this project or parts of it in your research, please cite the original paper of Flownet 2.0:

@InProceedings{flownet2,
  author       = "E. Ilg and N. Mayer and T. Saikia and M. Keuper and A. Dosovitskiy and T. Brox",
  title        = "FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks",
  booktitle    = "IEEE Conference on Computer Vision and Pattern Recognition (CVPR)",
  month        = "Jul",
  year         = "2017",
  url          = "http://lmb.informatik.uni-freiburg.de//Publications/2017/IMKDB17"
}

See the paper website for more details.

0. Requirements

We use nvidia-docker for reliable GPU support in the containers. This is an extension to Docker and can be easily installed with just two commands.

To run the FlowNet2 networks, you need an Nvidia GPU (at least Kepler). For the smaller networks (e.g. FlowNet2-s) 1GB of VRAM is sufficient, while for the largest networks (the full FlowNet2) at least 4GB must be available. A GTX 970 can handle all networks.

1. Building the FN2 Docker image

Simply run make. This will create two Docker images: The OS base (an Ubuntu 16.04 base extended by Nvidia, with CUDA 8.0), and the "flownet2" image on top. In total, about 8.5GB of space will be needed after building. Build times are a little slow.

2. Running FN2 containers

Make sure you have read/write rights for the current folder. Run the run-network.sh script. It will print some help text, but here are two examples to start from:

2.1 Optical flow for two single images

• we use the full FlowNet2 variant for maximum accuracy
• we assume that we are on a single-GPU system
• we want debug outputs, but not the whole network stdout

$ ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo

2.2 Optical flow for entire lists of images

• we use the lean FlowNet2-s variant for maximum speed
• we want to use GPU "1" on a multi-GPU system
• we want to see the full network stdout printfest

$ ./run-network.sh -n FlowNet2-s -g 1 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt

NOTE: All the network outputs will be files belonging to "root". As a regular user, you cannot change these files, but you can copy them to files that belong to you, and then delete the originals:

$ cp 0000000-flow.flo user-owned-0000000-flow.flo
$ rm 0000000-flow.flo

3. License

The files in this repository are under the GNU General Public License v3.0

flownet2-docker's Issues

How can I see result file

I installed nvidia docker and entered make.
After that, I follow your examples

command
./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo
result
00000-imgL.png data/0000001-imgL.png flow.flo Using GPU: 0 Running network: FlowNet2 Working dir: /flownet2/flownet2/scripts First input: data/0000000-imgL.png Second input: data/0000001-imgL.png Output: flow.flo

command
./run-network.sh -n FlowNet2-s -g 1 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt
result
Using GPU: 1 Running network: FlowNet2-s Working dir: /flownet2/flownet2/scripts First input: data/flow-first-images.txt Second input: data/flow-second-images.txt Output: data/flow-outputs.txt

but I can't see any flo.flo file in data folder.
I'm sorry about asking this silly question but I'm a newbie in docker.
So I don't know much about it.
My goal is calculating optical flow by flownet2
If I can see the results, it will help me a lot.

So how can I find my results and convert them into readable format?

Hash Sum mismatch: shared-mime-info

Just in case when running "make" someone is getting the error (Hash Sum mismatch) referent to the package shared-mime-info.

I solved it by modifying the flownet2-docker/Dockerfile.
In Line 20, under command "RUN apt-get update && \ " add the instruction to specifically install this package " apt-get install shared-mime-info -y --no-install-recommends && \ "

That worked for me.

path issue

I ran flownet2 for entire lists of images on the sample images successful. However, when I wrote three new .txt file for Sintel dataset, the error occurs as "img0 does not exist: /input-output/data/SINTEL/test/clean/alley_1/frame_0001.png".

I checked the .py file and found that the dockerize_filepath joins '/input-output' with the dirname I wrote. I wonder how the sample images can be run successful without join the /input-output to the path. And how can I run with my own image lists.

How to open .flow file?

cuda error when feeding list of images

First single image example execute without error.
However, when feeding list the following error shows.

$ sudo ./run-network.sh -n FlowNet2-s -g 1 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt
Using GPU:       1
Running network: FlowNet2-s
Working dir:     /flownet2/flownet2/scripts
First input:     data/flow-first-images.txt
Second input:    data/flow-second-images.txt
Output:          data/flow-outputs.txt
/flownet2/flownet2/scripts
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1107 11:47:48.281453    10 common.cpp:166] Check failed: error == cudaSuccess (10 vs. 0)  invalid device ordinal
*** Check failure stack trace: ***
/bin/bash: line 1:    10 Aborted                 (core dumped) python run-flownet-docker.py --verbose --gpu 1 /flownet2/flownet2/models/FlowNet2-s/FlowNet2-s_weights.caffemodel* /flownet2/flownet2/models/FlowNet2-s/FlowNet2-s_deploy.prototxt.template data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt

Different resolution of input images

Hello, first of all, thanks a lot for providing docker version! It really saves me a lot of time for setting up the environment.

My question is that it seems that all the image pairs listed in flow_first_images.txt & flow_second_images.txt should be of the same resolution; otherwise while running run-network.sh the network would crash.

For example, suppose I have flow_first_images.txt like this:

A1.jpg
A2.jpg
A3.jpg
B1.jpg
B2.jpg
B3.jpg

with flow_second_images.txt being:

A2.jpg
A3.jpg
A4.jpg
B2.jpg
B3.jpg
B4.jpg

where all A*.jpg are frames saved from video A with resolution 406*720, and all B*.jpg are frames saved from video B with resolution 960*720. Then after starting run-network.sh, the network would crash when it switches from A part to B part.

I guess it is because the network is initialized based on the resolution of the first part of images, which is the part A in the above example. It is unable to deal with different resolution once it is initialized.

So I wonder if there exists a faster way than that I create different sets of flow_first_images.txt, flow_second_images.txt, and flow_output.txt files for several groups of images while each group with different resolution?

Installation problem such as RPC failed and gnutls_handshake() failed

while I have prepared all the requirements for make in the flownet-docker directory, in the step 14, it failed with different reasons:
(1)
Step 14/18 : RUN git clone https://github.com/lmb-freiburg/flownet2 && cp ./FN2_Makefile.config ./flownet2/Makefile.config && cp ./FN2_run-flownet-docker.py ./flownet2/scripts/run-flownet-docker.py && cd flownet2 && rm -rf .git && cd models && bash download-models.sh && rm flownet2-models.tar.gz && cd .. && make -jnproc && make -jnproc pycaffe
---> Running in e21fff5da1e7
Cloning into 'flownet2'...
fatal: unable to access 'https://github.com/lmb-freiburg/flownet2/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
The command '/bin/sh -c git clone https://github.com/lmb-freiburg/flownet2 && cp ./FN2_Makefile.config ./flownet2/Makefile.config && cp ./FN2_run-flownet-docker.py ./flownet2/scripts/run-flownet-docker.py && cd flownet2 && rm -rf .git && cd models && bash download-models.sh && rm flownet2-models.tar.gz && cd .. && make -jnproc && make -jnproc pycaffe' returned a non-zero code: 128
Makefile:8: recipe for target 'flownet2' failed
make: *** [flownet2] Error 128

(2)
Step 14/18 : RUN git clone https://github.com/lmb-freiburg/flownet2 && cp ./FN2_Makefile.config ./flownet2/Makefile.config && cp ./FN2_run-flownet-docker.py ./flownet2/scripts/run-flownet-docker.py && cd flownet2 && rm -rf .git && cd models && bash download-models.sh && rm flownet2-models.tar.gz && cd .. && make -jnproc && make -jnproc pycaffe
---> Running in ef5027eaa99b
Cloning into 'flownet2'...
error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
The command '/bin/sh -c git clone https://github.com/lmb-freiburg/flownet2 && cp ./FN2_Makefile.config ./flownet2/Makefile.config && cp ./FN2_run-flownet-docker.py ./flownet2/scripts/run-flownet-docker.py && cd flownet2 && rm -rf .git && cd models && bash download-models.sh && rm flownet2-models.tar.gz && cd .. && make -jnproc && make -jnproc pycaffe' returned a non-zero code: 128
Makefile:8: recipe for target 'flownet2' failed
make: *** [flownet2] Error 128

Actually I have made it several times, and it can successfully make the final result with about 8 Gb in the docker images. But this time, it stopped in the 14 step and there is only 2.76 Gb in the docker images. I have tried my git, it is fast with a proxy setting. Is there any suggestions?

about image quality

Everything works fine for me, I can generate .flo without trouble and I can convert them to .png
However, when I see the .png image it looks very pixelated as in Figure 1 (left) of this paper:

• https://lmb.informatik.uni-freiburg.de/Publications/2017/IMSKDB17/paper-FlowNet_2_0__CVPR.pdf

but I would like to have the image as in figure 1 (right)

can you help me?

How long should a frame take to process?

On a 1280x720 image, it's taking about 8 seconds using a K80.

I'm not seeing much activity on my GPU, and the CPU is pinned at 100%, so I'm inclined to think the work is not being done on the GPU.

I am using a modified dockerfile, so maybe I screwed something up there. I just downgraded in order to use an older version of cuda. https://github.com/positlabs/fast-artistic-videos-docker/blob/master/flownet2-docker/Dockerfile

Seems like the drivers / nvidia-docker are working, at least with the style transfer stuff.

How can I generate 2-channel optical flow image and save its x, y channel as .jpg image?

After using flownet2.0 method, I have generated *.flo files. But I don't know how to convert * .flo files to 2-channel optical flow image and save its x, y channel as .jpg image. Can you help me? Thanks so much.

CUDA 8 on GTX 690 doesn't work -> could I use with CUDA 7.5 ?

I am trying to put docker-nvidia on Debian 9 with GTX 690 and there is a problem with the installation of the driver with CUDA kernel. I would like to know if I can use FlowNet2 with CUDA 7.5 ?

Thank you.

Will this work on latest version of CUDA 10? I am new please suggest me the changes in case it won’t I am getting error docker:error response from daemon:oci runtime create failed when i the 2.1

No output

I do not have any output when using run-network.sh.
The printed output is :

Using GPU:       1
Running network: FlowNet2
Working dir:     /flownet2/flownet2/scripts
First input:     ./data/0000000-imgL.png
Second input:    ./data/0000001-imgL.png
Output:          ./flow.flo

But I have no file created.
I do not know what "flownet/flownet/scripts" folder the script refers to.
Is there anything I am doing wrong ?

About the installation problem.

I have tested your project before. It give me good result.
But now while for a new computer, I met problems. I have installed all the things I think. But while input the command to run the test, it just give me result very quickly even without time cost.

while I type in the command:
sudo ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo

it gives me feedback of:
Using GPU: 0
Running network: FlowNet2
Working dir: /flownet2/flownet2/scripts
First input: data/0000000-imgL.png
Second input: data/0000001-imgL.png
Output: flow.flo

But it takes no time to load paramters and I can't find flow.flo file.

My docker version is:
Docker version 19.03.5

Docker images:
REPOSITORY TAG IMAGE ID CREATED SIZE
flownet2 latest c98d87d4f617 4 minutes ago 8.14GB
nvidia/cuda 8.0-cudnn5-devel-ubuntu16.04 db7f23a9ae16 7 weeks ago 1.85GB

NVIDIA and gpu information:
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GTX 980
Result = PASS

It is quite strange and I have tried many times. Problem is still here.

standard_init_linux.go:211: exec user process caused "exec format error"

Not possible to do Make

I tried to install via make. I have read that this mistake is often due to the architecture. Im trying to install it on the Nvidia-Jetson-Xavier.

My environment (Nvidia-Jetson-Xavier)

Linux jetson-desktop 4.9.140-tegra (#)1 SMP PREEMPT Mon Dec 9 22:52:02 PST 2019 aarch64 aarch64 aarch64 GNU/Linux

Linux installed via NVIDIA SDK Manager
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"

Full Error Message

sudo make
docker build                    \
       -f Dockerfile            \
       -t flownet2              \
       --build-arg uid=$UID    \
       --build-arg gid=$GROUPS \
       .
Sending build context to Docker daemon  15.36kB
Step 1/18 : FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
 ---> db7f23a9ae16
Step 2/18 : WORKDIR "/flownet2"
 ---> Using cache
 ---> 6d11a4bf3625
Step 3/18 : COPY FN2_Makefile.config ./
 ---> Using cache
 ---> 416096567bdc
Step 4/18 : COPY FN2_run-flownet-docker.py ./
 ---> Using cache
 ---> e6e2413e4ad8
Step 5/18 : ARG uid
 ---> Using cache
 ---> 03907a57a7c0
Step 6/18 : ARG gid
 ---> Using cache
 ---> f7148ec1f168
Step 7/18 : ENV uid=${uid}
 ---> Using cache
 ---> 245a61872056
Step 8/18 : ENV gid=${gid}
 ---> Using cache
 ---> 07fa35abb8b7
Step 9/18 : ENV USER=flownet
 ---> Using cache
 ---> 69498a94a870
Step 10/18 : ENV GROUP=flownet
 ---> Using cache
 ---> a83c4ce4d014
Step 11/18 : RUN mkdir -p /home/$USER                                               &&     echo "${USER}:x:${uid}:${gid}:${USER},,,:/flownet2:/bin/bash"                  >> /etc/passwd                                                &&     echo "${GROUP}:x:${gid}:${uid}"                                                >> /etc/group
 ---> Running in e664aa8646db
standard_init_linux.go:211: exec user process caused "exec format error"
The command '/bin/sh -c mkdir -p /home/$USER                                               &&     echo "${USER}:x:${uid}:${gid}:${USER},,,:/flownet2:/bin/bash"                  >> /etc/passwd                                                &&     echo "${GROUP}:x:${gid}:${uid}"                                                >> /etc/group' returned a non-zero code: 1
Makefile:8: recipe for target 'flownet2' failed
make: *** [flownet2] Error 1

If you have any information for me to solve this problem, I would be very grateful. Thank you in advance.

I can't see any output in the root folder

I run the example:
$ ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo

Using GPU: 0
Running network: FlowNet2
Working dir: /flownet2/flownet2/scripts
First input: data/0000000-imgL.png
Second input: data/0000001-imgL.png
Output: flow.flo

but no .flo output can be found

os.system error

I'm trying to use os.system( ) to use FlowNet and convertion together by running one .py file.
However there is a problem

this command works for me ( compare 2 images)
os.system("bash run-network.sh -n FlowNet2-c -v data/0000000-imgL.png data/0000001-imgL.png flow.flo")

but another command doesn't work for me ( multiple comparison )
os.system("run-network.sh -n FlowNet2-s -g 0 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt")

after running that command, I get

Using GPU: 0 Running network: FlowNet2 Working dir: /flownet2/flownet2/scripts First input: data/flow-first-images.txt Second input: data/flow-second-images.txt Output: data/flow-outputs.txt /flownet2/flownet2/scripts Done!

However there's no result files.
Also there's no log command while running.
How can I solve this problem?

variable input sizes

To allow variable input sizes, Line 92 in FN2_run-flownet-docker.py should be exchanged by sth like

if width != input_data[0].shape[3] or height != input_data[0].shape[2]:

where width and height might be initialised with -1.

convert .flo file to .npy file

I want to get vector map of optical flow
So I try to convert .flo file into vector map file such as .npy file.
However I can only find the way how to convert it into .png file or something for image file.
Can you help me with this matter?

Also I want to try many layers and make custom network
but I'm not familiar with Docker and Ubuntu, so I don't know how to access and modify source codes
I can trace back the source codes to get vector map of optical flow if there's no way to convert .flo to .npy or something else.

Fine-tuning models in Docker containers

Hi,
I've tried to use the docker to reproduce fine-tune models for SINTEL datasets as follows:
enter the container as:

> $ nvidia-docker run --volume "${PWD}:/input-output:rw" -it "flownet2" /bin/bash   

enter the input-output folder(where the edited .prototxt are kept)

> $ cd .....into/the/folder

Fine-tuning the models with

> $ ./../../../flownet2/flownet2/build/tools/caffe train --solver ../solver.prototxt --weights FlowNet2_weights.caffemodel --gpu 0

It start to train, however, the fine-tuned models is much worse than the original one. in 5000 iterations the average EPE for the whole datasets is about 11, in 10000 iterations the EPE is about 9. I wonder if you could have any idea where I did is wrong.

In my opinion, the lmdb file may leads to the error, but I'm not sure. Will it matters on account that the SINTEL ground truth is .flo but not .pfm like other datasets?

Best regards

CUDA driver version is insufficient for CUDA runtime version

Hi，I met some difficulties and tried all night without success.
My host OS:Ubuntu 18.04 GPU:NVIDIA 2070 Super CUDA_Version:10.2(host os) Driver_Version:440.100

I entered the command:
python scripts/run-flownet.py models/FlowNet2/FlowNet2_weights.caffemodel.h5 models/FlowNet2/FlowNet2_deploy.prototxt.template
Images/image_left/0001.png Images/image_left/0002.png
Images/image_left/01.flo

Then,the error as follows:
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0626 15:00:13.070574 21 common.cpp:160] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
Aborted (core dumped)

It seems that I have a problem with the driver version or CUDA version?But I do not know how to solve it,Please give me some advice,Thank you!

make error while building on docker.

Hi, i was try to build it using docker on my windows machine using make, after some time i get below error.
Can you please help me with this!!

viewing the output

Hi,

I have compiled and run a test sample. However, the output is in .flo format. Can you tell me how I can view the output, exporting it as a .png or other data file?

libcudart.so.8.0: cannot open shared object file: No such file or directory

sudo make

ran successfully. It created two images flownet2 and nvidia/cuda. But, when i try to run

sudo ./run-network.sh -n FlowNet2-s -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt

or

sudo ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo

It says ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory.

Specification
Distribution: Ubuntu 18.10 (PopOS)
CPU: Intel i7-3840QM
GPU: Nvidia Quadro k2000m (Dedicated 2GB)
RAM: 8 GB

No Outputs

I face the same issue as the last one. I run the cmd provided by your readme.md. But I got no outputs. Where is the output .flo file supposed to appear? I run the
"./run-network.sh -n FlowNet2 -vv data/0000000-imgL.png data/0000001-imgL.png /home/v-xiangli/flownet2-docker/flow.flo"
on Ububtu server (not local server, I cannot get root right and only have key-pair to ssh login). I don't have the permission to change files in root folder but I didn't get error when generation.

I just tried to use my local vm server to run. It posts:
Using GPU: 0
Running network: FlowNet2
Working dir: /flownet2/flownet2/scripts
First input: data/0000000-imgL.png
Second input: data/0000001-imgL.png
Output: flow.flo
/flownet2/flownet2/scripts
Network forward pass using /flownet2/flownet2/models/FlowNet2/FlowNet2_weights.caffemodel.h5.
Succeeded.
Done!

The server which I cannot find outputs posts:
Using GPU: 0
Running network: FlowNet2
Working dir: /flownet2/flownet2/scripts
First input: data/0000000-imgL.png
Second input: data/0000001-imgL.png
Output: flow.flo

Could you please help me with that? Thank you!

.flo file data

Now I'm using data from .flo file.
However I want to know exact magnitude of vector
So does .flo file provide vector by pixel size?
(ex/ (2, 5) means 2 pixel for x axis and 5 pixel for y axis)