path issue about flownet2-docker

[uthynauta]

Ok, here's what I did:

In my TXT file I copied (from the command line window) the name it says is not finding, in my case data/optflow/000039.png and pasted it into the TXT, ran it again and now I'm getting this new error:

./run-network.sh -n FlowNet2-s -g 0 -v data/names1.txt data/names2.txt data/output.txt
Using GPU:       0
Running network: FlowNet2-s
Working dir:     /flownet2/flownet2/scripts
First input:     data/names1.txt
Second input:    data/names2.txt
Output:          data/output.txt
/flownet2/flownet2/scripts
Network forward pass using /flownet2/flownet2/models/FlowNet2-s/FlowNet2-s_weights.caffemodel.
Succeeded.
Traceback (most recent call last):
  File "run-flownet-docker.py", line 146, in <module>
    writeFlow(out, blob)
  File "run-flownet-docker.py", line 38, in writeFlow
    f = open(name, 'wb')
IOError: [Errno 2] No such file or directory: '/input-output/\xef\xbb\xbfdata/optflow/flow038.flo'

[ZikeYan]

Finally I get it. Yes, it's the symbol link problem. I forgot about it. I copy the dataset to the folder and now it works.

Really appreciate your kindness and sorry for wasting your time :p

[nikolausmayer]

Hi,

/input-output is the mountpoint inside the container where the current working directory will be mounted, i.e. if data/SINTEL/test/clean/alley_1/frame_0001.png is a valid file, it is visible as /input-output/data/SINTEL/test/clean/alley_1/frame_0001.png inside the container.

Note that the data must be within the directory from which you call the run-network.sh script. I am not sure whether symlinks work.

My best guess is that your list files do not contain the correct paths. Note that the paths must be relative to $PWD, not relative to the list file.

Best,
Nikolaus

[ZikeYan]

I checked the path and didn't find the mistake. Maybe I didn't know how docker works. I'll present the my problem as follows and wish that you could find something wrong:

The $PWD is ~/develop/flownet2-docker
the Sintel dataset is copied in the ./data folder as ~/develop/flownet2-docker/data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png

Error occurs as "BaseException: img0 does not exist: /input-output/data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png"
when I run:

$ ./run-network.sh -n FlowNet2-s -g 1 -vv ./run-network.sh -n FlowNet2-s -g 1 -vv data/Sintel1_list_test.txt data/Sintel2_list_test.txt data/Sintel_flo_list_test.txt
or
$./run-network.sh -n FlowNet2-s -g 1 -vv data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png data/SINTEL/test/clean/PERTURBED_market_3/frame_0002.png flow.flo

However, it succeed when running:

$ ./run-network.sh -n FlowNet2 -v data/0000000-imgL.png data/0000001-imgL.png flow.flo
or
$ ./run-network.sh -n FlowNet2-s -g 1 -vv data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt
as the readme.md instructed

The file do exist because it gives the image information when running:

$ file data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png

[nikolausmayer]

Hm, nothing bad so far. Can you please post the first lines of all three .txt files?

[ZikeYan]

1.data/SINTEL/test/clean/PERTURBED_market_3/frame_0001.png
2.data/SINTEL/test/clean/PERTURBED_market_3/frame_0002.png
3.Result/SINTEL/PERTURBED_market_3/frame_0001.flo

[nikolausmayer]

Strange. Can you start an interactive container and manually look what's in /input-output?

nvidia-docker run --rm --volume "${PWD}:/input-output:rw" -it "flownet2" /bin/bash

[ZikeYan]

it says "root@8997ac58b23c:/flownet2#". seemingly it's empty

[nikolausmayer]

Hmm, in flownet2 there should be at least these entries
FN2_Makefile.config FN2_run-flownet-docker.py flownet2/
and /input-output should contain whatever was in the folder from which you executed that command...

Can you please post some info about your system? What's your OS and Docker version?

[ZikeYan]

Strange as it is, the three entries

FN2_Makefile.config, FN2_run-flownet-docker.py & flownet2
appears.

However, the error still occurs

[nikolausmayer]

That's really weird. If /input-output is empty, then the list files that ship with this repo should not work either, and neither should running the networks on single images. Did you do all of your tests in the same folder? (I remember I had problems in an NFS-mounted location)

[nikolausmayer]

Great, no worries 👍 I should probably put that into the README.

[uthynauta]

I'm sorry I'm having the same issue, what is that "Symbol link problem" that ZikeYan is talking about?

[nikolausmayer]

@uthynauta If you are accessing the data in /input-output via some symbolic link, this can cause problems. I think this can happen when the symlink points into another filesystem.

Example: In this repo there is a data folder with images:

$ ls -l data
drwxr-xr-x  2 user group 4,0K Jun  1  2017 ./
drwxr-xr-x 13 user group 4,0K Jun  6 15:51 ../
-rw-r--r--  1 user group 686K Apr 26  2017 0000000-imgL.png
-rw-r--r--  1 user group 680K Apr 26  2017 0000001-imgL.png
...

but you might have links there, too:

$ ls -l data
drwxr-xr-x  2 user group 4,0K Jun  1  2017 ./
drwxr-xr-x 13 user group 4,0K Jun  6 15:51 ../
-rw-r--r--  1 user group 686K Apr 26  2017 0000000-imgL.png
-rw-r--r--  1 user group 680K Apr 26  2017 0000001-imgL.png
lrwxrwxrwx  1 user group   14 Jun  7 18:59 a_symbolic_link -> /media/user/0000002-imgL.png
...

These links can cause problems (I think it's because within the Docker container, you are not your own user and you don't have your usual access permissions).

[uthynauta]

I'm such a n00b I'm sorry, apparently running

sudo ./run-network.sh -n FlowNet2-s -g 0 -v data/flow-first-images.txt data/flow-second-images.txt data/flow-outputs.txt

does work, but when running my own list:

sudo ./run-network.sh -n FlowNet2-s -g 0 -v data/names1.txt data/names2.txt data/output.txt

I get this message again:

BaseException: img1 does not exist: /input-output/data/optflow/000039.png

I've already logged into the docker and navigated to /input-output/data and everything seems on its place. I'm really lost in this.

[nikolausmayer]

We are all noobs and we can only hope to continue learning from slightly lesser noobs 😏

Can you post the outputs of some commands?

• ls -l in data from outside the container
• ls -l in /input-output from within the container
• ls -l data/optflow/000039.png outside the container
• stat data/optflow/000039.png outside the container
• stat /input-output/data/optflow/000039.png in the container

Are you on a normal PC? Or in a cloud instance?

[uthynauta]

I've already ran an ls -l and apparently I don't have any symlinks. Could it be that the format of the TXT files is wrong?

[nikolausmayer]

That's also possible, but I don't think it's the case here. Your container throws this error:
BaseException: img1 does not exist: /input-output/data/optflow/000039.png
So it has already correctly parsed the txt file.

[uthynauta]

From outside the container for data

ls -l data
total 38988
-rwxrwxrwx 1 uthy uthy  702321 may 23 13:58 0000000-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000001-imgL.flo
-rwxrwxrwx 1 uthy uthy  696171 may 23 13:58 0000001-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000002-imgL.flo
-rwxrwxrwx 1 uthy uthy  728680 may 23 13:58 0000002-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000003-imgL.flo
-rwxrwxrwx 1 uthy uthy  705242 may 23 13:58 0000003-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000004-imgL.flo
-rwxrwxrwx 1 uthy uthy  668696 may 23 13:58 0000004-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000005-imgL.flo
-rwxrwxrwx 1 uthy uthy  649812 may 23 13:58 0000005-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000006-imgL.flo
-rwxrwxrwx 1 uthy uthy  650301 may 23 13:58 0000006-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000007-imgL.flo
-rwxrwxrwx 1 uthy uthy  666499 may 23 13:58 0000007-imgL.png
-rwxrwxrwx 1 uthy uthy 4147212 jun  7 12:32 0000008-imgL.flo
-rwxrwxrwx 1 uthy uthy  686211 may 23 13:58 0000008-imgL.png
drwxrwxrwx 5 uthy uthy    4096 may 24 12:29 cabina
drwxrwxrwx 2 uthy uthy    4096 may 23 20:50 cflow
drwxrwxrwx 2 uthy uthy    4096 may 23 18:57 flow
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:09 flow-first-images.txt
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:10 flow-outputs.txt
-rwxrwxrwx 1 uthy uthy     176 jun  7 12:10 flow-second-images.txt
drwxrwxrwx 2 uthy uthy    4096 may 23 20:50 flow_vis
-rwxrwxrwx 1 uthy uthy    6984 jun  7 12:15 names1.txt
-rwxrwxrwx 1 uthy uthy    6987 jun  7 11:31 names2.txt
drwxrwxrwx 2 uthy uthy   12288 jun  7 12:14 optflow
-rwxrwxrwx 1 uthy uthy    8733 jun  7 11:28 output.txt
drwxrwxrwx 5 uthy uthy  180224 may 23 20:49 PNG
-rwxrwxrwx 1 uthy uthy  302049 may 23 13:58 teaser.png
drwxrwxrwx 2 uthy uthy    4096 may 23 19:01 test
-rwxrwxrwx 1 uthy uthy     136 jun  7 11:56 text.txt

From within the container:

root@2565204a334b:/flownet2# ls -l /input-output
total 4132
-rw-rw-r-- 1 1000 1000    1837 May 23 18:58 Dockerfile
-rw-rw-r-- 1 1000 1000    1746 May 23 18:58 FN2_Makefile.config
-rw-rw-r-- 1 1000 1000    4913 Jun  7 17:03 FN2_run-flownet-docker.py
-rw-rw-r-- 1 1000 1000   35141 May 23 18:58 LICENSE
-rw-rw-r-- 1 1000 1000      91 May 23 18:58 Makefile
-rw-rw-r-- 1 1000 1000    2997 May 23 18:58 README.md
drwxrwxrwx 9 1000 1000    4096 Jun  7 17:15 data
-rw-r--r-- 1 root root 4147212 Jun  7 16:54 flow.flo
-rw------- 1 root root      62 May 23 19:07 nohup.out
-rwxrwxr-x 1 1000 1000    5624 May 23 18:58 run-network.sh
-rw-rw-r-- 1 1000 1000     147 May 30 22:01 runme.sh

From outside the container:

ls -l data/optflow/000039.png
-rwxrwxrwx 1 uthy uthy 358295 jun  7 11:12 data/optflow/000039.png

stat data/optflow/000039.png
stat: cannot stat 'data/optflow/000039.png': No such file or directory

Inside the container:

root@6e528d8c6492:/flownet2# stat /input-output/data/optflow/000039.png
  File: '/input-output/data/optflow/000039.png'
  Size: 358295    	Blocks: 704        IO Block: 4096   regular file
Device: 807h/2055d	Inode: 2931184     Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/ UNKNOWN)   Gid: ( 1000/ UNKNOWN)
Access: 2018-06-07 16:42:58.025642979 +0000
Modify: 2018-06-07 16:12:18.106159381 +0000
Change: 2018-06-07 17:52:23.862836387 +0000
 Birth: -

[uthynauta]

Ok, here's what happened, I do not know how I solved it but I solved it.

I opened the TXT files from the original GIT download, deleted the info contained there and added my list, then I saved it with another name. Now everything is working fine. I guess it had something to do with the format of the TXT file, I leave it here in case anyone else faces the same problem.

I really appreciate the time and effort you dedicated to these @nikolausmayer

[nikolausmayer]

Ok.. this
IOError: [Errno 2] No such file or directory: '/input-output/\xef\xbb\xbfdata/optflow/flow038.flo'
definitely looks like a formatting problem (\xef\xbb\xbf are bad characters in the path).

It's so strange that you can stat the file from within the container, but not outside... anyway, I'm glad that it works now 😉