failed to convert docker://kubernetes/redis-slave:v2 about docker2aci

[iaguis]

Hmmm... I just converted that image successfully on master:

$ docker2aci docker://kubernetes/redis-slave:v2
Downloading 511136ea3c5a: [====================================] 32 B/32 B
Downloading f0dde87450ec: [====================================] 65.8 MB/65.8 MB
Downloading 76b658ecb564: [====================================] 71.5 KB/71.5 KB
Downloading 4faa69f72743: [====================================] 682 B/682 B
Downloading 2103b00b3fdf: [====================================] 32 B/32 B
Downloading 4303edc5e986: [====================================] 85.4 MB/85.4 MB
Downloading 14dfde0f38c5: [====================================] 643 B/643 B
Downloading d489fc3e869f: [====================================] 438 B/438 B
Downloading a8253f7d9cc1: [====================================] 22.7 KB/22.7 KB
Downloading 38cf7a363b14: [====================================] 32 B/32 B
Downloading d9343f29a1c4: [====================================] 32 B/32 B
Downloading 9cb3dca21a1f: [====================================] 32 B/32 B
Downloading c9eda0ec97c0: [====================================] 2.93 MB/2.93 MB
Downloading a44f8ddf5fb8: [====================================] 32 B/32 B
Downloading cf0fc1a9cd2b: [====================================] 32 B/32 B
Downloading 8567350addd7: [====================================] 32 B/32 B
Downloading dff9dfa9220d: [====================================] 32 B/32 B
Downloading 3fd8168c2f23: [====================================] 511 B/511 B
Downloading 85b1106c3ae7: [====================================] 544 B/544 B
Downloading 48a1e2b8f859: [====================================] 32 B/32 B

Converted volumes:
        name: "volume-data", path: "/data", readOnly: false

Converted ports:
        name: "6379-tcp", protocol: "tcp", port: 6379, count: 1, socketActivated: false

Generated ACI(s):
kubernetes-redis-slave-v2.aci

The only differences between master and the version in rkt are some README updates.

Maybe some weird corruption. Can you reproduce it consistently?

[iaguis]

Just noticed a small difference in the outputs:

--- working.txt 2015-07-30 10:41:39.506387827 +0200
+++ error.txt   2015-07-30 10:42:20.592816839 +0200
@@ -1,7 +1,7 @@
 Downloading 511136ea3c5a: [====================================] 32 B/32 B
 Downloading f0dde87450ec: [====================================] 65.8 MB/65.8 MB
 Downloading 76b658ecb564: [====================================] 71.5 KB/71.5 KB
-Downloading 4faa69f72743: [====================================] 682 B/682 B
+Downloading 4faa69f72743: [====================================] 679 B/679 B
 Downloading 2103b00b3fdf: [====================================] 32 B/32 B
 Downloading 4303edc5e986: [====================================] 85.4 MB/85.4 MB
 Downloading 14dfde0f38c5: [====================================] 643 B/643 B

[yifan-gu]

Interesting... I got the error several times yesterday, but I cannot reproduce now. Going to close it...

[jonboulle]

Weird - bit flips? solar flare?

[yifan-gu]

I am going to reopen this as I found the failure has a high possibility to appear if I fetch two images at the same time:

#!/bin/bash
sudo /home/yifan/rkt/build-rkt-0.7.0+git/bin/rkt --debug=false --insecure-skip-verify=true fetch docker://kubernetes/redis-slave:v2&
sudo /home/yifan/rkt/build-rkt-0.7.0+git/bin/rkt --debug=false --insecure-skip-verify=true fetch docker://kubernetes/redis-slave:v2&

And I got:

error converting docker image to ACI: error squashing image: error validating image: archive/tar: invalid tar header
error converting docker image to ACI: error squashing image: error validating image: archive/tar: invalid tar header

Sometimes I got:

error copying image: gzip: invalid header
error copying image: gzip: invalid header

or

error converting docker image to ACI: error squashing image: error validating image: archive/tar: invalid tar header
error copying image: gzip: invalid header

Sounds like there is a race here.
The temporal workaround for k8s is not to allow multiple fetches.

[iaguis]

#76 should fix it

[sgotti]

@iaguis I was thinking that the race was caused primarly by rkt:

Now running two rkt fetch passes to docker2aci.Convert the same tmpDir as the output dir (https://github.com/coreos/rkt/blob/master/rkt/images.go#L385), so what happens is that a file generated by the first rkt is overwritten by the second. Probably that's not a problem if the first rkt already has an open file descriptor to the first file but it doesn't look very clean.

Perhaps providing different tmpDir will be cleaner.

[iaguis]

It is docker2aci who opens the file for writing the squashed image.

I think writing the squashed image to a tempfile and then rename it (the solution in #76) is the right thing to do.

[sgotti]

@iaguis sorry I wasn't clear. #76 look good to me on the docker2aci side.

But on the rkt side, when docker2aci.Convert is executed by multiple rkt processes fetching the same image the same tmpDir is used by all the processes and the same file generated by the first rkt finishing the conversion is then overwritten by the second one rkt finishing the conversion. It'll be cleaner to use a real tmpDir with a random name instead of just "/var/lib/rkt/tmp/".
Probably it won't hurt as the two generated files should be identical and probably the first rkt already has an open fd on it (https://github.com/coreos/rkt/blob/master/rkt/images.go#L402).

[iaguis]

Yeah, I agree we should fix that in rkt too although "it should work fine"

[yifan-gu]

fwiw, two fetches of the same docker://redis can result in two different sha512 hashes.

[yifan-gu]

But on the rkt side, when docker2aci.Convert is executed by multiple rkt processes fetching the same image the same tmpDir is used by all the processes and the same file generated by the first rkt finishing the conversion is then overwritten by the second one rkt finishing the conversion. It'll be cleaner to use a real tmpDir with a random name instead of just "/var/lib/rkt/tmp/".
Probably it won't hurt as the two generated files should be identical and probably the first rkt already has an open fd on it (https://github.com/coreos/rkt/blob/master/rkt/images.go#L402).

+1 There is a race here. Or can we just provide a random file name to docker2aci.Convert, AFAICS the file name doesn't really matter here.

[yifan-gu]

Thanks @iaguis !