dogestry / dogestry Goto Github PK

View Code? Open in Web Editor NEW

417.0 417.0 29.0 3.07 MB

Use Amazon S3 as your Docker registry - NO LONGER SUPPORTED

Go 99.60% Shell 0.40%

dogestry's Introduction

Dogestry

Simple CLI app + server for storing and retrieving Docker image(s) from Amazon S3.

Prerequisites

AWS account with at least one S3 bucket
Go 1.4 or higher (development only)
github.com/tools/godep (development only)
Docker 1.4 or higher

Installation

If you just want to run Dogestry, get the binary release that works for your platform. It's a statically linked binary: there are no dependencies. Download and run it!

If you prefer to build it yourself, clone the repo and godep get && godep go build

Usage

Setup

Typical S3 Usage:

$ export AWS_ACCESS_KEY=ABC
$ export AWS_SECRET_KEY=DEF
$ export DOCKER_HOST=tcp://localhost:2375
$ dogestry push s3://<bucket name>?region=us-east-1 <image name>
$ dogestry pull s3://<bucket name>?region=us-east-1 <image name>

Push

Push the hipache image to the S3 bucket ops-goodies located in us-west-2:

dogestry push s3://ops-goodies/ hipache

Push the hipache image to the S3 bucket ops-goodies located in us-west-2 with tag latest:

dogestry push s3://ops-goodies/ hipache:latest

Pull

Pull the hipache image and tag from S3 bucket ops-goodies:

dogestry pull s3://ops-goodies/ hipache

Pull the hipache image and tag from S3 bucket ops-goodies with tag latest:

dogestry pull s3://ops-goodies/ hipache:latest

If you want to pull an image from S3 to multiple hosts, you can use the -pullhosts option. The value for the -pullhosts option is a comma-separated list of hosts, in the following format: tcp://[host][:port] or unix://path.

The s3 version, with pullhosts:

dogestry -pullhosts tcp://host-1:2375,tcp://host-2:2375,tcp://host-3:2375 s3://ops-goodies/docker-repo/ hipache

List

List the images in the S3 bucket ops-goodies:

$ dogestry list s3://ops-goodies/
REPOSITORY                                                    TAG
hipache                                                       0.3.1
hipache                                                       latest
hipache                                                       0.2.8
hipache                                                       0.2.4

Server Mode (Accelerator)

Dogestry can also be run in server mode with the -server parameter; doing so can dramatically speed up image pulls when using -pullhosts. It also directly supports pulls from the docker client itself.

To make use of server mode:

Deploy and run Dogestry with the -server param on all Docker servers that are the destinations of the '-pullhosts' parameter. This can be done directly, or inside a container with access to the docker.sock.
Ensure your firewall on the host(s) is configured to allow incoming requests on port 22375 (this is what dogestry server listens on by default). Map this port if running it in a container.
Perform your pull (with -pullhosts) as usual:

$ dogestry -pullhosts tcp://host-1:2375,tcp://host-2:2375,tcp://host-3:2375 s3://ops-goodies/docker-repo/ hipache

Dogestry (client) will automatically detect that the remote host is running Dogestry server and issue the pull command directly to the host (instead of pulling the image down first and then uploading it to the host via Docker API).

In addition, you can also perform a pull against a server running Dogestry, avoiding the need for the dogestry binary:

# Update your .dockercfg to include your AWS credentials
$ dogestry login opsgoodies.com
Updating docker file /root/.dockercfg...
AWS_ACCESS_KEY: MyAwsAccessKey
AWS_SECRET_KEY: MyAWSSecretKey
S3_URL: s3://ops-goodies
# You can now pull via the Docker binary
$ docker -H tcp://host-1:22375 pull opsgoodies.com/docker-repo/hipache

S3 files layout

Dogestry will create two directories within your S3 bucket called "images" and "repositories". Example contents:

Images:

images/5d4e24b3d968cc6413a81f6f49566a0db80be401d647ade6d977a9dd9864569f/layer.tar
images/5d4e24b3d968cc6413a81f6f49566a0db80be401d647ade6d977a9dd9864569f/VERSION
images/5d4e24b3d968cc6413a81f6f49566a0db80be401d647ade6d977a9dd9864569f/json

Repositories:

repositories/myapp/20131210     (content: 5d4e24b3d968cc6413a81f6f49566a0db80be401d647ade6d977a9dd9864569f)
repositories/myapp/latest       (content: 5d4e24b3d968cc6413a81f6f49566a0db80be401d647ade6d977a9dd9864569f)

License

The MIT License (MIT)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

dogestry's People

Contributors

Stargazers

Watchers

dogestry's Issues

Ability to push several images (or tags) at once

For example dogestry push image:1.0.0 image:latest

It would be nice to have ability to push several images at once. In my case if I build image and tag it with custom tag and 'latest' I need to push this image two times. Although second run is fast, it's a little bit annoying.

Cannot connect to Docker endpoint

Installed from binary.

Why is it trying to use locallhost instead of localhost?

dogestry push s3://xxx-xxx/?region=us-west-2 yyyyyy:latest
WorkDir: /tmp/dogestry078930855/yyyyyy_latest
Using docker endpoint for push: tcp://locallhost:2375
Error getting image history: cannot connect to Docker endpoint
2015/12/01 12:23:03 cannot connect to Docker endpoint

Remove the download and upload commands

Now that dogestry pull can handle pulling to multiple docker hosts, there's no need for separate download and upload commands.

Installation instructions on Wiki?

I could not see any instructions on how to install dogestry. Also, doesn't you provide a download link to a dogestry binary? Or the only option would be to clone the repository and build it on my own?

Signature Version support v4

Frankfurt region (eu-central-1) uses only signature v4. When I try to push something there i get this message:
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

Optimizations to exporting an image

Capturing @relistan's ideas for reducing the IO writes.

Get the list of missing layers in S3 that needs to be uploaded. Start streaming the tar file from docker export for an image. For each file that is encountered in the streaming tar, only uncompress the files that are in the missing layers list, skip over the files that are not in the missing list. This way we only uncompress/write the missing layers to disk.
If item 1 is not possible, then export the docker image as a tar file into the temp folder. Instead of untaring the whole archive and picking the files we need to upload, only pull the necessary files from the tar archive directly, thus saving a disk write for untaring all the files.

/cc @toffer

Make the pull command work with multiple docker hosts

When using dogestry to deploy to multiple hosts, it would be ideal to be able to make a single call to dogestry pull in order to download from S3 one time, and then upload the image to multiple hosts.

Currently, to pull to multiple hosts, you would either run multiple separate dogestry pull processes, which means re-downloading the images from S3 multiple times, or else you need to call dogestry download, followed by multiple calls to dogestry upload.

To configure multiple docker hosts, we could add a -hosts command line option, so that the interface would look like this:

$ dogestry pull -hosts tcp://host1:port1,tcp://host2:port2 s3://bucket image-name

Dogestry push must fail immediately when something is wrong during upload.

This is causing a bug when an image is missing one or two files.

Dogestry does not tag image if image already exist without the tag

If image already exists dogestry says "there's nothing to do" and it does not tag that image.

Handle downloading large files from S3 better

During the first part of the dogestry pull command, when dogestry downloads files from S3, I've seen several instances where downloading a large (1+ GB) layer.tar file will fail.

One time the failure resulted in an "Unexpected EOF" Error,. Another time, dogestry just hung midway through the download, never completing the download, but also never exiting.

To make this more robust, we should explore downloading large files in chunks (using the range header in the AWS S3 REST API.)

Don't send unnecessary files to pull hosts

The dogestry pull command consists of two parts:

Download the needed files from S3.
Tar up the downloaded files and upload them to each pull host.

However, right now, the tarball created contains all files that are downloaded from S3. Sometimes, that includes files that aren't needed by a particular pull host.

For instance, say you are deploying a docker image to Host_A and Host_B. The docker image was created by adding an application layer on top of a base image. Now, let's say that Host_A already has a copy of the base image, while Host_B doesn't. When dogestry compiles a list of files to download from S3, it will include all the files for the base image and the application layer, since Host_B requires all those files to assemble the complete docker image. Then, it tars up all downloaded files and sends the tarball to both Host_A and Host_B.

It would be more efficient if dogestry only uploaded the files needed on each host. In this case, only send the application layer to Host_A, while sending all layers to Host_B.

A simple solution would be to create multiple download directories, one for each pull host, and copy the files required for each host into the appropriate directory. Then, it's easy to create a separate tarball for each pull host. However, that's not very efficient in terms of disk space.

Perhaps a better solution would be to create multiple download directories, but rather than copy the files, instead create symlinks. Then, pass the -h (or --dereference) option to tar when creating the tarball. Even better, perhaps we could do the equivalent with the go tar package, and not have to shell out at all in cli.sendTar(). (Note: I haven't tested this approach at all. Just brainstorming...)

Region is ignored while pulling

dogestry pull 's3://foo-bar/?region=eu-west-1' foo:2015-11-30T11.46.59.190Z.2de802a

got this error:

Errors during getFiles:
map[images/75dc35548036ee0dac9587d690d0d1da5de7609e245ba29585ccad6ac44cd248/VERSION:400: 
"The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'"
images/75dc35548036ee0dac9587d690d0d1da5de7609e245ba29585ccad6ac44cd248/json:400: 
"The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'"
images/75dc35548036ee0dac9587d690d0d1da5de7609e245ba29585ccad6ac44cd248/layer.tar:400:
"The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'"]

2.0.2 everything works fine

Rethink default value for docker host

In config.GetDockerHost(), if the DOCKER_HOST environment variable isn't found, the docker host is set to tcp://localhost:2375.

This is often not the correct value. Since the docker daemon binds to unix:///var/run/docker.sock by default, perhaps that would be a better default.

The other alternative that I think would be better than the current behavior is to not set a default value at all, which would force the user to set the DOCKER_HOST environment variable.

Add retry logic for S3 500 errors

S3 can sometimes return a 500; add logic to retry a $few times before giving up.

push/pull doesn't respect <path name> per documentation

The doc says:

$ dogestry push s3://<bucket name>/<path name>/?region=us-east-1 <image name>

I tried making changes to <path name> but it appears to be ignored. The images and repositories directories are always created in the root of the S3 bucket.

Am I doing something wrong?

It would be nice if the path was working, since it allows managing multiple registries without the overhead of managing a bucket for each (and IAM roles, etc).

Don't export docker image, if all layers already exist in the remote.

When pushing an image, dogestry will export the docker image, even if all of the layer files are already present in the remote. It would be more optimal to check the remote first, and if all of the layers are already present, then we don't need to export the image at all. Instead, we just need to write the repositories/<image_name>/<tag_name> metadata file (that contains the id of the "top layer" of the image.)

This optimization is useful when pushing a single image that has multiple tags. The first push will take the usual amount of time, but pushing each subsequent tagged version will be much faster.

(Of course, if any of the layers are different, or not present in the remote, then we will still need to export the entire image, since there's no docker API call to export individual layers.)

Always overwrite the repositories metadata file in a dogestry push

During an S3 push, all keys are checked before uploading, so that no unnecessary uploading is done.

This works fine for all keys beginning with images/, but it causes problems when the repositories/<image_name>/<tag_name> isn't updated.

Example:

# Build my_service image
$ docker build -t my_service .

# Tag image
$ docker tag my_service my_service:foo

# Push image
$ dogestry push s3://bucket_name my_service:foo

# Make change to my_service
(edit files...make changes)

# Re-build docker image
$ docker build -t my_service .

# Update tag
$ docker tag my_service my_service:foo

# Push updated image
$ dogestry push s3://bucket_name my_service:foo

After pushing the updated image, the repositories/my_service/foo key should contain the id of the new docker image layer created. But, since the key already exists, it isn't updated, so the contents still have the old value.

The solution is always to upload a new repositories/<image_name>/<tag_name> key during every push.

Update goamz import

I noticed that the aws package we use is no longer maintained at its current repo, but instead has moved to a new one:

https://github.com/crowdmob/goamz -> https://github.com/AdRoll/goamz

We should update our imports accordingly.

Add a list command

Right now, there's no good way to see what images/tags exist on the remote. It would be nice to have a dogestry list command.

Possible output:

image             tag
----------------  ------
busybox           latest
progrium/busybox  foo
progrium/busybox  latest

Push of a non-existant image blows up with `index out of range`

$ DOCKER_HOST=tcp://docker2:2375 dogestry push s3://mybucket/
Using docker endpoint for push: tcp://docker2:2375
Remote: s3(bucket=mybucket, region=us-east-1)
Error getting image history: no such image
Checking layers on remote
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/dogestry/dogestry/cli.(*DogestryCli).exportToFiles(0xc2080a20b0, 0x7fff5fbff938, 0x1e, 0x1025488, 0xc208042240, 0xc2080102a0, 0x61, 0x0, 0x0)
    /Users/kmatthias/go/src/github.com/dogestry/dogestry/cli/push.go:223 +0x814
github.com/dogestry/dogestry/cli.(*DogestryCli).CmdPush(0xc2080a20b0, 0xc20800a020, 0x2, 0x2, 0x0, 0x0)
    /Users/kmatthias/go/src/github.com/dogestry/dogestry/cli/push.go:58 +0x711
reflect.callMethod(0xc20803cf30, 0xc20830dda0)
    /usr/local/go/src/reflect/value.go:605 +0x179
reflect.methodValueCall(0xc20800a020, 0x2, 0x2, 0xc20803cf30, 0xc20803e001, 0x1, 0x1, 0x12, 0x0, 0x0, ...)
    /usr/local/go/src/reflect/asm_amd64.s:29 +0x36
github.com/dogestry/dogestry/cli.(*DogestryCli).RunCmd(0xc2080a20b0, 0xc20800a010, 0x3, 0x3, 0x0, 0x0)
    /Users/kmatthias/go/src/github.com/dogestry/dogestry/cli/cli.go:122 +0x29e
main.main()
    /Users/kmatthias/go/src/github.com/dogestry/dogestry/main.go:81 +0x4be

Use a better S3 client when uploading and downloading

The existing S3 client is pretty naive in terms of connection handling and it does not even retry at all.

https://github.com/rlmcpherson/s3gof3r does everything we need:

checksum
retries
multipart stream upload/download, in parallel.

Note that this library does only upload and download. So we still need the other library for the rest of the features. Using this library will fix: #55 and potentially #43

push skips layers that are missing layer.tar on S3

I don't know the root cause, but here's the symptom:

dogestry pull fails for a particular image
Inspect the layer on S3 -- notice there is no layer.tar
Run dogestry push from a machine with the image -- nothing gets copied
Delete the layer directory on S3
Run dogestry push again -- this time the layer is copied
dogestry pull succeeds

I think this might be fixed if S3Remote.ImageMetadata verified that json, layer.tar, and VERSION all exist for the layer.

I'm going to test a patch on my fork for this, but if anyone else has suggestions, please chime in.

must push container twice

After a push a remote pull results in "No such image"
After the second push the image is found.

See output here:
https://gist.github.com/chrisferry/55762840104b0ae075a9

Program hangs on failed upload

If one of the file uploads fails, dogestry will hang indefinitely. This is because the channel closing logic in S3Remote.Push(..) is blocking, and will thus hang if any of the push goroutines has completed.

The for loop on line 170 of s3.go is the main culprit. It makes a blocking write to each channel in goroutineQuitChans[]. A separate go routine has already been launched for each of those channels, and some of them may have completed successfully by the time the channel closing logic runs. For each finished channel c, there is no go routine reading goroutineQuitChans[c]. This makes the channel write on line 171 block forever.

I believe that the write to the quit channels should be changed to be non-blocking. I've been able to fix this on my local branch with the following change:

for i := 0; i < numGoroutines; i++ {
    select {
    case goroutineQuitChans[i] <- true:
    default:
}

Push to S3 should check if file exists on S3 already

When pushing files to S3 using s3.Push(), dogestry will always upload the file to S3, even if the file already exists in S3.

It would be much more efficient to check if the file exists in S3 first, and skip the call to s3.putFile() if it does.

Add a "version" option/command

Right now, there's no way to tell what version of dogestry you are running. It would be nice if you could use the following to see the version:

$ dogestry -v
$ dogestry -version
$ dogestry version

Make `bucket` be configurable as environment variable

to use dogestry like this:

export AWS_ACCESS_KEY=ABC
export AWS_SECRET_KEY=DEF
export DOCKER_HOST=tcp://localhost:2375
export AWS_BUCKET=s3://<bucket name>/<path name>/?region=us-east-1
dogestry push  <image name>
dogestry pull <image name>

Error in sendTar, no such file or directory

Hi,

I've tried to pull images from s3 and found that sometimes I get an error "no such file or directory":

Importing image(504273d1d86d) TAR file to docker hosts: [tcp://192.168.59.103:2376]
Loading image to: tcp://192.168.59.103:2376
[{"Host":"tcp://192.168.59.103:2376","Status":"failed","Error":"API error (404): open /mnt/sda1/var/lib/docker/tmp/docker-import-632579556/repo/c0ed0404a62c757b25a98dcfd6f7ea5ad5408e56c60277cebfc848adc458ef03/layer.tar: no such file or directory\n"}]
2015/04/04 20:17:05 Error in sendTar

This happens on my mac with boot2docker using TCP connection, as well as on linux with native LXC with unix socket. Is there any additional logging I can find out as to why this is happening.

Cheers,
Julian.

High RAM usage during push and pull

We have some moderately large images (~1.5GB total) that we run on some machines with small RAM footprints.

During dogestry push and dogestry pull invocations I'm seeing RAM usage that approaches the size of the image. For example:

23149 root      20   0 1374236 1.145g   4116 S   4.3 14.7   0:15.78 dogestry

In one instance I was unable to complete the pull due to RAM constraints.

I'm going to try to figure this one out and send a PR, but if anyone has some theories on where to look that would help. My first guess is the code that's calculating the SHA1 for a layer.

Should we vendor all dependencies?

We currently vendor the gcfg package, but we don't vendor any of the others we depend on. I'm relatively new to Go, so I'm not sure if vendoring everything is the best practice, but, I started thinking about this when the goamz package moved to a new repo: #30.

These are the additional dependencies we would need to vendor:

"github.com/crowdmob/goamz/aws"
"github.com/fsouza/go-dockerclient"
"github.com/mitchellh/go-homedir"

In addition, these are the dependencies of our dependencies. (This may not be a complete list...Someone would need to double-check my work here.) Would vendoring these be required also?

"github.com/docker/docker/pkg"
"github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar"
"github.com/Sirupsen/logrus"

cc: @didip @relistan @amjith

Support for instance profile (roles)

dogestry needs AWS Keyid and Secret Key to work. It would be quite nice if it can use Instance profile roles for credentials to assess S3.

deadlock in S3Remote.Push

We saw this on our CI box today. goroutine dump here:

https://gist.github.com/coopernurse/6bd79f786372cd82ec10

goroutine 1 is blocked on a channel send (line 172): goroutineQuitChans[i] <- true
goroutines 261 and 267 are blocked on a channel send: putFileErrChan <- putFileResult{}

My guess is if we make the putFileErrChan buffered to numGoRoutines we would avoid this.

I have a patched version running on our CI box now. I'll open a PR

Push doesn't work with buckets in eu-central-1 and eu-west-1

Bug with eu-central-1: #71

Bug with eu-west-1: it just hangs forever on step "Pushing files to S3 remote:" and nothing happens.

panic: send on close channel - during 'dogestry push'

This could be user error, but my guess is there's at least a problem with some error handling code.

version:

[root@localhost conf]# dogestry version
Dogestry 2.0.1

[root@localhost conf]# uname -a
Linux localhost.localdomain 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

command:

dogestry push s3://mybucket-docker?region=us-west-2 someuser/register

error:

Pushing files to S3 remote:
2015/07/07 14:59:59 error when uploading to S3: 307: "Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests."
{"Status":"error", "Message": "Error when uploading to S3: 307: "Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests.""}
panic: send on closed channel

goroutine 131 [running]:
github.com/dogestry/dogestry/remote.func·002()
    /Users/didip/go/src/github.com/dogestry/dogestry/remote/s3.go:137 +0x24e
created by github.com/dogestry/dogestry/remote.(*S3Remote).Push
    /Users/didip/go/src/github.com/dogestry/dogestry/remote/s3.go:143 +0x499

dogestry pull doesn't create the tag when the image id is available in local docker

It is possible to push the same image which is tagged with multiple names to S3. But while pulling it back from S3 using dogestry, only one of the tags is pulled back. The next time we pull the same image with a different tag, the new tag is not created in the local docker.

Steps:

Create a new tag for an existing image.

docker tag <image_id> test_tag

Push the original image and the new tag using dogestry.

dogestry push s3://bucket-name original_image
dogestry push s3://bucket-name test_tag

Delete the image and it's tag in local docker.

docker rmi original_image
docker rmi test_tag

Pull the original image.

dogestry pull s3://bucket-name original_image

This will work.

Pull the tag.

dogestry pull s3://bucket-name test_tag

This will not work.

Dogestry needs to display "spinner" when uploading Tar file.

Since this process can take minutes, we need to give feedback to user.

On line: https://github.com/dogestry/dogestry/blob/master/cli/cli.go#L311, we can have a MultiWriter where one of them scans the bytes and print "... still going...".

Example config file,

I don't seem to see the config file mentioned in the readme.md file anywhere. Could that reference be changed to a link?

Dogestry has no `list` command

Since the repository format for Dogestry is a bit opaque, it would be great to have a list command so I can see which images and tags I have.

Server mode: stop pulling image when client disconnects

It would be nice if server would cancel the current, on-going pull if dogestry client disconnects mid-way through the pull.

2.0.2 still returns 2.0.1 when given -v

Looks like the version string wasn't incremented.

https://github.com/dogestry/dogestry/blob/v2.0.2/cli/version.go#L7

Optionally download files via HTTP

Opening this to start a discussion. Feel free to close if you feel it's not appropriate for the main project.

I have a client that runs their own hardware (i.e. external to AWS), so bandwidth to AWS is a consideration. In the production zone they have over a dozen VMs in a cluster.

When releases are done, all VMs pull a number of images and restart them. This causes considerable bandwidth contention.

I realize dogestry has the -pullhosts flag, but if I understand it correctly, hosts must expose the docker daemon on a TCP port, which is not something we want to do for security reasons.

The approach we settled on was to run a nginx proxy with the nginx_aws_auth module compiled in, configured as a caching reverse proxy to our private S3 bucket.

The only change to dogestry was to S3Remote.getFile() so that it optionally downloads files from the HTTP endpoint, rather than from S3 directly.

I made this patch on my fork here:

coopernurse@db5603f

We run nginx as a docker container using this image:

https://registry.hub.docker.com/u/coopernurse/nginx-s3-proxy/

If we think this is a generally useful addition I'd be happy to clean up this implementation a bit and try to land it in the main repo. It may simply be that I misunderstand the security implications of -pullhosts, which appears to be the supported way to solve the concurrent pull problem.

Thanks!

Pushing an image will erase an identically-named image already in the bucket

This seems like wanted behavior, except when rolling out across a company using a handful of shared buckets. Team 1 could push ImageA and Team 2 could also push an ImageA in ignorance of the former, therefore erasing the other team's work.

I propose Dogestry checks for identical images and offers a prompt "This push will overwrite image already on server. Okay to proceed y/n" with the addition of a -y force flag for scripted jobs.

Pushing a new tag fails

After creating a new docker tag, dogestry push s3://$BUCKET image:newly-created-tag fails.

Error

2015/06/05 10:30:52 API error (500): mkdir /mnt/sda1/tmp/docker-export-143439678/image:newly-created-tag: no such file or directory

Confirming it with @toffer

s3Client never times out

If there's a loss of network connectivity while dogestry is transferring files to/from S3, then dogestry will just hang.

EDIT:

(My proposed solution below is wrong. I need to recognize when the transfer has stopped, not limit the amount of time that the transfer must complete in.)

~~The solution is to set the ConnectTimeout and ReadTimeout when creating the s3Client. Right now, both timeouts use the default value of 0, which is in effect, no timeout value at all.~~

dogestry push s3://docker-s3-repo/images/?region=eu-west-1 quay.io/johnae/test

but this doesn't push the actual tags:

dogestry push s3://docker-s3-repo/images/?region=eu-west-1 quay.io/johnae/test:latest

it does, however, push tags if you simply execute the above command again.

Remove the progress reader

When pulling and pushing multiple files concurrently, the progress reader output isn't very readable, so I think we would be better off removing it altogether.

Can't find dogestry.eg.cfg

Readme references file dogestry.eg.cfg that isn't in the repo.