seatgeek / docker-mirror Goto Github PK
View Code? Open in Web Editor NEWMirror docker images across image repositories
License: BSD 3-Clause "New" or "Revised" License
Mirror docker images across image repositories
License: BSD 3-Clause "New" or "Revised" License
Hi!
Your tool seems to be the solution for us to mirror distinct Images from DockerHub, but what I don't understand is if your tool can be used with a corporate private registry instead of ECR? Is ECR mandatory?
Regards,
Holger
Using dind and docker-mirror within container i can push manually to my private registry using the helper. (using .docker/conf.json)
Not seem to work when calling the app because it rely to .dockercfg
I was attempting to mirror quay.io/coreos/kube-state-metrics but it failed with:
time="2021-07-09T04:11:44Z" level=warning msg="Get https://registry.hub.docker.com/v2/repositories/quay.io/coreos/kube-state-metrics/tags/?page_size=2048 [registry.hub.docker.com] failed with 404,
retrying" full_repo=quay.io/coreos/kube-state-metrics
It appears that line 342 of mirror.go just hardcodes registry.hub.docker.com in the url to mirror from.
Would it be possible to check if the start of the location was a domain and mirror from there instead of docker.com?
Wasn't sure if I could upload to a public registry since the docs only mention private registries, but got it working like so
Login:
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
config.yaml
target:
registry: public.ecr.aws
# Default or custom alias
prefix: "g6hzp2z3/"
repositories:
- name: alpine # repository needs to be manually created
match_tag:
- "3.10"
Dependabot couldn't find a Gopkg.toml for this project.
Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml
.
If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.
Our developers recently started to receive Apple Sillicon (M1) laptops.
We would like to be able to mirror both amd64 and arm version of public images.
I can take a stab at a PR, if you can point out where in the code we would add such a functionality.
docker-mirror does not support ghcr.io , please add it.
Also it would be nice to add new container registry via additional option
The Docker Hub and Docker Registry APIs appear to be different. Different JSON is returned when querying the two most similar URLs:
https://registry.hub.docker.com/v2/repositories/jippi/hashi-ui/tags/?page_size=2048
http://my.docker.registry.com/v2/namespace/repo/tags/list?n=2048
It looks like mirror.go is expecting 'count' etc, that is not present in the Registry JSON. I'm sure there are other differences but I did not pursue much further once I realized the APIs were different.
Scrubbed example of Registry JSON:
{"name":"namespace/reponame","tags":["0.1.0-10","0.1.0-11","0.1.0-13","0.1.0-14","0.1.0-15","0.1.0-16","0.1.0-17","0.1.0-18","0.1.0-19","0.1.0-2","0.1.0-20","0.1.0-21","0.1.0-22","0.1.0-23","0.1.0-24","0.1.0-25","0.1.0-26","0.1.0-27","0.1.0-28","0.1.0-29","0.1.0-3","0.1.0-4","0.1.0-6","0.1.0-7","0.1.0-8","0.1.0-9","latest-ci","latest-qa","latest"]}
Add a flag to clean up (docker system prune
) after the run
I'm using docker-mirror to manage a very large number of docker images. If I run this on a new machine, each image needs to be pulled and re-tagged regardless of whether or not that image tag already exists in the target mirror repo. It would be great if there was a config option to change this behavior and have docker-mirror check the target repo first to validate whether the proposed tag exists first. This way it would only pull the image from the source repo when the tag doesn't exist in the target. This saves time and disk space, and avoids the issue of dockerhub pull rate-limiting.
I'm not sure whether SHA validation would be necessary or not, but it could be achieved if the arch was also added to the config, an env var, or otherwise read from the system.
Let me know if you want to split them into two issues.
I can make the PR.
After building with go build
using Go 1.17 on macOS, docker-mirror panics:
$ go version
go version go1.17.3 darwin/amd64
$ go build
$ ./docker-mirror --help
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff20412c9e]
runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20412c9e
stack: frame={sp:0x7ffeefbff368, fp:0x7ffeefbff3b8} stack=[0x7ffeefb80408,0x7ffeefbff470)
0x00007ffeefbff268: 0x01007ffeefbff288 0x0000000000000004
0x00007ffeefbff278: 0x000000000000001f 0x00007fff20412c9e
0x00007ffeefbff288: 0x0b01dfacedebac1e 0x0000000000000001
0x00007ffeefbff298: 0x0000000004035fd1 <runtime.throw+0x0000000000000071> 0x00007ffeefbff338
0x00007ffeefbff2a8: 0x0000000004423a1c 0x00007ffeefbff2f0
0x00007ffeefbff2b8: 0x0000000004036288 <runtime.fatalthrow.func1+0x0000000000000048> 0x0000000004719840
0x00007ffeefbff2c8: 0x0000000000000001 0x0000000000000001
0x00007ffeefbff2d8: 0x00007ffeefbff338 0x0000000004035fd1 <runtime.throw+0x0000000000000071>
0x00007ffeefbff2e8: 0x0000000004719840 0x00007ffeefbff328
0x00007ffeefbff2f8: 0x0000000004036210 <runtime.fatalthrow+0x0000000000000050> 0x00007ffeefbff308
0x00007ffeefbff308: 0x0000000004036240 <runtime.fatalthrow.func1+0x0000000000000000> 0x0000000004719840
0x00007ffeefbff318: 0x0000000004035fd1 <runtime.throw+0x0000000000000071> 0x00007ffeefbff338
0x00007ffeefbff328: 0x00007ffeefbff358 0x0000000004035fd1 <runtime.throw+0x0000000000000071>
0x00007ffeefbff338: 0x00007ffeefbff340 0x0000000004036000 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff348: 0x000000000442aa54 0x000000000000002a
0x00007ffeefbff358: 0x00007ffeefbff3a8 0x000000000404b776 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff368: <0x000000000442aa54 0x0000000004719840
0x00007ffeefbff378: 0x00007ffeefbff3e8 0x00000000040292a6 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff388: 0x000000c00020c000 0x0000000000002000
0x00007ffeefbff398: 0x0000000000000008 0x0000000004719840
0x00007ffeefbff3a8: 0x00007ffeefbff3f0 !0x00007fff20412c9e
0x00007ffeefbff3b8: >0x00007ffeefbff3f0 0x00000000046bc000
0x00007ffeefbff3c8: 0x0000000000000648 0x000000000427fa05 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005>
I can include more of the traceback if you want but it doesn't seem super useful.
Seems like updating the x/sys package will address this issue. From golang/go#46763
Dependabot couldn't find a Gopkg.toml for this project.
Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml
.
If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.
Iโd like to have the ability to use docker-mirror to support mirroring of non-Docker repo images from Quay container registry and Google Container Registry (GCR) to be published in public ECR repos.
An example YAML file would look as follows:
target:
registry: ACCOUNT_ID.dkr.REGION.amazonaws.com
prefix: "hub/"
repositories:
- name: quay.io/opentelemetry/opentelemetry-operator
- "v*"
- name: gcr.io/kubebuilder/kube-rbac-proxy
- max_tags: 10
We are working on an enhancement which we plan to contribute to the project.
cc: @Saber-W
There are a couple of related issues open which will hopefully also get addressed by our upcoming PR.
I'm considering using this tool in our CI system.
But it not have specific versions tags in dockerhub is would hard to pin to a specific version and it might break easily down the road for us.
Also it a small security concern since a tool that would have access to AWS credentials, it would prefer having more visibility into which version we are using. i.e. just using latest we can't verify the checksums.
Hey Folks,
I'm currently unable to pick up a specified Redis image of 5.0.3, I've tried with 5.0.3-stretch, etc. However it doesn't seem to load the tag, however, when I had switched it to a wildcard on the version, and set it to the past 14 days. It seems to scrape all the relevant tags and upload them just fine to the destination.
Could ya'll take a look?
---
target:
registry: $account_id.dkr.ecr.us-west-2.amazonaws.com
prefix: "hub/"
repositories:
- name: redis
match_tag:
- "5.0.3"
$ docker-mirror
INFO[0000] Creating Docker client
INFO[0000] Connected to Docker daemon: docker-desktop @ 20.10.0
INFO[0000] Creating AWS client
INFO[0000] Loading list of ECR repositories
INFO[0000] Done loading ECR repositories
INFO[0000] Repository mirror completed full_repo=redis num_tags=0 repo=redis
INFO[0000] Done
---
target:
registry: $account_id.dkr.ecr.us-west-2.amazonaws.com
prefix: "hub/"
repositories:
- name: redis
match_tag:
- "5*"
max_tag_age: 14d
INFO[0000] Creating Docker client
INFO[0000] Connected to Docker daemon: docker-desktop @ 20.10.0
INFO[0000] Creating AWS client
INFO[0000] Loading list of ECR repositories
INFO[0000] Done loading ECR repositories
INFO[0000] Start mirror tag full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0000] Starting docker pull full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Completed docker pull in 980.730612ms full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Starting docker tag full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Completed docker tag in 7.22202ms full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
INFO[0001] Starting docker push full_repo=redis num_tags=18 repo=redis tag=5.0.10-alpine3.12
OS:
macOS Catalina 10.15.6
golang version:
go1.15.5 darwin/amd64
Authentication Method:
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin $account_id.dkr.ecr.us-west-2.amazonaws.com
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.