docker-archive / migrator Goto Github PK

Add support for a v1 registry without auth

When ERROR_ACTION=abort is set and the v2 availability check is failing, it will get stuck in an infinite loop for as long as the check fails. This should error out on first failure when ERROR_ACTION=abort is set.

The environment variables used, V1_REGISTRY and V2_REGISTRY should be changed to something that is more accurate such as SOURCE_REGISTRY and DESTINATION_REGISTRY as the source may not necessarily be a v1.

At the moment, there is no verification performed to ensure that each tag is present in the v2 registry after migration; it is assumed based off of the exit codes from each push. While this is relatively safe, it would be better to verify.

Hello,
I'm trying to migrate from a 0.9.0 registry running in a docker, bound on port 5000, to a registry v2 running in a docker on port 5001.

I tried:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY="localhost:5000" \
    -e V2_REGISTRY="localhost:5001" \
    -e USE_HTTP=true \
    docker/migrator
[!!] Please login to localhost:5000:
Username: 
WARNING: login credentials saved in /root/.dockercfg.
Account created. Please see the documentation of the registry http://localhost:5000/v1/ for instructions how to activate it.

[INFO] Getting a list of images from localhost:5000

[ERROR] curl => API failure
[ERROR] Migration from v1 to v2 failed!

I also tried using NO_LOGIN=true with the same result.

What can I do ?

Hi again,

In fact it's not working. It fails at the push step.
I used:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY="my.old.registry" \
    -e V2_REGISTRY="10.x.x.x:5001" \
    -e USE_HTTP=true \
    docker/migrator

It fails with:

[ERROR] v2 registry (10.x.x.x:5001) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

From what you explained to me, I think I understand what's going on, but I do not know how to work around it using migrator:
I cannot use localhost:5001 as my V2_REGISTRY since it is executed in the docker container in which localhost is a different computer. That's why I used either the IP address or different DNS names. But it is only localhost that is trusted by default by the docker/registry:2

I guess I could finish the pull manually by executing a push loop on the host using localhost, but is there any way to make it work via the migrator ?

Thanks

On a machine with Debian 9.5 and docker-ce 18.06.1~ce~3-0~debian just installed, I am not able to login the V1 registry. It seems the migrator is accessing the old registry via the v2 protocol...

So, it's trying to access /v2/.

I'm using the FQDN of the v1 registry, without https://.

The v1 registry is running registry:0.9.1. I can see the request made by the migrator:

registry_1  | 172.17.0.8 - - [17/Sep/2018:09:16:54 +0000] "GET /v2/ HTTP/1.1" 404 233 "-" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.9.0-8-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.6.2)"

The same happens when I try to do docker login (with a up-to-date client), but I guess that's normal as new clients need a v2 registry...

What am I doing wrong?

Add a test that uses NO_LOGIN migration.

We have found there to be a few Docker images in our registry with encoded characters in their names, like '/' encoded as "%2F".
This is causing issues when trying to do the API curls resulting in 404 errors.
It would be good if there was an option to skip these images without failing the migration.

In migrator.sh line 811

https://github.com/docker/migrator/blob/5e7ef10ace96e34d33e8870173f230e738a292cf/migrator.sh#L811

The "if" expects "Docker-Distribution-Api-Version: registry/2.0", but it fails with my current registry 2.6.2, because the returned header is the following:

...
docker-distribution-api-version: registry/2.0
...

Am I using some wrong version of the migrator/registry or is the check maybe outdated?

Thanks!

Lucas

If your migration takes more than 12 hours, the ECR token expires and subsequent pushes will fail.

i have a docker registry:2 private registry which is running for quite sometime in my testing environment with auth ( httpasswd ) and i want to change it's auth method to token based. can i use the migrator to migrate its database ( images, tags, layers everything ? ) or is there another way to do it.

I tried to migrate the registry V1 to V2 our own registry. I'm facing an issue like

[INFO] Getting a list of images from cmtools

[ERROR] curl => API failure
[ERROR] Migration from v1 to v2 failed!

My command is :-

sudo docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=admin -e V2_REGISTRY=test -e NO_LOGIN=true -e V1_NO_LOGIN=true -e V2_NO_LOGIN=true -e USE_INSECURE_URL=true -e V1_USE_HTTP=true -e V2_USE_HTTP=true docker/migrator

Kindly help me to do the migration.

Thanks,

Kaleesastha

In DTR 1.4, repositories have to be created prior to pushing images. There are a few challenges that need to be addressed:

• Determine whether or not the v2 registry is DTR or not (need to find endpoint to test against)
• Adding the repos via the DTR API; adding an organization if the user namespace does not exist

I am attempting to migrate images from a 0.8.1 registry running on a rhel 6.5 server with docker engine 1.4 to a v2 registry running on a rhel 6.7 server with docker engine 1.7.1. I am able to pull the list of images but when the script tries to pull the images I get an error response from the daemon : invalid registry endpoint <v1_registry>/v0/:unable to ping registry endpoint.
I get the same error when trying to pull from an individual command. I noticed the registry adds https in front of the registry name even though I have added the --insecure-registry flag on both machines.
Here is my command:

  sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=<v1_registry> -e V2_REGISTRY=<v2_registry> -e ERROR_ACTION=prompt -e NO_LOGIN=true -e USE_HTTP=true docker/migrator

Any help will be greatly appreciated.

Add more detailed information about the status during of the migration to indicate how far along the migration has progressed.

"impossible right now since we can't query the v2 registry" - this makes no sense.

The newest docker version creates a cfg file under .docker/config.json, while the script searches the user information under .docker.cfg file.

I have to do migration of a V1 registry to a v2 registry by using the new client version. This script seems to use the old client.

Hello,
I used the migrator to migrate from registry V1 to DTR 1.4.3, but now we are planning on upgrading to DTR 2.0, and there is no path to migrate the images. is it possible to support V2 to V2 migration?

Thanks

Hi,

I'm migrating a v1 registry without SSL to a v2 registry with SSL. Both registries don't need authentication. Here is the command I ran:

docker run -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e V1_REGISTRY=10.76.2.13:5002 \
    -e V2_REGISTRY=10.76.2.13:5001 \
    -e NO_LOGIN=true -e V1_USE_HTTP=true docker/migrator

and it fails with

FATA[0004] Error response from daemon: Get https://10.76.2.13:5002/v1/_ping: EOF

Add support to migrate from Docker Hub organization/user namespace to a v2 registry

Add checks to make sure the V1_REGISTRY_URL and V2_REGISTRY_URL environment variables are set before running script.

Hi,

We use an nginx container for authentication to our docker registry. As of now there isn't anyway to authenticate with this service before pulling images. It'd be nice if there was similar support for http_auth as there is for docker.io.

Maybe http_auth_user and http_auth_pass as environment variables?

Thanks

When moving thousands of images, you don't always want to retry from start. Is it possible to add this feature?

Add cleanup of images pulled to local docker engine

The current implementation only makes a single query to the v2 repository to get the list of tags. Some implementations (like Amazon's ECR) hard-limits the number of tags it reports to 100, so at most 100 tags will be skipped.

We need to check if the query returns a Link; rel="next" header and loop trough all pages until we have a complete list.

Expect a PR fixing this in a few hours.

We have a few thousands of images to migrate and 60% of them fail with an error like:

FATA[0051] could not verify layer data for: sha256:5362dab0aca263b53324da67a2769502e7e2f99417ae39594d7a15171e194b75. This may be because internal files in the layer store were modified. Re-pulling or rebuilding this image may resolve the issue 

A similar issue is reported by someone else on Docker forums as well:
https://forums.docker.com/t/could-not-verify-layer-data-error-when-trying-to-push-image-with-content-trust/21790

moby/moby#10906
https://github.com/docker/docker-registry#search-engine-options

If you don't have search enabled the migrator script will return an empty list of images as the list of images you have to migrate over to the new repo.

Does the docker migrator work with public images, not under the users account?

How do I add custom CA's into the tool?

I have an image under my organization that I know exists. When I type the name in REPO_FILTER or part of the name, it returns zero results for migration. I'm able to find older images under my org so it is weird that some images are not found.

When both registries use client certificates the migrator doesn't work, it fails with curl => API failure because curl doesn't use Dockers client certificates.

We used migrator script to migrate from v1 to a v2 registry which was setup without any authentication/authorization. This worked great, thanks!

Now we have a v2 registry setup with both token-based authentication along with authorization, i.e. using teams and authorized namespaces (public/private) via the Portus project:
https://github.com/SUSE/Portus/

I'm thinking that even if the migrator script can authenticate, I can't see how it would handle authorizations. Any thoughts?

I'm running docker 1.9.1 and migrating from an old registry to v2.2.1 registry. The migrator pulls in all the images from the old repository, but fails to recognize the v2 repository is present

[ERROR] v2 registry (docker2.example.com:5000) is not available
[!!] Verify v2 registry is functioning as expected; press any key to continue to retry [ctrl+c to abort]

The registry is running behind an nginx front end which provides the SSL termination. I can access it in a browser, and if I provide the username and password, I can request https://docker2.example.com:5000/v2/ and get a {} response, with a Docker-Distribution-Api-Version: registry/2.0 header.

If I look at what the migrate script is doing, in my nginx logs, I can see a HEAD request for /v2/ failing with a 401, indicating no credentials supplied - this request is made here https://github.com/docker/migrator/blob/master/migrator.sh#L456, and tracing back the V2_OPTIONS doesn't appear to use the V2_USERNAME or V2_PASSWORD

Is it possible to migrate from V1 registry (V0.9.1) to the new V2 registry without pushing and pulling all images? I know that this is not the normal migration path, but for what reason do the image need to get pulled? We have a lot of images in our private registry and it would be great not to generate too much traffic if there is no good reason. Would it be possible to convert existing meta-data only ?

Migrating from any workspace other than "library" name space works fine. For some reason it drops the library namespace:

[INFO] Retagging all images from 'dockerdev1.mitre.org' to 'dockeradm1.mitre.org'
[OK] dockerdev1.mitre.org/alpine:2.0 > dockeradm1.mitre.org/alpine:2.0
[OK] dockerdev1.mitre.org/alpine:2015-02-27 > dockeradm1.mitre.org/alpine:2015-02-27
[OK] Successfully retagged all images

my alpine image is stored as dockerdev1.mitre.org/library/alpine

Hello,

we have a problem with the migration process. When the migraton script pulls the images from the old registry a directory is added. The tags of the images for the new registry contain this added directory. This look something like this: /library/imagename:tag. since its complicated to pull those not correct named images from the new registry it would be nice to change the tag so that they are tagged as image name:tag as usual. Does anyone knows about this "bug" or could tell me where it comes from?

Regards, DrDth

The environment variables used don't make complete sense due to the fact that they are not URLs that you provide. Instead of V1_REGISTRY_URL and V2_REGISTRY_URL, V1_REGISTRY and V2_REGISTRY would make much more sense.

The tool complains that [ERROR] v2 registry (http://r10a-docker7c.mitre.org) is not available

Error pulling image (59882a691179728f576b972a8af29f9ec6936196) from $myPrivateV1Repo/image, ApplyLayer exit status 1 stdout: stderr: open
/usr/include/openssl/stack.h: read-only file system

This error pops when the machine executing the script reaches 100G of used space by this process.

I tried running the migrator.sh script instead of the docker thinking that maybe I will be able to see something more but to no avail.

I'm working on preparing the open source repositories for the new centralized maintainers file, but noticed this repository does not yet have a MAINTAINERS file.

I can create a PR to fix this, but I'm not sure who should be included as maintainer(s).

@mbentley are you the only maintainer, or should I add other people here?

for reference, see docker/opensource#35 and moby/moby#18321

On failed pushes or pulls, migrator exits immediately. It might be better to log the error and continue, outputting the failed step(s) for further troubleshooting.

Instead of downloading the entire v1 registry with all images and tags, can we download one at a time and push to the v2 registry?

The new process would look like :

1. Query v1 registry for all repositories and their tags
2. Pull one image (with a particular tag) to local docker engine
3. Retag it and push to v2
4. Delete the image from local docker engine
5. Repeat this for all images in v1

This addresses the major issue of Disk space on the docker engine host not being sufficient to hold entire v1 registry.

We use S3 for storing our images and all the images cannot fit on a single host's disk.

When doing a GET on https://hub.docker.com/v2/repositories/namespaces
it returns only namespaces in which members are owners, rather than read access. Thus causing this code block to raise an exception incorrectly

https://github.com/docker/migrator/blob/master/migrator.sh#L430-L436

Hi,
I'm trying to migrate my images from a v1 registry to a v2 registry. To do so, I use a reverse proxy to add SSL but I'm not using authentication.
I've got no problem to pull images from registry v1 but when I push I've got this error :

[OK] Verified v2 registry (registry.si3si.com) is available
[!!] Please login to registry.si3si.com:
Username:
FATA[0001] Error response from daemon: no successful auth challenge for https://registry.si3si.com/v2/ - errors: []

I also tried to add NO_LOGIN=true env variable but in this case I've got another error :

cat: /root/.dockercfg: No such file or directory

[ERROR] Non-zero exit code
[ERROR] Migration from v1 to v2 failed!

I've got no other idea of what I can tried

If you need any other information, do not hesitate

Gilles DEHAUDT