daocloud / public-image-mirror

很多镜像都在国外。比如 gcr 。国内下载很慢，需要加速。致力于提供连接全世界的稳定可靠安全的容器镜像服务。

public-image-mirror

Sync: 定期同步列表里的 image tag 的同步率, 只要 tag 存在就是同步的.

由于近期使用量剧增, 为保证大多数的可用性

• 白名单 & 限流 & 降级 的公开信息 #2328
• 如有疑问请咨询 #4183

背景 & 目标

很多镜像都在国外。比如 gcr 。国内下载很慢，需要加速。

• 一个简洁有效的方法能够加速这些包。简洁的名称映射
• 易于添加，添加新的包，不需要去修改代码。
• 稳定可靠，更新实时。每天检查同步情况。
• 此项目仅是源镜像仓库 (Registry) 的 Mirror
  • 所有 hash(sha256) 均和源保持一致 (懒加载机制)。
  • 由于缓存的存在, 可能存在 1 小时的延迟。
  • 如超过 1 小时还未更新, 估计是国际带宽挂了。
• 对于 镜像层(blob) 会缓存在第三方对象存储上
  • 当前暂未对内容做任何检测, 计划会添加检测。

快速开始

docker run -d -P m.daocloud.io/docker.io/library/nginx

使用方法

增加前缀 (推荐方式)。比如：

              docker.io/library/busybox
                 |
                 V
m.daocloud.io/docker.io/library/busybox

或者 支持的镜像仓库 的 前缀替换 就可以使用。比如：

           docker.io/library/busybox
             |
             V
docker.m.daocloud.io/library/busybox

单次单镜像同步

!!!!!!!!!!!!!!!!!!!!!!!!强烈推荐!!!!!!!!!!!!!!!!!!!!!!!!

您可以根据 镜像同步 Issue 模板 创建一个 Issue, 将会有机器人帮您优先主动同步指定的镜像

原先已经在下的镜像还是会继续走原来的, 需要重启 docker 再重新拉取才能走已经同步好的缓存过, 所以推荐先单次同步再尝试拉取 对于 latest 这种经常内容会发生变更的 tag 变更后会需要重新同步

懒加载

仅推荐用于小于 20MiB 的镜像 这条通道主要用于保持 image 是最新的, 每次拉取都会检查是否和上游保持同步

如果您看到下没有进度这是由于带宽有限只要有几人在下较大的文件就会阻塞后续的下载, 可以尝试单次单镜像同步

就算没同步也能 直接拉取, 初次拉取会比已经同步过的慢.

所有懒加载 带宽

• 国际带宽 6 * 30 Mbps

定期同步列表 (不推荐)

强烈建议使用单次单镜像同步

mirror.txt

如果想要新增, 提 PR 修改即可。例如 PR#1， 并请在 PR 提交前排序： ./hack/fmt.sh mirror.txt

支持前缀替换的 Registry (不推荐)

推荐使用添加前缀的方式.

前缀替换的 Registry 的规则, 这是人工配置的, 有需求提 Issue.

源站	替换为	备注
cr.l5d.io	l5d.m.daocloud.io	将废弃请使用添加前缀的方式
docker.elastic.co	elastic.m.daocloud.io
docker.io	docker.m.daocloud.io
gcr.io	gcr.m.daocloud.io
ghcr.io	ghcr.m.daocloud.io
k8s.gcr.io	k8s-gcr.m.daocloud.io	k8s.gcr.io 以被迁移到 registry.k8s.io
registry.k8s.io	k8s.m.daocloud.io
mcr.microsoft.com	mcr.m.daocloud.io
nvcr.io	nvcr.m.daocloud.io
quay.io	quay.m.daocloud.io
registry.jujucharms.com	jujucharms.m.daocloud.io	将废弃请使用添加前缀的方式
rocks.canonical.com	rocks-canonical.m.daocloud.io	将废弃请使用添加前缀的方式

最佳实践

• 通过 加速 安装 kubeadm

kubeadm config images pull --image-repository k8s-gcr.m.daocloud.io

• 通过 加速 安装 kind

kind create cluster --name kind --image m.daocloud.io/docker.io/kindest/node:v1.22.1

• 通过 加速 部署 应用(这里以 Ingress 为例)

wget -O image-filter.sh https://github.com/DaoCloud/public-image-mirror/raw/main/hack/image-filter.sh && chmod +x image-filter.sh

wget -O deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/baremetal/deploy.yaml

cat ./deploy.yaml | ./image-filter.sh | kubectl apply -f -

• Docker 加速

添加到 /etc/docker/daemon.json

{
  "registry-mirrors": [
    "https://docker.m.daocloud.io"
  ]
}

[友情链接]加速三剑客

• 镜像加速：https://github.com/DaoCloud/public-image-mirror
• 二进制文件加速：https://github.com/DaoCloud/public-binary-files-mirror
• Helm 加速：https://github.com/DaoCloud/public-helm-charts-mirror

贡献者

Made with contrib.rocks.

public-image-mirror's Issues

SYNC IMAGE nvcr.io/nvidia/k8s-device-plugin:v0.14.1

SYNC nvcr.io/nvidia/k8s-device-plugin:v0.14.1

多架构不知道支不支持

ARM

Failed to pull

docker pull m.daocloud.io/registry.k8s.io/build-image/kube-cross:v1.26.0-go1.19.4-bullseye.1
v1.26.0-go1.19.4-bullseye.1: Pulling from registry.k8s.io/build-image/kube-cross
4948a51a9a3f: Pull complete 
b4da1b3775b3: Pull complete 
6efc1b20f435: Pull complete 
4c81e9b9b636: Pull complete 
cba7ab2dd679: Pull complete 
3ba0ec045e9d: Downloading [=================================================> ]  114.1MB/115.2MB
0a317f9f8ea9: Download complete 
4f4fb700ef54: Download complete 
51f8d15afdec: Download complete 
74d5db56232b: Download complete 
133265ae10e5: Downloading 
50bed5563018: Downloading 
02ca928c3ed2: Download complete 
be0ef21e3e6d: Download complete 
error parsing HTTP 403 response body: invalid character 'c' looking for beginning of value: "connecting to docker-cn.cn-sh2.ufileos.com:80: connecting to 106.75.235.58:80: dial tcp 106.75.235.58:80: i/o timeout"

感觉 us3 的服务挂了

/bug

智能 inject 镜像地址。

例如 ： https://istio.io/latest/docs/setup/getting-started/#dashboard。
有很多情况是需要 运行 类似 $ kubectl apply -f samples/addons 的命令的。

在这样的情况下，需要手动打开文件，一个个的比对修改，比较麻烦。
如果能够自动化的完成，比如

就很 厉害了

A single image sync fails frequently

/assign
/bug

SYNC IMAGE ghcr.io/huggingface/text-generation-inference:1.0.3

SYNC ghcr.io/huggingface/text-generation-inference:1.0.3

Support breakpoint continuation

Support breakpoint continuation

/enhancement
/assign

给打开 https://m.daocloud.io 页面的用户做一个引导页面?

RT

/assign @yankay
/assign @panpan0000
/assign @pacoxu

SYNC IMAGE docker.io/library/busybox:latest

SYNC docker.io/library/busybox:latest

请修改上面的内容，将 docker.io/library/busybox:latest 改为你要同步的镜像

多架构不知道支不支持

Support lazy load

unable to pull ghcr.m.daocloud.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.34b0

$ docker pull ghcr.m.daocloud.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.34b0
Error response from daemon: manifest for ghcr.m.daocloud.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.34b0 not found: manifest unknown: manifest unknown

$ docker pull ghcr.m.daocloud.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.3.1-beta.1
Error response from daemon: manifest for ghcr.m.daocloud.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.3.1-beta.1 not found: manifest unknown: manifest unknown

failed to sync veth-plugin

➜   docker pull ghcr.io/spidernet-io/cni-plugins/veth:v0.1.1-amd64
v0.1.1-amd64: Pulling from spidernet-io/cni-plugins/veth
213ec9aee27d: Already exists
ce898f80885e: Pull complete
470881f24a7d: Pull complete
Digest: sha256:3aab10cf52cb49c656ee081b40e8379f15a790416277a259e7d9e2659b2e6fa1
Status: Downloaded newer image for ghcr.io/spidernet-io/cni-plugins/veth:v0.1.1-amd64
ghcr.io/spidernet-io/cni-plugins/veth:v0.1.1-amd64

➜   docker pull ghcr.m.daocloud.io/spidernet-io/cni-plugins/veth:v0.1.1-amd64
Error response from daemon: manifest for ghcr.m.daocloud.io/spidernet-io/cni-plugins/veth:v0.1.1-amd64 not found: manifest unknown: manifest unknown

有外链的镜像同步失败

https://github.com/DaoCloud/public-image-mirror/actions/runs/1804207223

初步猜测是 manifest platform 中的 windows 信息 skopeo 没有正确的处理

storage.k8s.io 和docker.io/openebs下面的全部插件和驱动 找不着

storage.k8s.io 和docker.io/openebs下面的全部插件和驱动 我grep了一下好像没有收录齐全 是否有必要收录一下？
我找到了Openebs的部分插件 新的zfs-csi-driver好像没进来导致插件安装缓慢

Add istio image

pull image very slow

I pull the image of [docker.m.daocloud.io/redis:7.2-rc2-bookworm],the pull rate of the part layer[layer-sha256:edc694ca19a6c032cd855e0b47af01def1a682deb1fb14bf1e417437ab7c69c3] is very slow.
And the other parts are both very fast.
I don't know why the slow rate.

I don't think it is because local network.

elastic.m.daocloud.io 貌似不生效

in k8s:

Failed to pull image "elastic.m.daocloud.io/elasticsearch/elasticsearch:7.16.2": rpc error: code = Unknown desc = Error response from daemon: manifest for elastic.m.daocloud.io/elasticsearch/elasticsearch:7.16.2 not found: manifest unknown: manifest unknown

for docker pull:

docker pull elastic.m.daocloud.io/docker.elastic.co/eck/eck-operator:1.9.1
Error response from daemon: manifest for elastic.m.daocloud.io/docker.elastic.co/eck/eck-operator:1.9.1 not found: manifest unknown: manifest unknown

SYNC IMAGE docker.io/grafana/grafana:9.1.7

SYNC docker.io/grafana/grafana:9.1.7

manifests 为空的镜像同步失败

https://github.com/DaoCloud/public-image-mirror/runs/5135203284

/bug

elastic.m.daocloud.io/elasticsearch/elasticsearch:7.16.2 仍然没有同步过来

貌似已经过了 48 小时，需要 check 一下，谢谢～

docker pull elastic.m.daocloud.io/elasticsearch/elasticsearch:7.16.2
Error response from daemon: manifest for elastic.m.daocloud.io/elasticsearch/elasticsearch:7.16.2 not found: manifest unknown: manifest unknown

SYNC IMAGE quay.io/kiwigrid/k8s-sidecar:1.19.2

SYNC quay.io/kiwigrid/k8s-sidecar:1.19.2

是否能增加mcr镜像加速地址

ARG VARIANT=11-bullseye
FROM mcr.microsoft.com/vscode/devcontainers/java:0-${VARIANT}

SYNC IMAGE docker.io/alpine:latest

SYNC docker.io/alpine:latest

Can not sync nginx:1.23.1-alpine

I think the code : https://github.com/DaoCloud/public-image-mirror/blob/main/hack/diff-image.sh#L179, the 1.23.1-alpine is filtered by the tag 1.23.1.

同步情况日报(不要删)

此 ISSUE 用于机器人汇报同步情况

同步架构：

SRC => MIDDLE（jp） => DEST（cn）。

一天 = 86400s

add registry.k8s.io

Accroding to the kubernetes/kubeadm#2670.

k8s would use registry.k8s.io.

SYNC IMAGE grafana/grafana:9.1.7

SYNC grafana/grafana:9.1.7

SYNC IMAGE apachepulsar/pulsar-manager:v0.3.0

SYNC apachepulsar/pulsar-manager:v0.3.0

SYNC IMAGE

SYNC docker.io/library/busybox:latest

Not all tags being sync ? if so, should we document it ?

for example. minio/minio:RELEASE.2022-07-29T19-40-48Z is not sync.
If it's expected behavior, I guess some regular expression filter it out ? (maybe only sync those tags with vx.y.z ?)
if so ,please write this NOTES down in documents to avoid confusion.

# docker pull docker.m.daocloud.io/minio/minio:RELEASE.2022-07-29T19-40-48Z
Error response from daemon: manifest for docker.m.daocloud.io/minio/minio:RELEASE.2022-07-29T19-40-48Z not found: manifest unknown: manifest unknown

感谢这种项目的贡献

感谢这种项目的贡献，解决了大问题，应该大力推广

image sync error

gcr.m.daocloud.io/tekton-releases/github.com/tektoncd/dashboard/cmd/dashboard:v0.29.2@sha256:67425f5fa32c591578ba6c8b6d4d6a135f8ca195745e023f3f311e3f1ee6ea37 is not work

manifest unknown: manifest unknown

docker pull quay.m.daocloud.io/iovisor/bcc:latest
Error response from daemon: manifest for quay.m.daocloud.io/iovisor/bcc:latest not found: manifest unknown: manifest unknown

移除一些冷门 image 的镜像同步

深度同步 和 检查深度同步 的 CI 执行速度提升

找个方式提升镜像深度对比的速度

造成的影响

• 一些固定名字的镜像(如 latest , v1等) 可能不会更新到

已经采取的优化措施

• 跳过明确不会更新的镜像(如 v1.0.0 等明确版本号的) 的深度对比

/bug
/assign

SYNC IMAGE docker.io/alpine:latest

SYNC docker.io/alpine:latest

SYNC IMAGE

SYNC docker.io/library/busybox:latest

Add kubeflow:v1.7 related images

gcr.io/arrikto/kubeflow/oidc-authservice

gcr.io/ml-pipeline/api-server
gcr.io/ml-pipeline/cache-server
gcr.io/ml-pipeline/frontend
gcr.io/ml-pipeline/metadata-writer
gcr.io/ml-pipeline/minio
gcr.io/ml-pipeline/mysql
gcr.io/ml-pipeline/persistenceagent
gcr.io/ml-pipeline/scheduledworkflow
gcr.io/ml-pipeline/viewer-crd-controller
gcr.io/ml-pipeline/visualization-server
gcr.io/ml-pipeline/workflow-controller
gcr.io/tfx-oss-public/ml_metadata_store_server
docker.io/kubeflowkatib/enas-cnn-cifar10-cpu
docker.io/kubeflowkatib/katib-controller
docker.io/kubeflowkatib/katib-db-manager
docker.io/kubeflowkatib/katib-ui
docker.io/kubeflowkatib/mxnet-mnist
docker.io/kubeflowkatib/pytorch-mnist-cpu
docker.io/kubeflownotebookswg/centraldashboard
docker.io/kubeflownotebookswg/jupyter-web-app
docker.io/kubeflownotebookswg/kfam
docker.io/kubeflownotebookswg/notebook-controller
docker.io/kubeflownotebookswg/poddefaults-webhook
docker.io/kubeflownotebookswg/profile-controller
docker.io/kubeflownotebookswg/tensorboard-controller
docker.io/kubeflownotebookswg/tensorboards-web-app
docker.io/kubeflownotebookswg/volumes-web-app
docker.io/metacontrollerio/metacontroller
docker.io/seldonio/mlserver

有一些镜像有大量 中间版本，实在太大，动辄 几千个 tag，全部同步非常耗时间

比如 calico

https://quay.io/repository/calico/cni?tab=tags

https://quay.io/repository/jetstack/cert-manager-webhook

同步情况日报(不要删) 2

此 ISSUE 用于机器人汇报同步情况

同步架构：

SRC => MIDDLE（jp） => DEST（cn）。

一天 = 86400s

The limit of the Github Action has been reached

The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings.

SYNC IMAGE docker.io/library/busybox:latest

SYNC docker.io/library/busybox:latest

请修改上面的内容，将 docker.io/library/busybox:latest 改为你要同步的镜像
注意, 标题不要改哦

或者可以直接使用命令

gh -R DaoCloud/public-image-mirror issue create \
  --title "SYNC IMAGE" \
  --body "SYNC docker.io/library/busybox:latest"

无法拉取redis:7.0.4-alpine

redis已经在mirror.list里了
但是

docker pull docker.m.daocloud.io/library/redis:7.0.4-alpine
7.0.4-alpine: Pulling from library/redis
no matching manifest for linux/amd64 in the manifest list entries

docker pull docker.m.daocloud.io/redis:7.0.4-alpine
7.0.4-alpine: Pulling from redis
no matching manifest for linux/amd64 in the manifest list entries

直接拉可以

docker pull redis:7.0.4-alpine
7.0.4-alpine: Pulling from library/redis
Digest: sha256:dc1b954f5a1db78e31b8870966294d2f93fa8a7fba5c1337a1ce4ec55f311bc3
Status: Image is up to date for redis:7.0.4-alpine
docker.io/library/redis:7.0.4-alpine

Deep Sync cannot run within 6 hours

/assign
/enhancement

SYNC gcr.io/kubebuilder/kube-rbac-proxy:v0.14.1

SYNC gcr.io/kubebuilder/kube-rbac-proxy:v0.14.1

请修改上面的内容，将 docker.io/library/busybox:latest 改为你要同步的镜像

增加健康检查脚本

当 镜像拉取失败，报错为 500 Internal Server Error， 通常是因为 ufile 故障导致的。

此时 比较简单的解决方法是 重启对应的容器。（需要验证）
镜像仓库不能自动恢复。

必要时需要重启docker 😓

短域名

m.daocloud.io/ 的前缀感觉有点长啊, 是不是搞一个短域名, 让使用起来更加方便

Image Pull Failure: 503 Error Encountered

I'm encountering a '503 Service Temporarily Unavailable' error when I attempt to pull any image using ctr. Please take a look.

ctr image pull m.daocloud.io/gcr.io/ml-pipeline/mysql:5.6
m.daocloud.io/gcr.io/ml-pipeline/mysql:5.6:                                       resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:a5c683ba65b44dbd7fd09e62c1ab23def0f3957bab0f670ad5c92693ae230a4f: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8a01a78fff5a38ac5f471df1ff3dc4e9f1ece82eae5e9d55555ad6053d7dc5c5:    downloading    |--------------------------------------|    0.0 B/121.0 B
config-sha256:3ed1080b793fc4a10cab741a04ce090caf1ad2932cbcc679b6587624af9f6157:   downloading    |--------------------------------------|    0.0 B/6.8 KiB
layer-sha256:fc7181108d403205fda45b28dbddfa1cf07e772fa41244e44f53a341b8b1893d:    waiting        |--------------------------------------|
layer-sha256:787a24c80112f841ccb00f57af3f3ee555c5a758adc0c70b4488834ca42c12cc:    waiting        |--------------------------------------|
layer-sha256:a08cb039d3cd3acf3a5bf2ed1ff76fc1eb7ea0c6db0689de8c34f5f7a2138bd6:    downloading    |--------------------------------------|    0.0 B/4.3 MiB
layer-sha256:4f7d35eb5394caab0079bcfc00af4bd30e83cc1b47a201a78c203efdff5649c8:    downloading    |--------------------------------------|    0.0 B/1.2 MiB
layer-sha256:5aa21f895d959ad145996cdc9696faef2c27111fa6b46b16078f21566eb127c7:    waiting        |--------------------------------------|
layer-sha256:345649b63bc3b040f1ec6621892ad657005c5c6f707e5d8fcc1d1d51b5b7eb21:    waiting        |--------------------------------------|
layer-sha256:591a87fc59ec1810ebb02cfb88a30efa5bd8ea6e1208aad23086607645124662:    waiting        |--------------------------------------|
layer-sha256:6019ecdb5901fcfab79fc7ea21ed7e1c6a861d8098a35b61314da24a007e98eb:    waiting        |--------------------------------------|
layer-sha256:aadc682b6a759c6d9f572a665a1da3eae0603b509ff06d5f974a1c74616e9638:    waiting        |--------------------------------------|
layer-sha256:e0969183ab5f6f774417a2a4035c694577778233f0af2cf1a1b9699108f35092:    downloading    |--------------------------------------|    0.0 B/2.8 KiB
elapsed: 3.4 s                                                                    total:  2.6 Ki (770.0 B/s)
ctr: failed to copy: httpReadSeeker: failed open: unexpected status code https://m.daocloud.io/v2/gcr.io/ml-pipeline/mysql/blobs/sha256:591a87fc59ec1810ebb02cfb88a30efa5bd8ea6e1208aad23086607645124662: 503 Service Temporarily Unavailable

/bug

docker pull k8s.gcr.io的镜像

如何设置镜像代理，实现docker pull来下载k8s.gcr.io的镜像