Image promotion between environment about osio-pipeline

[chmouel]

Here is an email sent to the list abotu it:

Hello,

As part of building the support for OSD onto devtools build we have to
tackle the issue to be able to promote an image from one stage to
another.

In other words, "I am building my image on the 'build' OSD cluster
and I am going to promote that image to the 'staging' OSD cluster".

There is a few solutions we have discussed and investigated and we were
wondering if someone else has another opinion on this and provide us for
some guidance.

1) Using skopeo to import/export the images between the registries.

With skopeo we are able get the image that was just built from the
internal docker registry for example I can do :

$ skopeo --insecure-policy copy --src-tls-verify=false  --src-creds=openshift:$(cat /run/secrets/kubernetes.io/serviceaccount/token) docker://docker-registry.default.svc:5000/cboudjna/nodejs-health-check-master@sha256:e97130eb311ee8d789f01c3a198ad5e0eded7d969c4d2c826823290ffba1e9bb docker-archive:myimage.tar:latest

and this will produce a tarball called myimage.tar

The issue here is that we don't have access to the target
docker-registry, since AFAIK it's not exposed, which mean the only
solution is to copy that tarball over and then importing with skopeo on
the target cluster. (streaming via some oc rsh magic could be too
flaky). 

This brings some challenges, with regard to disk space management
and network transfer of a large blob that can go wrong. If the network
is something that we can rely on I think it may work, but there can be quite
a lot of uncertainties (storage and network) on our way.

2) Using an external container registry

If we are able to just use an external container registry then we may be
able to avoid the network/storage transfer issue (the docker push happen from
openshift and they do this already well).

Assuming we are able to tell the user that if you are using OSD,
you need to provide a secret with the container-url/username/password/ssl,
which could be (ideally) on quay.io, docker registry or other enterprise
container registry (i.e: artifactory) and we will use this to push from
build and pull from staging/prod.

The way the user would provide that information is to do a "oc create
secret" etc.. or do this from the openshift console UI. Which is not
very user friendly.

Ideally we would want to have a way to add a external container registry when
doing OSD, for example the user would do the Oauth dance at launcher
time with quay.io, auth would store that token (like the way it does for
GitHub) and we would just request this from the build pipeline to push
and pull.

But I guess it could take quite a while to get , and having to tell
the user to  "oc create secret" could be a compromise until this get
prioritized and happening.

3) Rebuilding on each stage,

This is IMHO a big no-no when doing build, you need to have the exact
image that you have built and tested on stage going to be on prod, but
opinions may vary on it.

Conclusion:

My preference would be to be able to do 2), which is something that is much
nicer for the end user including having all that image security checks for
free on quay.

But then we need to be able to get the green light from quay and if
product thinks it's a good option for our users.

Thanks for reading this email until the end ;)

Chmouel

[chmouel]

Reworded the issue, pref goes to external registry.

[chmouel]

Skopeo is working:

/tmp $ skopeo --insecure-policy copy --src-tls-verify=false  --src-creds=openshift:$(oc whoami -t) docker://docker-registry.default.svc:5000/cboudjna/nodejs-health-check-master@sha256:e97130eb311ee8d789
f01c3a198ad5e0eded7d969c4d2c826823290ffba1e9bb docker-archive:myimage:latest
Getting image source signatures
Copying blob sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
 70.77 MB / 71.24 MB [=========================================================]
[...]
/tmp $ tar tvf myimage
-r--r--r-- 0/0  74703002 1970-01-01 00:00:00 sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7
-r--r--r-- 0/0        32 1970-01-01 00:00:00 sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
-r--r--r-- 0/0   9894678 1970-01-01 00:00:00 sha256:9291e350be08c93d4e48286db9e72d9c74c7cba5778729d167220c664656d314
-r--r--r-- 0/0      4746 1970-01-01 00:00:00 sha256:4d3620628f7cc3d26ed80469f503f521a9939c2aaac68ec1a1bb4076d92b77e8
-r--r--r-- 0/0    190724 1970-01-01 00:00:00 sha256:94d526a435e926d08aaf39d22731d1d58935203409978a2a7eb170fb76a9d4d8
-r--r--r-- 0/0 103226229 1970-01-01 00:00:00 sha256:2603cb6c0d494298779d11c9f9888824a121946162ad28397377d47738b15f5e
-r--r--r-- 0/0      2046 1970-01-01 00:00:00 sha256:728a0a7db0d02d8e9c81e51e106bda10c32b91f86b9b4edb39dba75661ea08b5
-r--r--r-- 0/0       846 1970-01-01 00:00:00 sha256:e080bc258aaac755ec6925b07713752321ff5b2ea8a50255c56a85bb572f3cbb
-r--r--r-- 0/0  54210233 1970-01-01 00:00:00 sha256:6e34dfa7f5a85d01ce922fbb50aff377dd26bd647a42b81863de75ec4f349633
-r--r--r-- 0/0   3599162 1970-01-01 00:00:00 sha256:37b7bdc845844079a2c019a2873485831713a30e68819a9c48c1d8eed31b87d9
-r--r--r-- 0/0     18399 1970-01-01 00:00:00 sha256:311f87888d3fa018f1de2386e92978ade26e963a5f2e61d90a795e577eabd272
-r--r--r-- 0/0       810 1970-01-01 00:00:00 manifest.json

My main worry with this method is :

• Transfers timeouts, those are big images things can go wrong while transferring between envs, how do we handle this properly.
• Space management, how can we be sure we have enough space for export/import ?

[chmouel]

#103 is.a requirement first to implement this,

[chmouel]

May be dependent of #115