opensciencegrid / cvmfs-singularity-sync Goto Github PK

View Code? Open in Web Editor NEW

22.0 22.0 139.0 654 KB

Scripts to synchronize Singularity images to a CVMFS repository.

License: Apache License 2.0

Python 100.00%

cvmfs singularity

cvmfs-singularity-sync's People

Stargazers

Watchers

Forkers

whit2333 lincolnbryant soichih duncan-brown rynge decibelcooper rjones30 dmbala lukasheinrich kratsg ucatlas djw8605 hskarlupka efajardo alemsh adwasser brichards64 slr71 kreczko drtmfigy jbustamante35 jthiltges clelange rafaelfsilva upendrak christinalk wdconinc chuncheong smuzaffar mambelli yixingfu novaexperiment carriebrown stranda chunshen1987 lscsoft npavlovikj josh-willis evvaletov astroclark hzhu16 rat-h asnaylor amcooksey tcarver1234 npcooley bhpire atuldeshpande dgarijo khurtado andrestanasijczuk dominickzhang rfqed relugzosiraba maureeungaro bardelch danc2050 gregthain nguyenatrowan chadhanna tylern4 sam-may schenke alecgunny 4ndr85 sinan-yavuz vedularaghu cnatzke-forked mtint mcoughlin luca-rei amitkr2410 mynameispatrick rndsrc ca-113 yquana hugo-leung libgober cailmdaley cmessick xwcl danosu fitzlab-al lmlepin9 00mjk bastodian mfrayer ericmjonas achim1 eic wweiphy spxiwh eharstad rodrigo-tenorio lipeles grassla npanicker1 patrickrmiles deltarod jcha40

cvmfs-singularity-sync's Issues

please delete ligo-wn image from CVMFS

I'm filing a GitHub issue on the principle that this is easier to track than e-mail threads. I would like you to delete the following path and all its contents:

/cvmfs/singularity.opensciencegrid.org/ligo/ligo-wn:latest

It has been replaced by several tags under ligo/software. Let me know if you prefer this request to come other ways.

ubuntu and r-ver keeps failing

We have two images which keeps failing:

registry.hub.docker.com/library/ubuntu:latest
registry.hub.docker.com/rocker/r-ver:latest

But they can be pulled by hand, which makes me think there is something in our code causing the problem.

Traceback (most recent call last):
  File "/github/workspace/./cvmfs-singularity-sync", line 480, in verify_image
    hub.manifest(namespace, repo_name, repo_tag, head=True)
  File "/github/workspace/dockerhub.py", line 532, in manifest
    return self._do_requests_head(url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 340, in _do_requests_head
    return self._do_request('HEAD', address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 318, in _do_request
    return self._do_request(method, address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 320, in _do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://registry.hub.docker.com/v2/library/ubuntu/manifests/latest
Traceback (most recent call last):
  File "/github/workspace/./cvmfs-singularity-sync", line 204, in main
    retval = verify_image(image, registry, doauth, manifest_cache)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/./cvmfs-singularity-sync", line 480, in verify_image
    hub.manifest(namespace, repo_name, repo_tag, head=True)
  File "/github/workspace/dockerhub.py", line 532, in manifest
    return self._do_requests_head(url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 340, in _do_requests_head
    return self._do_request('HEAD', address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 318, in _do_request
    return self._do_request(method, address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 320, in _do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://registry.hub.docker.com/v2/library/ubuntu/manifests/latest
Traceback (most recent call last):
  File "/github/workspace/./cvmfs-singularity-sync", line 480, in verify_image
    hub.manifest(namespace, repo_name, repo_tag, head=True)
  File "/github/workspace/dockerhub.py", line 532, in manifest
    return self._do_requests_head(url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 340, in _do_requests_head
    return self._do_request('HEAD', address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 318, in _do_request
    return self._do_request(method, address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 320, in _do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://registry.hub.docker.com/v2/library/ubuntu/manifests/latest
Traceback (most recent call last):
  File "/github/workspace/./cvmfs-singularity-sync", line 204, in main
    retval = verify_image(image, registry, doauth, manifest_cache)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/./cvmfs-singularity-sync", line 480, in verify_image
    hub.manifest(namespace, repo_name, repo_tag, head=True)
  File "/github/workspace/dockerhub.py", line 532, in manifest
    return self._do_requests_head(url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 340, in _do_requests_head
    return self._do_request('HEAD', address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 318, in _do_request
    return self._do_request(method, address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 320, in _do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://registry.hub.docker.com/v2/library/ubuntu/manifests/latest
Traceback (most recent call last):
  File "/github/workspace/./cvmfs-singularity-sync", line 480, in verify_image
    hub.manifest(namespace, repo_name, repo_tag, head=True)
  File "/github/workspace/dockerhub.py", line 532, in manifest
    return self._do_requests_head(url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 340, in _do_requests_head
    return self._do_request('HEAD', address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 318, in _do_request
    return self._do_request(method, address, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/github/workspace/dockerhub.py", line 320, in _do_request
    resp.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://registry.hub.docker.com/v2/library/ubuntu/manifests/latest

cannot download unofficial dockerhub repos

I am trying to use this script in my own workflow. I find that it works fine for official repos such as centos:6 (etc) but not for ones in other namespaces. e.g. LIGO

thomas.downes@downes-dev:~/cvmfs-singularity-sync$ ./cvmfs-singularity-sync --docker ligo/ligo-wn:latest --rootfs /tmp/tpdownes/
Docker image to publish: ligo/ligo-wn:latest
Error getting tags using url https://registry-1.docker.io/v2/ligo/ligo-wn/tags/list

I do not believe this was always the case. It affects the bulk of the repos listed in docker_images.txt and the behavior is the same if I use the images-url argument instead.

I have also tried reverting to a version prior to glob support with the same outcome. I have even tried the cli.py file inside the singularity package. Same outcome.

Is this what it would look like if I have been blocked by Docker hub?

Cleanup of old images appears to be broken

There are currently 2194 images in /cvmfs/singularity.opensciencegrid.org/.images but only 158 names. Looking at the timestamps of the top level directories of the 2038 images that are not linked from anywhere, they are dating back to February 4 2017. Is there any attempt to clean out old images? If so, it is not working.

I think it is clear that old images shouldn't be deleted immediately after they are replaced, but they should get deleted after some period of time, for example a month after they were last current. Perhaps when an image is replaced the old one can be "touched" to update its modification time and then any unlinked image older than 30 days removed.

Delete images that are no longer in the docker_images.txt

The sync does not delete containers that no longer are listed in the docker_images.txt file.

Add README.md

Add a README.md that gives details on how to add custom images to the repo, or at very least contains a link to the docs with that info.

add atlas analysisbase image

ATLAS is starting to build docker images of its analysis software and I wanted to test it via singularity

can you add atlas/analysisbase:21.2.4 to the OSG cvmfs repo?

Thanks,
Lukas

CVMFS not syncing for drtmfigy/fbps

I have pushed a new Docker container to drtmfigy/fbps on the Docker hub. However,
the CVMFS has not been rebuilt on OSG.

Multiarchitecture images: is amd64 guaranteed to be synced?

We are exploring expanding to multiarchitecture images for some of our images that are synced to cvmfs. Does the infrastructure here guarantee that the amd64 image will continue to get synced to cvmfs, not the arm64 image, or is this untested behavior?

Use singularity to build images from Docker

This issue came up initially as a bug filed against PyCBC in issue 3184. Because the existing script for creating singularity images from docker images uses a custom script that does not call singularity directly, we have (and possibly other users have as well) placed into our docker images some of the directories and files that singularity uses (the directory /.singularity.d and some subdirectories and files beneath it). While this works for our singularity images created and published to CVMFS via this repo, it breaks the ability of any other users to create a singularity image from our docker images, as it fails because the files and symlinks that process would create are already present.

It looks like the open PR 196 and PR 195 are both relevant here. I don't know what that status of that work is, nor how they interact with each other (they look to me both to be PRs to accomplish the same thing, but I may be misunderstanding). Can someone comment on the timeline for the work in those PRs, or otherwise on the feasibility of modifying the cvmfs-singularity-sync script to invoke singularity directly, so that we could remove the specialized directories in our docker images and use the corresponding docker constructs instead?

I'm also tagging @duncan-brown for feedback on this, as he may know other gotchas and because he created the structure that PyCBC is using initially.

Deploying non-x86_64 images e.g. aarch64 and ppc64le

Can we deploy images for aarch64 and ppc64le? For CMSSW we have images like

cmssw/el8:x86_64 (same as cmssw/cms:rhel8)
cmssw/el8:aarch64
cmssw/el8:ppc64le

cmssw/el8:x86_64 are already deployed. Is it possible to deploy/unpack aarch64/ppc64le images?

Note that these are not multiarch images but have different tag for each image

Allow single character glob pattern '?' in tags

It would be nice if the script processing the docker_images.txt file could also deal with single character glob pattern '?' in addition to '*'.

For example, we might have the following tags in the registry:

image:v1-compiler1, image:v1
image:v1-compiler2
image:v2-compiler1
image:v2-compiler2, image:v2

where image:v1 and image:v2 are aliases to our default images built with the corresponding compiler (or OS). Now, if we wanted to export only the short alias tags we could specify in docker_images.txt something like this:

image:v?

My understanding is that if we use image:v* then all tags will be exported which is undesirable.

opensciencegrid / cvmfs-singularity-sync Goto Github PK

cvmfs-singularity-sync's People

Stargazers

Watchers

Forkers

cvmfs-singularity-sync's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs